diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..77ac75498fba8af48ba35bda27fca8ff915cdff3 --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +.venv/ +__pycache__/ +*.pyc diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..5897d00bfc8eee48d6ebad72053fe66553643149 --- /dev/null +++ b/README.md @@ -0,0 +1,154 @@ +--- +library_name: pytorch +tags: + - point-cloud + - semantic-segmentation + - point-transformer-v3 + - lidar + - 3d +datasets: + - RPTU-FGMB/DeKH +license: cc-by-nc-sa-4.0 +--- + +# BIMStruct3D-segmentation + +Point-cloud semantic segmentation model (PT-v3m1) from the +[BIMStruct3D](#citation) scan-to-BIM pipeline. + +[![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/) [![arXiv](https://img.shields.io/badge/arXiv-2604.24311-b31b1b.svg)](https://arxiv.org/abs/2604.24311) [![EC3 2026](https://img.shields.io/badge/EC3-2026-blue.svg)](https://ec-3.org/publication/ec32026_197/) [![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-DeKH-ffd21e.svg)](https://huggingface.co/datasets/RPTU-FGMB/DeKH) + +This folder is a self-contained package: model weights, config, a minimal copy +of the [Pointcept](https://github.com/Pointcept/Pointcept) codebase (which +this model was trained with), and one script (`segment_scan.py`) that takes a +raw `.las`/`.laz`/`.ply` scan and produces a segmented file of the same +format back out. + +## What's in here + +``` +configs/model_config.py model architecture + class list +weights/model_best.pth trained checkpoint +pointcept/ trimmed copy of the Pointcept codebase (see below) +segment_scan.py chunk -> infer -> merge inference script +pyproject.toml / uv.lock environment definition +``` + +## Model + +- Backbone: **PT-v3m1** (Point Transformer V3), ~46M parameters. +- Pretrained jointly on Structured3D + ScanNet + S3DIS (Point Prompt + Training / PPT-v1m1) with a `RandomColorDrop` augmentation (points + randomly lose their color during training, so the model doesn't over-rely + on it). +- Fine-tuned on **CV4AEC** training data (indoor/construction point clouds). +- 10 output classes: `clutter, floor, ceiling, wall, column, door, window, + stairs, railing, lights` (see `configs/model_config.py` -> `data.names`). + +The `pointcept/` directory here is deliberately **not** a full checkout of +the Pointcept codebase or of the internal repo this model was trained in — +it's trimmed to only what's needed to build this specific model and run +inference (backbone + generic dataset/transform machinery), so it doesn't +pull in unrelated dataset loaders or the CV4AEC-specific dataset class +(`CV4AECDataset`) at all -- `segment_scan.py` builds tiles as plain +`pointcept.datasets.DefaultDataset` items instead, so you don't need the +CV4AEC dataset or its raw data to run this. If you want the full upstream +codebase instead (e.g. to train further), see the notes at the bottom. + +## Setup + +Requires a CUDA GPU (tested on an RTX 4090) and +[`uv`](https://docs.astral.sh/uv/): + +```bash +uv sync +``` + +This creates `.venv/` with Python 3.12, PyTorch 2.7 (CUDA 12.6), spconv, +torch-scatter/torch-cluster, and the handful of other packages +`pointcept/` needs -- pinned in `uv.lock` for a reproducible install. +`flash-attn` is deliberately **not** installed (it requires a slow +from-source build); the model falls back to standard attention, which this +config already reflects (`enable_flash=False`). + +## Running it + +```bash +uv run segment_scan.py --input your_scan.las --output your_scan_segmented.las +``` + +`--input`/`--output` accept `.las`, `.laz`, or `.ply` (mix and match -- +input and output don't need to be the same format). + +What it does: +1. Reads the point cloud and estimates per-point normals (the model expects + color + normals as input features). +2. Splits it into overlapping ~20m tiles (large scans don't fit the model in + one shot) -- `--tile-size` / `--overlap` to change this. +3. Runs each tile through the model with 10-way test-time augmentation + (5 scales x flip/no-flip), softmax-averaged. +4. Merges tiles back into one point cloud: for points covered by more than + one tile (the overlap region), takes a majority vote across tiles. +5. Writes a file with the original coordinates/colors plus a per-point + `classification` field (0-9, matching the class list above -- for `.ply` + this is a custom scalar field, readable in CloudCompare/MeshLab) and a + companion `_labels.npy`. + +On a ~6.6M point scan this takes about 8 minutes end-to-end on an RTX 4090 +(most of it is the 10x test-time-augmentation passes per tile). Use +`--tile-size` to trade off tile count vs. points-per-tile if you need to fit +tighter GPU memory, or edit `TEST_CFG["aug_transform"]` at the top of +`segment_scan.py` to drop some of the TTA passes for a faster, slightly +less accurate run. + +## If you want the full upstream Pointcept instead + +This delivery's `pointcept/` is intentionally minimal. If you need the rest +(other model architectures, training scripts, the full dataset zoo), clone + and drop `configs/model_config.py` ++ `weights/model_best.pth` into it -- the checkpoint loads cleanly against +current upstream (`strict=True`), with one config-level rename to be aware +of: newer Pointcept renamed the backbone's `cls_mode` argument to +`enc_mode` (already reflected in `configs/model_config.py` here). + +## License + +- **Code** (`segment_scan.py`, `pointcept/`, configs): MIT. +- **Weights** (`weights/model_best.pth`): CC-BY-NC-SA-4.0. + +## Citation + +If you use this model, please cite the paper it was built for: + +```bibtex +@inproceedings{chamseddine2026bimstruct3d, + author = {Chamseddine, Mahdi and Kaufmann, Fabian and Schellen, Marius and Glock, Christian and Stricker, Didier and Rambach, Jason}, + booktitle = {Proceedings of the European Conference on Computing in Construction (EC3)}, + organization = {European Council on Computing in Construction}, + title = {BIMStruct3D: A Fully Automated Hybrid Learning Scan-to-BIM Pipeline with Integrated Topology Refinement}, + year = {2026}, + doi = {10.35490/EC3.2026} +} +``` + +Please also credit the [CV4AEC Scan-to-BIM Competition 2024](https://cv4aec.github.io/cvpr2024) +([challenge repo](https://github.com/GradientSpaces/cv4aec-challenge), Gradient +Spaces @ Stanford) if you use data or results tied to that benchmark -- this +is the specific challenge whose data the model was fine-tuned on. + +## Related dataset + +This model was used to produce the semantic segmentation underlying the +BIMStruct3D pipeline's results on the **DeKH (German Hospital) dataset**: +real hospital point clouds (four scenes across three buildings) with +semantic annotations and ground-truth IFC BIM models, released alongside +the paper above. + +- Dataset: +- License: CC BY-NC-SA 4.0 (non-commercial) + +## Acknowledgement + +This research was funded by the European Union as part of the projects: +HumanTech (Grant Agreement 101058236) and ShieldBOT (Grant Agreement +101235093). diff --git a/configs/model_config.py b/configs/model_config.py new file mode 100644 index 0000000000000000000000000000000000000000..2d25b3d60b02922808cf1805a92074ab2e35c214 --- /dev/null +++ b/configs/model_config.py @@ -0,0 +1,114 @@ +# Resolved training/model config for the bundled checkpoint (weights/model_best.pth). +# Trimmed down from the original training config: the original `data.train` / +# `data.val` / `data.test` blocks referenced our internal dataset class and +# training-data paths, which aren't needed to run inference and have been +# removed. `segment_scan.py` builds the model straight from `model` below and +# applies its own preprocessing pipeline (see PREPROCESS_TRANSFORM / +# TEST_TIME_AUGMENTATIONS in that script) instead of going through a dataset +# class, so nothing here is unused cruft other than the two notes below. + +weight = "weights/model_best.pth" # relative to this delivery folder's root +resume = False +evaluate = True +test_only = False +seed = 37743921 + +# save_path is only consulted by the standard Pointcept tools/train.py / +# tools/test.py entry points (dataset-based train/eval), which this delivery +# doesn't use. Left here only so this file stays loadable by those scripts if +# you ever want to wire up your own dataset for standard evaluation. +save_path = "./output" + +num_worker = 8 +batch_size = 1 +batch_size_val = None +batch_size_test = None +epoch = 3000 +eval_epoch = 100 +sync_bn = False +enable_amp = True +empty_cache = False +empty_cache_per_epoch = False +find_unused_parameters = False +mix_prob = 0.8 +param_dicts = [dict(keyword="block", lr=0.0003)] +hooks = [ + dict(type="CheckpointLoader"), + dict(type="IterationTimer", warmup_iter=2), + dict(type="InformationWriter"), + dict(type="SemSegEvaluator"), + dict(type="CheckpointSaver", save_freq=None), + dict(type="PreciseEvaluator", test_last=False), +] +train = dict(type="DefaultTrainer") +test = dict(type="SemSegTester", verbose=True) +model = dict( + type="DefaultSegmentorV2", + num_classes=10, + backbone_out_channels=64, + backbone=dict( + type="PT-v3m1", + in_channels=6, + order=("z", "z-trans", "hilbert", "hilbert-trans"), + stride=(2, 2, 2, 2), + enc_depths=(2, 2, 2, 6, 2), + enc_channels=(32, 64, 128, 256, 512), + enc_num_head=(2, 4, 8, 16, 32), + enc_patch_size=(1024, 1024, 1024, 1024, 1024), + dec_depths=(2, 2, 2, 2), + dec_channels=(64, 64, 128, 256), + dec_num_head=(4, 4, 8, 16), + dec_patch_size=(1024, 1024, 1024, 1024), + mlp_ratio=4, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.3, + shuffle_orders=True, + pre_norm=True, + enable_rpe=False, + enable_flash=False, # flash-attn deliberately excluded from the minimal install; standard attention is used instead + upcast_attention=False, + upcast_softmax=False, + enc_mode=False, # renamed from `cls_mode` in newer upstream Pointcept + pdnorm_bn=False, + pdnorm_ln=False, + pdnorm_decouple=True, + pdnorm_adaptive=False, + pdnorm_affine=True, + pdnorm_conditions=("ScanNet", "S3DIS", "Structured3D"), + ), + criteria=[ + dict(type="CrossEntropyLoss", loss_weight=1.0, ignore_index=-1), + dict(type="LovaszLoss", mode="multiclass", loss_weight=1.0, ignore_index=-1), + ], +) +optimizer = dict(type="AdamW", lr=0.003, weight_decay=0.05) +scheduler = dict( + type="OneCycleLR", + max_lr=[0.003, 0.0003], + pct_start=0.05, + anneal_strategy="cos", + div_factor=10.0, + final_div_factor=1000.0, +) + +# Class scheme the model was trained on. Predicted label indices from +# segment_scan.py map directly onto this list (no remap table needed). +data = dict( + num_classes=10, + ignore_index=-1, + names=[ + "clutter", + "floor", + "ceiling", + "wall", + "column", + "door", + "window", + "stairs", + "railing", + "lights", + ], +) diff --git a/pointcept/__init__.py b/pointcept/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/pointcept/datasets/__init__.py b/pointcept/datasets/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..9d6eab5e1696fa5697f62742c3a80c56a4ef0c3d --- /dev/null +++ b/pointcept/datasets/__init__.py @@ -0,0 +1,9 @@ +# Trimmed for inference-only use: segment_scan.py only needs the generic +# DefaultDataset (directory-of-.npy-assets loader) plus the transform/collate +# machinery. The named dataset classes below each need their own source +# data on disk to even be useful, and some pull in extra devkits (nuscenes, +# waymo, ...) we don't need here. +from .defaults import DefaultDataset, ConcatDataset +from .builder import build_dataset +from .utils import point_collate_fn, collate_fn +from .transform import * diff --git a/pointcept/datasets/aeo.py b/pointcept/datasets/aeo.py new file mode 100644 index 0000000000000000000000000000000000000000..70f45c904c67486ad21cc21bc79b3ffd180394ae --- /dev/null +++ b/pointcept/datasets/aeo.py @@ -0,0 +1,29 @@ +""" +AEO Dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os + +import numpy as np + +from .defaults import DefaultDataset +from .builder import DATASETS + + +@DATASETS.register_module() +class AEODataset(DefaultDataset): + def __init__(self, **kwargs): + super().__init__(**kwargs) + label_mapping = np.ones(41, dtype=int) * -1 + label_mapping[[0, 1, 3, 4, 13, 16, 19, 21, 22, 28, 29, 34, 36, 37, 38, 39]] = ( + np.arange(16) + ) + self.label_mapping = label_mapping + + def get_data(self, idx): + data_dict = super().get_data(idx) + data_dict["segment"] = self.label_mapping[data_dict["segment"]] + return data_dict diff --git a/pointcept/datasets/builder.py b/pointcept/datasets/builder.py new file mode 100644 index 0000000000000000000000000000000000000000..1fa5f0ee71bf934d5c1bfe5c71446bfecba49f11 --- /dev/null +++ b/pointcept/datasets/builder.py @@ -0,0 +1,15 @@ +""" +Dataset Builder + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from pointcept.utils.registry import Registry + +DATASETS = Registry("datasets") + + +def build_dataset(cfg): + """Build datasets.""" + return DATASETS.build(cfg) diff --git a/pointcept/datasets/cap3d.py b/pointcept/datasets/cap3d.py new file mode 100644 index 0000000000000000000000000000000000000000..6a0a48f5159080cdf815a3e3f776cb6b0d01a9a5 --- /dev/null +++ b/pointcept/datasets/cap3d.py @@ -0,0 +1,245 @@ +""" +Objaverse Point Cloud Dataset Loader + +Load pre-processed .pt files. +Each .pt file is assumed to be a dictionary of PyTorch tensors. +""" + +import os +import torch +import numpy as np +import open3d as o3d +from PIL import Image +import json +from collections.abc import Sequence +import random + +from .builder import DATASETS +from .defaults import DefaultDataset, DefaultImagePointDataset + + +@DATASETS.register_module() +class Cap3DDataset(DefaultDataset): + """ + Cap3D Dataset for loading .pt files. + """ + + def __init__( + self, + data_num=None, + **kwargs, + ): + super().__init__(**kwargs) + if data_num: + self.data_list = self.data_list[:data_num] + + @staticmethod + def get_normals(center, coords): + Cs = np.repeat(center.reshape((1, -1)), coords.shape[0], axis=0) + view_dirs = coords - Cs + view_dirs = view_dirs / np.linalg.norm(view_dirs, axis=-1, keepdims=True) + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coords) + pcd.estimate_normals() + normals = np.asarray(pcd.normals) + dot_product = np.sum(normals * view_dirs, axis=-1) + flip_mask = dot_product < 0 + normals[flip_mask] = -normals[flip_mask] + + # Normalize normals a nd m + normals = normals / np.linalg.norm(normals, axis=-1, keepdims=True) + return normals + + def get_data(self, idx): + """ + The core data loading function. + + Args: + idx (int): The index of the data sample. + + Returns: + dict: A dictionary containing the loaded and processed point cloud data. + """ + data_path = self.data_list[idx % len(self.data_list)] + name = self.get_data_name(idx) + split = self.get_split_name(idx) + + try: + data = torch.load(data_path, map_location="cpu") + except Exception as e: + raise IOError(f"Failed to load a .pt file at {data_path}. Error: {e}") + + data = data.numpy().transpose() + color = data[:, 3:] + coord = data[:, :3] + normal = self.get_normals(np.mean(coord, axis=0), coord) + + data_dict = {} + + data_dict["coord"] = coord + data_dict["color"] = color + data_dict["normal"] = normal + + data_dict["name"] = name + data_dict["split"] = split + + return data_dict + + +@DATASETS.register_module() +class Cap3DImagePointDataset(DefaultImagePointDataset): + def __init__( + self, + data_num=-1, + **kwargs, + ): + self.data_num = data_num + super().__init__(**kwargs) + + def get_data_list(self): + split_list = {} + if isinstance(self.split, str): + data_path = os.path.join(self.data_root, "splits", f"{self.split}.json") + with open(data_path, "r", encoding="utf-8") as file: + data_list = json.load(file) + split_list[self.split] = list(data_list.keys()) + elif isinstance(self.split, Sequence): + data_list = {} + for split in self.split: + data_path = os.path.join(self.data_root, "splits", f"{split}.json") + with open(data_path, "r", encoding="utf-8") as file: + data_split_dict = json.load(file) + data_list.update(data_split_dict) + split_list[split] = list(data_split_dict.keys()) + else: + raise NotImplementedError + + if self.data_num < 0 or self.data_num >= len(data_list): + selected_data_list = data_list + selected_split_list = split_list + return selected_data_list, selected_split_list + + all_keys = list(data_list.keys()) + selected_keys = random.sample(all_keys, self.data_num) + selected_data_list = {key: data_list[key] for key in selected_keys} + valid_keys = set(selected_keys) + + selected_split_list = {} + for split_name, key_list in split_list.items(): + filtered_keys = [key for key in key_list if key in valid_keys] + if filtered_keys: + selected_split_list[split_name] = filtered_keys + return selected_data_list, selected_split_list + + @staticmethod + def get_normals(center, coords): + Cs = np.repeat(center.reshape((1, -1)), coords.shape[0], axis=0) + view_dirs = coords - Cs + view_dirs = view_dirs / np.linalg.norm(view_dirs, axis=-1, keepdims=True) + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coords) + pcd.estimate_normals() + normals = np.asarray(pcd.normals) + dot_product = np.sum(normals * view_dirs, axis=-1) + flip_mask = dot_product < 0 + normals[flip_mask] = -normals[flip_mask] + normals = normals / np.linalg.norm(normals, axis=-1, keepdims=True) + return normals + + def get_pc_data(self, data_path): + try: + data = torch.load(data_path, map_location="cpu") + except Exception as e: + raise IOError(f"Failed to load a .pt file at {data_path}. Error: {e}") + + data = data.numpy().transpose() + color = data[:, 3:] + coord = data[:, :3] + normal = self.get_normals(np.mean(coord, axis=0), coord) + + data_dict = {} + + data_dict["coord"] = coord + data_dict["color"] = color + data_dict["normal"] = normal + + return data_dict + + def get_data(self, idx): + data_dict = {} + name = self.get_data_name(idx) + split = self.get_split_name(idx) + data_dict["name"] = name + data_dict["split"] = split + data_path = self.data_list[name] + + pointclouds_path = data_path["pointclouds"] + pc_dict = self.get_pc_data(pointclouds_path) + data_dict.update(pc_dict) + imgs_path = data_path["images"] + imgs = [Image.open(asset).convert("RGB") for asset in imgs_path] + if len(imgs) > 0: + img_width, img_height = imgs[0].size + div_w = img_width // self.patch_w + div_h = img_height // self.patch_h + div_min = max(min(div_w, div_h), 1) + crop_img_width = div_min * self.patch_w + crop_img_height = div_min * self.patch_h + left = int((img_width - crop_img_width) / 2) + top = int((img_height - crop_img_height) / 2) + right = int((img_width + crop_img_width) / 2) + bottom = int((img_height + crop_img_height) / 2) + imgs = [img.crop((left, top, right, bottom)) for img in imgs] + imgs = [self.transform_img(img) for img in imgs] + imgs_list = torch.stack(imgs) + data_dict["images"] = imgs_list.float() + else: + data_dict["images"] = torch.empty( + (0, 3, self.patch_h * self.patch_size, self.patch_w * self.patch_size) + ) + data_dict["img_num"] = np.array([data_dict["images"].shape[0]], dtype=np.int32) + + correspondences_path = data_path["correspondences"] + correspondence_infos = np.ones( + (data_dict["coord"].shape[0], len(correspondences_path), 2), + dtype=np.float32, + ) * (-1) + for asset_id, asset in enumerate(correspondences_path): + correspondence_info = np.load(asset).astype(np.float32) + if np.array_equal(correspondence_info, -np.ones((1, 3))): + continue + correspondence_info = self.resize_correspondence_info( + correspondence_info, + (self.patch_h * self.patch_size, self.patch_w * self.patch_size), + (img_height, img_width), + (left, top, right, bottom), + self.patch_size, + ) + correspondence_infos[ + correspondence_info[:, -1].astype(np.int32), asset_id, : + ] = correspondence_info[:, :-1] + data_dict["correspondence"] = correspondence_infos + + if "coord" in data_dict.keys(): + data_dict["coord"] = data_dict["coord"].astype(np.float32) + + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"].astype(np.float32) + + if "normal" in data_dict.keys(): + data_dict["normal"] = data_dict["normal"].astype(np.float32) + + if "segment" in data_dict.keys(): + data_dict["segment"] = data_dict["segment"].reshape([-1]).astype(np.int32) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = data_dict["instance"].reshape([-1]).astype(np.int32) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + return data_dict diff --git a/pointcept/datasets/dataloader.py b/pointcept/datasets/dataloader.py new file mode 100644 index 0000000000000000000000000000000000000000..f1118c08ed884136edbeba1b7402e27ca1b2bf83 --- /dev/null +++ b/pointcept/datasets/dataloader.py @@ -0,0 +1,205 @@ +from functools import partial +import weakref +import torch +import math +import torch.utils.data +import torch.distributed as dist +from torch.utils.data import Sampler + +import pointcept.utils.comm as comm +from pointcept.datasets.utils import point_collate_fn +from pointcept.datasets import ConcatDataset +from pointcept.utils.env import set_seed + + +class MultiDatasetDummySampler: + def __init__(self): + self.dataloader = None + + def set_epoch(self, epoch): + if comm.get_world_size() > 1: + for dataloader in self.dataloader.dataloaders: + dataloader.sampler.set_epoch(epoch) + return + + +class MultiDatasetDataloader: + """ + Multiple Datasets Dataloader, batch data from a same dataset and mix up ratio determined by loop of each sub dataset. + The overall length is determined by the main dataset (first) and loop of concat dataset. + """ + + def __init__( + self, + concat_dataset: ConcatDataset, + batch_size_per_gpu: int, + num_worker_per_gpu: int, + mix_prob=0, + seed=None, + ): + self.datasets = concat_dataset.datasets + self.ratios = [dataset.loop for dataset in self.datasets] + # reset data loop, original loop serve as ratios + for dataset in self.datasets: + dataset.loop = 1 + # determine union training epoch by main dataset + self.datasets[0].loop = concat_dataset.loop + # build sub-dataloaders + num_workers = num_worker_per_gpu // len(self.datasets) + self.dataloaders = [] + for dataset_id, dataset in enumerate(self.datasets): + if comm.get_world_size() > 1: + sampler = torch.utils.data.distributed.DistributedSampler(dataset) + else: + sampler = None + + init_fn = ( + partial( + self._worker_init_fn, + dataset_id=dataset_id, + num_workers=num_workers, + num_datasets=len(self.datasets), + rank=comm.get_rank(), + seed=seed, + ) + if seed is not None + else None + ) + self.dataloaders.append( + torch.utils.data.DataLoader( + dataset, + batch_size=batch_size_per_gpu, + shuffle=(sampler is None), + num_workers=num_worker_per_gpu, + sampler=sampler, + collate_fn=partial(point_collate_fn, mix_prob=mix_prob), + pin_memory=True, + worker_init_fn=init_fn, + drop_last=True, + persistent_workers=True, + ) + ) + self.sampler = MultiDatasetDummySampler() + self.sampler.dataloader = weakref.proxy(self) + + def __iter__(self): + iterator = [iter(dataloader) for dataloader in self.dataloaders] + while True: + for i in range(len(self.ratios)): + for _ in range(self.ratios[i]): + try: + batch = next(iterator[i]) + except StopIteration: + if i == 0: + return + else: + iterator[i] = iter(self.dataloaders[i]) + batch = next(iterator[i]) + yield batch + + def __len__(self): + main_data_loader_length = len(self.dataloaders[0]) + return ( + main_data_loader_length // self.ratios[0] * sum(self.ratios) + + main_data_loader_length % self.ratios[0] + ) + + @staticmethod + def _worker_init_fn(worker_id, num_workers, dataset_id, num_datasets, rank, seed): + worker_seed = ( + num_workers * num_datasets * rank + + num_workers * dataset_id + + worker_id + + seed + ) + set_seed(worker_seed) + + +class DistributedImbalancedSampler(Sampler): + def __init__( + self, + dataset, + sampled_dataset_index, + sampled_dataset_limit, + num_replicas=None, + rank=None, + shuffle=True, + seed=0, + ): + if num_replicas is None: + if not dist.is_available(): + num_replicas = 1 + rank = 0 + else: + try: + num_replicas = dist.get_world_size() + rank = dist.get_rank() + except: + num_replicas = 1 + rank = 0 + + self.dataset = dataset + self.num_replicas = num_replicas + self.rank = rank + self.epoch = 0 + self.seed = seed + self.sampled_dataset_index = sampled_dataset_index + self.sampled_dataset_limit = sampled_dataset_limit + self.shuffle = shuffle + + self.lengths = [len(d) for d in dataset.datasets] + self.offsets = [0] + list( + torch.cumsum(torch.tensor(self.lengths), 0)[:-1].numpy() + ) + self.total_size_per_epoch = ( + sum(self.lengths) + - self.lengths[self.sampled_dataset_index] + + self.sampled_dataset_limit + ) + self.num_samples = math.ceil(self.total_size_per_epoch / self.num_replicas) + self.total_size = self.num_samples * self.num_replicas + + def __iter__(self): + g = torch.Generator() + g.manual_seed(self.seed + self.epoch) + + indices = [] + for i, length in enumerate(self.lengths): + if i == self.sampled_dataset_index: + sub_indices = ( + torch.randperm(length, generator=g)[: self.sampled_dataset_limit] + + self.offsets[i] + ) + indices.append(sub_indices) + else: + sub_indices = torch.arange(length) + self.offsets[i] + indices.append(sub_indices) + + final_indices = torch.cat(indices) + + if self.shuffle: + final_indices = final_indices[ + torch.randperm(len(final_indices), generator=g) + ] + + final_indices = final_indices.tolist() + padding_size = self.total_size - len(final_indices) + if padding_size <= len(final_indices): + final_indices += final_indices[:padding_size] + else: + final_indices += ( + final_indices * math.ceil(padding_size / len(final_indices)) + )[:padding_size] + + assert len(final_indices) == self.total_size + + offset = self.num_samples * self.rank + subsample_indices = final_indices[offset : offset + self.num_samples] + + return iter(subsample_indices) + + def __len__(self): + return self.num_samples + + def set_epoch(self, epoch): + self.epoch = epoch diff --git a/pointcept/datasets/defaults.py b/pointcept/datasets/defaults.py new file mode 100644 index 0000000000000000000000000000000000000000..07ea2a71a8efc4fb5523d12954409ae9df91361c --- /dev/null +++ b/pointcept/datasets/defaults.py @@ -0,0 +1,676 @@ +""" +Default Datasets + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import glob +import json +import numpy as np +import torch +from copy import deepcopy +from torch.utils.data import Dataset +from collections.abc import Sequence +from torchvision.transforms import InterpolationMode +from PIL import Image +from torchvision.transforms import transforms as T + +from pointcept.utils.logger import get_root_logger +from pointcept.utils.cache import shared_dict + +from .builder import DATASETS, build_dataset +from .transform import Compose, TRANSFORMS + +INTERPOLATION_MODE = { + "bilinear": InterpolationMode.BILINEAR, + "bicubic": InterpolationMode.BICUBIC, + "nearest": InterpolationMode.NEAREST, +} + + +@DATASETS.register_module() +class DefaultDataset(Dataset): + VALID_ASSETS = [ + "coord", + "color", + "normal", + "strength", + "segment", + "instance", + "pose", + ] + + def __init__( + self, + split="train", + data_root="data/dataset", + transform=None, + test_mode=False, + test_cfg=None, + cache=False, + ignore_index=-1, + loop=1, + ): + super(DefaultDataset, self).__init__() + self.data_root = data_root + self.split = split + self.transform = Compose(transform) + self.cache = cache + self.ignore_index = ignore_index + self.loop = ( + loop if not test_mode else 1 + ) # force make loop = 1 while in test mode + self.test_mode = test_mode + self.test_cfg = test_cfg if test_mode else None + if test_mode: + self.test_voxelize = TRANSFORMS.build(self.test_cfg.voxelize) + self.test_crop = ( + TRANSFORMS.build(self.test_cfg.crop) if self.test_cfg.crop else None + ) + self.post_transform = Compose(self.test_cfg.post_transform) + self.aug_transform = [Compose(aug) for aug in self.test_cfg.aug_transform] + + self.data_list = self.get_data_list() + logger = get_root_logger() + logger.info( + "Totally {} x {} samples in {} {} set.".format( + len(self.data_list), self.loop, os.path.basename(self.data_root), split + ) + ) + + def get_data_list(self): + if isinstance(self.split, str): + split_list = [self.split] + elif isinstance(self.split, Sequence): + split_list = self.split + else: + raise NotImplementedError + + data_list = [] + for split in split_list: + if os.path.isfile(os.path.join(self.data_root, split)): + with open(os.path.join(self.data_root, split)) as f: + data_list += [ + os.path.join(self.data_root, data) for data in json.load(f) + ] + else: + data_list += glob.glob(os.path.join(self.data_root, split, "*")) + return data_list + + def get_data(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + name = self.get_data_name(idx) + split = self.get_split_name(idx) + if self.cache: + cache_name = f"pointcept-{name}" + return shared_dict(cache_name) + + data_dict = {} + assets = os.listdir(data_path) + for asset in assets: + if not asset.endswith(".npy"): + continue + if asset[:-4] not in self.VALID_ASSETS: + continue + data_dict[asset[:-4]] = np.load(os.path.join(data_path, asset)) + data_dict["name"] = name + data_dict["split"] = split + + if "coord" in data_dict.keys(): + data_dict["coord"] = data_dict["coord"].astype(np.float32) + + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"].astype(np.float32) + + if "normal" in data_dict.keys(): + data_dict["normal"] = data_dict["normal"].astype(np.float32) + + if "segment" in data_dict.keys(): + data_dict["segment"] = data_dict["segment"].reshape([-1]).astype(np.int32) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = data_dict["instance"].reshape([-1]).astype(np.int32) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + return data_dict + + def get_data_name(self, idx): + return os.path.basename(self.data_list[idx % len(self.data_list)]) + + def get_split_name(self, idx): + return os.path.basename( + os.path.dirname(self.data_list[idx % len(self.data_list)]) + ) + + def prepare_train_data(self, idx): + # load data + data_dict = self.get_data(idx) + data_dict = self.transform(data_dict) + return data_dict + + def prepare_test_data(self, idx): + # load data + data_dict = self.get_data(idx) + data_dict = self.transform(data_dict) + result_dict = dict(segment=data_dict.pop("segment"), name=data_dict.pop("name")) + if "origin_segment" in data_dict: + assert "inverse" in data_dict + result_dict["origin_segment"] = data_dict.pop("origin_segment") + result_dict["inverse"] = data_dict.pop("inverse") + + data_dict_list = [] + for aug in self.aug_transform: + data_dict_list.append(aug(deepcopy(data_dict))) + + fragment_list = [] + for data in data_dict_list: + if self.test_voxelize is not None: + data_part_list = self.test_voxelize(data) + else: + data["index"] = np.arange(data["coord"].shape[0]) + data_part_list = [data] + for data_part in data_part_list: + if self.test_crop is not None: + data_part = self.test_crop(data_part) + else: + data_part = [data_part] + fragment_list += data_part + + for i in range(len(fragment_list)): + fragment_list[i] = self.post_transform(fragment_list[i]) + result_dict["fragment_list"] = fragment_list + return result_dict + + def __getitem__(self, idx): + if self.test_mode: + return self.prepare_test_data(idx) + else: + return self.prepare_train_data(idx) + + def __len__(self): + return len(self.data_list) * self.loop + + +@DATASETS.register_module() +class DefaultImagePointDataset(Dataset): + PC_VALID_ASSETS = [ + "coord", + "color", + "normal", + "superpoint", + "pose", + "segment", + "instance", + "strength", + ] + + def __init__( + self, + split="train", + data_root="data/dataset", + transform=None, + test_mode=False, + test_cfg=None, + cache=False, + ignore_index=-1, + loop=1, + crop_h=630, + crop_w=1120, + patch_size=14, + interpolation="bilinear", + if_img=True, + ): + super(DefaultImagePointDataset, self).__init__() + self.if_img = if_img + self.data_root = data_root + self.split = split + self.transform = Compose(transform) + self.cache = cache + self.ignore_index = ignore_index + self.loop = ( + loop if not test_mode else 1 + ) # force make loop = 1 while in test mode + self.test_mode = test_mode + self.test_cfg = test_cfg if test_mode else None + + if test_mode: + self.test_voxelize = TRANSFORMS.build(self.test_cfg.voxelize) + self.test_crop = ( + TRANSFORMS.build(self.test_cfg.crop) if self.test_cfg.crop else None + ) + self.post_transform = Compose(self.test_cfg.post_transform) + self.aug_transform = [Compose(aug) for aug in self.test_cfg.aug_transform] + + self.data_list, self.split_list = self.get_data_list() + if isinstance(self.data_list, dict): + self.data_name = list(self.data_list.keys()) + elif isinstance(self.data_list[0], dict): + self.data_name = list([data["token"] for data in self.data_list]) + else: + self.data_name = self.data_list + logger = get_root_logger() + logger.info( + "Totally {} x {} samples in {} {} set.".format( + len(self.data_name), self.loop, os.path.basename(self.data_root), split + ) + ) + + self.crop_h = crop_h + self.crop_w = crop_w + self.patch_size = patch_size + self.patch_h = crop_h // patch_size + self.patch_w = crop_w // patch_size + self.transform_img = T.Compose( + [ + T.Resize( + (self.patch_h * self.patch_size, self.patch_w * self.patch_size), + interpolation=INTERPOLATION_MODE[interpolation], + ), + T.ToTensor(), + ] + ) + + def resize_correspondence_info( + self, correspondence, size, size0, crop_size, _alignment + ): + h, w = size + h0, w0 = size0 + left, top, right, bottom = crop_size + crop_h = bottom - top + crop_w = right - left + mask_crop = ( + (correspondence[:, 1] >= top) + & (correspondence[:, 1] < bottom) + & (correspondence[:, 0] >= left) + & (correspondence[:, 0] < right) + ) + correspondence = correspondence[mask_crop] + correspondence[:, 1] -= top + correspondence[:, 0] -= left + correspondence[:, 1] = (correspondence[:, 1] * h / crop_h / _alignment).astype( + np.float32 + ) + correspondence[:, 0] = (correspondence[:, 0] * w / crop_w / _alignment).astype( + np.float32 + ) + correspondence = correspondence[:, [1, 0, 2]] + correspondence = np.unique(correspondence, axis=0) + return correspondence + + def get_data_list(self): + split_list = {} + if isinstance(self.split, str): + data_path = os.path.join(self.data_root, "splits", f"{self.split}.json") + with open(data_path, "r", encoding="utf-8") as file: + data_list = json.load(file) + split_list[self.split] = list(data_list.keys()) + elif isinstance(self.split, Sequence): + data_list = {} + for split in self.split: + data_path = os.path.join(self.data_root, "splits", f"{split}.json") + with open(data_path, "r", encoding="utf-8") as file: + data_split_dict = json.load(file) + data_list.update(data_split_dict) + split_list[split] = list(data_split_dict.keys()) + else: + raise NotImplementedError + return data_list, split_list + + def get_data_name(self, idx): + return self.data_name[idx % len(self.data_name)] + + def get_split_name(self, idx): + for split, names in self.split_list.items(): + if self.data_name[idx % len(self.data_name)] in names: + return split + return None + + def get_data(self, idx): + data_dict = {} + name = self.get_data_name(idx) + split = self.get_split_name(idx) + data_dict["name"] = name + data_dict["split"] = split + data_path = self.data_list[name] + + pointclouds_path = data_path["pointclouds"] + assets = os.listdir(pointclouds_path) + for asset in assets: + if not asset.endswith(".npy"): + continue + if asset[:-4] not in self.PC_VALID_ASSETS: + continue + data_dict[asset[:-4]] = np.load(os.path.join(pointclouds_path, asset)) + if self.if_img: + imgs_path = data_path["images"] + imgs = [Image.open(asset) for asset in imgs_path] + if len(imgs) > 0: + img_width, img_height = imgs[0].size + div_w = img_width // self.patch_w + div_h = img_height // self.patch_h + div_min = max(min(div_w, div_h), 1) + crop_img_width = div_min * self.patch_w + crop_img_height = div_min * self.patch_h + left = int((img_width - crop_img_width) / 2) + top = int((img_height - crop_img_height) / 2) + right = int((img_width + crop_img_width) / 2) + bottom = int((img_height + crop_img_height) / 2) + imgs = [img.crop((left, top, right, bottom)) for img in imgs] + imgs = [self.transform_img(img) for img in imgs] + imgs_list = torch.stack(imgs) + data_dict["images"] = imgs_list.float() + else: + data_dict["images"] = torch.empty( + ( + 0, + 3, + self.patch_h * self.patch_size, + self.patch_w * self.patch_size, + ) + ) + data_dict["img_num"] = np.array( + [data_dict["images"].shape[0]], dtype=np.int32 + ) + + correspondences_path = data_path["correspondences"] + correspondence_infos = np.ones( + (data_dict["coord"].shape[0], len(correspondences_path), 2), + dtype=np.float32, + ) * (-1) + for asset_id, asset in enumerate(correspondences_path): + correspondence_info = np.load(asset).astype(np.float32) + if np.array_equal(correspondence_info, -np.ones((1, 3))): + continue + correspondence_info = self.resize_correspondence_info( + correspondence_info, + (self.patch_h * self.patch_size, self.patch_w * self.patch_size), + (img_height, img_width), + (left, top, right, bottom), + self.patch_size, + ) + correspondence_infos[ + correspondence_info[:, -1].astype(np.int32), asset_id, : + ] = correspondence_info[:, :-1] + data_dict["correspondence"] = correspondence_infos # .reshape(-1, 2) + + if "coord" in data_dict.keys(): + data_dict["coord"] = data_dict["coord"].astype(np.float32) + + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"].astype(np.float32) + + if "normal" in data_dict.keys(): + data_dict["normal"] = data_dict["normal"].astype(np.float32) + + if "segment" in data_dict.keys(): + data_dict["segment"] = data_dict["segment"].reshape([-1]).astype(np.int32) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = data_dict["instance"].reshape([-1]).astype(np.int32) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + return data_dict + + def prepare_train_data(self, idx): + # load data + data_dict = self.get_data(idx) + data_dict = self.transform(data_dict) + return data_dict + + def prepare_test_data(self, idx): + # load data + data_dict = self.get_data(idx) + data_dict = self.transform(data_dict) + result_dict = dict(segment=data_dict.pop("segment"), name=data_dict.pop("name")) + if "origin_segment" in data_dict: + assert "inverse" in data_dict + result_dict["origin_segment"] = data_dict.pop("origin_segment") + result_dict["inverse"] = data_dict.pop("inverse") + + data_dict_list = [] + for aug in self.aug_transform: + data_dict_list.append(aug(deepcopy(data_dict))) + + fragment_list = [] + for data in data_dict_list: + if self.test_voxelize is not None: + data_part_list = self.test_voxelize(data) + else: + data["index"] = np.arange(data["coord"].shape[0]) + data_part_list = [data] + for data_part in data_part_list: + if self.test_crop is not None: + data_part = self.test_crop(data_part) + else: + data_part = [data_part] + fragment_list += data_part + + for i in range(len(fragment_list)): + fragment_list[i] = self.post_transform(fragment_list[i]) + result_dict["fragment_list"] = fragment_list + return result_dict + + def __getitem__(self, idx): + if self.test_mode: + return self.prepare_test_data(idx) + else: + return self.prepare_train_data(idx) + + def __len__(self): + return len(self.data_list) * self.loop + + +@DATASETS.register_module() +class DefaultMultiViewImagePointDataset(DefaultImagePointDataset): + def __init__(self, depth_scale=1000.0, if_img=True, **kwargs): + self.depth_scale = depth_scale + self.if_img = if_img + super().__init__(if_img=if_img, **kwargs) + + def get_data_list(self): + split_list = {} + if isinstance(self.split, str): + splits_to_load = [self.split] + elif isinstance(self.split, Sequence): + splits_to_load = self.split + else: + raise NotImplementedError + + data_list = {} + for split in splits_to_load: + data_path = os.path.join(self.data_root, "splits", f"{split}.json") + if not os.path.exists(data_path): + raise FileNotFoundError(f"Split file not found: {data_path}") + + with open(data_path, "r", encoding="utf-8") as file: + data_split_dict = json.load(file) + data_list.update(data_split_dict) + split_list[split] = list(data_split_dict.keys()) + return data_list, split_list + + @staticmethod + def get_normals(center, coords): + normals = coords - center[None, :] + normals = normals / np.linalg.norm(normals, axis=-1, keepdims=True) + return normals + + def get_data(self, idx): + name = self.get_data_name(idx) + split = self.get_split_name(idx) + data_info = self.data_list[name] + + all_points_world, all_colors, camera_params = [], [], [] + correspondences = [] + coord_num = 0 + for i in range(len(data_info["images"])): + rgb_path = data_info["images"][i] + depth_path = data_info["depths"][i] + pose_path = data_info["Ts"][i] + intr_path = data_info["Ks"][i] + + try: + with Image.open(rgb_path) as img: + color_img = np.array(img) + with Image.open(depth_path) as img: + depth_img = np.array(img) + cam_pose_world = np.load(pose_path) + cam_intr = np.load(intr_path) + camera_params.append({"pose": cam_pose_world, "intr": cam_intr}) + except FileNotFoundError: + continue + + depth_m = depth_img.astype(np.float32) / self.depth_scale + h, w = depth_m.shape + fx, fy, cx, cy = ( + cam_intr[0, 0], + cam_intr[1, 1], + cam_intr[0, 2], + cam_intr[1, 2], + ) + v, u = np.indices((h, w)) + valid_mask = (depth_m > 0) & (depth_m < 0.8) + z_cam = depth_m[valid_mask] + x_cam, y_cam = (u[valid_mask] - cx) * z_cam / fx, ( + v[valid_mask] - cy + ) * z_cam / fy + points_cam = np.vstack((x_cam, y_cam, z_cam)).T + points_world = ( + cam_pose_world + @ np.hstack([points_cam, np.ones((points_cam.shape[0], 1))]).T + ).T[:, :3] + all_points_world.append(points_world) + all_colors.append(color_img[valid_mask]) + + pixel = np.vstack([u[valid_mask], v[valid_mask]]).T + pixel = np.concatenate([pixel, np.arange(pixel.shape[0])[:, None]], axis=1) + pixel[:, 2] += coord_num + coord_num += points_world.shape[0] + correspondences.append(pixel) + coord = ( + np.concatenate(all_points_world, axis=0) + if all_points_world + else np.zeros((0, 3)) + ) + coord[:, 2] = -coord[:, 2] + color = np.concatenate(all_colors, axis=0) if all_colors else np.zeros((0, 3)) + normal = self.get_normals(np.array([0, 0, 0]), coord) + + if self.if_img: + imgs_path = data_info["images"] + imgs = [Image.open(asset) for asset in imgs_path] + img_width, img_height = imgs[0].size + div_w = img_width // self.patch_w + div_h = img_height // self.patch_h + div_min = max(min(div_w, div_h), 1) + crop_img_width = div_min * self.patch_w + crop_img_height = div_min * self.patch_h + left = int((img_width - crop_img_width) / 2) + top = int((img_height - crop_img_height) / 2) + right = int((img_width + crop_img_width) / 2) + bottom = int((img_height + crop_img_height) / 2) + imgs = [img.crop((left, top, right, bottom)) for img in imgs] + imgs = [self.transform_img(img) for img in imgs] + if len(imgs) > 0: + imgs_list = torch.stack(imgs) + imgs_tensor = imgs_list.float() + else: + imgs_tensor = torch.empty( + ( + 0, + 3, + self.patch_h * self.patch_size, + self.patch_w * self.patch_size, + ) + ) + img_num = np.array([imgs_tensor.shape[0]], dtype=np.int32) + + num_points, num_imgs = coord.shape[0], len(imgs) + correspondence_infos = ( + np.ones((num_points, num_imgs, 2), dtype=np.float32) * -1 + ) + + for asset_id, correspondence_info in enumerate(correspondences): + correspondence_info = self.resize_correspondence_info( + correspondence_info, + (self.patch_h * self.patch_size, self.patch_w * self.patch_size), + (img_height, img_width), + (left, top, right, bottom), + self.patch_size, + ) + correspondence_infos[ + correspondence_info[:, -1].astype(np.int32), asset_id, : + ] = correspondence_info[:, :-1] + + data_dict = { + "name": name, + "split": split, + "coord": coord.astype(np.float32), + "color": color.astype(np.float32), + "normal": normal.astype(np.float32), + "images": imgs_tensor, + "img_num": img_num, + "correspondence": correspondence_infos, + } + else: + data_dict = { + "name": name, + "split": split, + "coord": coord.astype(np.float32), + "color": color.astype(np.float32), + "normal": normal.astype(np.float32), + } + return data_dict + + +@DATASETS.register_module() +class ConcatDataset(Dataset): + def __init__(self, datasets, loop=1): + super(ConcatDataset, self).__init__() + self.datasets = [build_dataset(dataset) for dataset in datasets] + self.loop = loop + self.data_list = self.get_data_list() + logger = get_root_logger() + logger.info( + "Totally {} x {} samples in the concat set.".format( + len(self.data_list), self.loop + ) + ) + + def get_data_list(self): + data_list = [] + for i in range(len(self.datasets)): + data_list.extend( + zip( + np.ones(len(self.datasets[i]), dtype=int) * i, + np.arange(len(self.datasets[i])), + ) + ) + return data_list + + def get_data(self, idx): + dataset_idx, data_idx = self.data_list[idx % len(self.data_list)] + return self.datasets[dataset_idx][data_idx] + + def get_data_name(self, idx): + dataset_idx, data_idx = self.data_list[idx % len(self.data_list)] + return self.datasets[dataset_idx].get_data_name(data_idx) + + def __getitem__(self, idx): + return self.get_data(idx) + + def __len__(self): + return len(self.data_list) * self.loop diff --git a/pointcept/datasets/hk.py b/pointcept/datasets/hk.py new file mode 100644 index 0000000000000000000000000000000000000000..190b9958149b1360d70b5f8a11065b1528e3beee --- /dev/null +++ b/pointcept/datasets/hk.py @@ -0,0 +1,94 @@ +""" +HK Remote Data Dataset + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import numpy as np +import torch + +from pointcept.utils.cache import shared_dict +from .builder import DATASETS +from .defaults import DefaultDataset +import numpy as np +from scipy.spatial import cKDTree + + +def find_closest_pair_kdtree(points): + n = points.shape[0] + if n < 2: + return float("inf"), None, None + + tree = cKDTree(points) + distances, indices = tree.query(points, k=2) + min_dist = np.min(distances[:, 1]) + min_idx_in_distances = np.argmin(distances[:, 1]) + point1_idx = min_idx_in_distances + point2_idx = indices[min_idx_in_distances, 1] + + return min_dist, (points[point1_idx], points[point2_idx]) + + +@DATASETS.register_module() +class HKDataset(DefaultDataset): + def __init__(self, crop_h=630, crop_w=1120, patch_size=14, if_img=True, **kwargs): + self.crop_h = crop_h + self.crop_w = crop_w + self.patch_size = patch_size + self.patch_h = crop_h // patch_size + self.patch_w = crop_w // patch_size + self.if_img = if_img + super().__init__(**kwargs) + + def get_data(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + name = self.get_data_name(idx) + split = self.get_split_name(idx) + if self.cache: + cache_name = f"pointcept-{name}" + return shared_dict(cache_name) + + data_dict = {} + assets = os.listdir(data_path) + for asset in assets: + if not asset.endswith(".npy"): + continue + if asset[:-4] not in self.VALID_ASSETS: + continue + data_dict[asset[:-4]] = np.load(os.path.join(data_path, asset)) + data_dict["name"] = name + data_dict["split"] = split + if self.if_img: + data_dict["images"] = torch.empty( + (0, 3, self.patch_h * self.patch_size, self.patch_w * self.patch_size) + ) + data_dict["img_num"] = np.array([0], dtype=np.int32) + data_dict["correspondence"] = np.ones( + (data_dict["coord"].shape[0], 0, 2), dtype=np.float32 + ) * (-1) + if "coord" in data_dict.keys(): + data_dict["coord"] = data_dict["coord"].astype(np.float32) + + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"].astype(np.float32) + + if "normal" in data_dict.keys(): + data_dict["normal"] = data_dict["normal"].astype(np.float32) + + if "segment" in data_dict.keys(): + data_dict["segment"] = data_dict["segment"].reshape([-1]).astype(np.int32) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = data_dict["instance"].reshape([-1]).astype(np.int32) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + return data_dict diff --git a/pointcept/datasets/hm3d.py b/pointcept/datasets/hm3d.py new file mode 100644 index 0000000000000000000000000000000000000000..2665f028147c35ab6c1e63a715660a8095359f94 --- /dev/null +++ b/pointcept/datasets/hm3d.py @@ -0,0 +1,45 @@ +""" +Habitat-Matterport 3D Dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import glob +import os +from collections.abc import Sequence +from .builder import DATASETS +from .defaults import DefaultDataset + + +@DATASETS.register_module() +class HM3DDataset(DefaultDataset): + def __init__( + self, + force_label=True, + **kwargs, + ): + # if force_label, only load data with label + self.force_label = force_label + super().__init__(**kwargs) + + def get_single_data_list(self, split): + if self.force_label: + data_list = glob.glob( + os.path.join(self.data_root, split, "*", "segment.npy") + ) + data_list = [os.path.dirname(data) for data in data_list] + else: + data_list = glob.glob(os.path.join(self.data_root, split, "*")) + return data_list + + def get_data_list(self): + if isinstance(self.split, str): + data_list = self.get_single_data_list(self.split) + elif isinstance(self.split, Sequence): + data_list = [] + for split in self.split: + data_list += self.get_single_data_list(split) + else: + raise NotImplementedError + return data_list diff --git a/pointcept/datasets/modelnet.py b/pointcept/datasets/modelnet.py new file mode 100644 index 0000000000000000000000000000000000000000..b00fd935aebbb2ea6735c45cf444374df86a95eb --- /dev/null +++ b/pointcept/datasets/modelnet.py @@ -0,0 +1,156 @@ +""" +ModelNet40 Dataset + +get sampled point clouds of ModelNet40 (XYZ and normal from mesh, 10k points per shape) +at "https://shapenet.cs.stanford.edu/media/modelnet40_normal_resampled.zip" + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import numpy as np +import copy +import pointops +import torch +from torch.utils.data import Dataset +from copy import deepcopy + + +from pointcept.utils.logger import get_root_logger +from .builder import DATASETS +from .transform import Compose + + +@DATASETS.register_module() +class ModelNetDataset(Dataset): + def __init__( + self, + split="train", + data_root="data/modelnet40", + class_names=None, + transform=None, + num_points=8192, + uniform_sampling=True, + save_record=True, + test_mode=False, + test_cfg=None, + loop=1, + if_color=False, + ): + super().__init__() + self.data_root = data_root + self.class_names = dict(zip(class_names, range(len(class_names)))) + self.split = split + self.num_point = num_points + self.uniform_sampling = uniform_sampling + self.transform = Compose(transform) + self.loop = ( + loop if not test_mode else 1 + ) # force make loop = 1 while in test mode + self.if_color = if_color + self.test_mode = test_mode + self.test_cfg = test_cfg if test_mode else None + if test_mode: + self.post_transform = Compose(self.test_cfg.post_transform) + self.aug_transform = [Compose(aug) for aug in self.test_cfg.aug_transform] + + self.data_list = self.get_data_list() + logger = get_root_logger() + logger.info( + "Totally {} x {} samples in {} set.".format( + len(self.data_list), self.loop, split + ) + ) + + # check, prepare record + record_name = f"modelnet40_{self.split}" + if num_points is not None: + record_name += f"_{num_points}points" + if uniform_sampling: + record_name += "_uniform" + record_path = os.path.join(self.data_root, f"{record_name}.pth") + if os.path.isfile(record_path): + logger.info(f"Loading record: {record_name} ...") + self.data = torch.load(record_path, weights_only=False) + else: + logger.info(f"Preparing record: {record_name} ...") + self.data = {} + for idx in range(len(self.data_list)): + data_name = self.data_list[idx] + logger.info(f"Parsing data [{idx}/{len(self.data_list)}]: {data_name}") + self.data[data_name] = self.get_data(idx) + if save_record: + torch.save(self.data, record_path) + + def get_data(self, idx): + data_idx = idx % len(self.data_list) + data_name = self.data_list[data_idx] + if data_name in self.data.keys(): + return copy.deepcopy(self.data[data_name]) + else: + data_shape = "_".join(data_name.split("_")[0:-1]) + data_path = os.path.join( + self.data_root, data_shape, self.data_list[data_idx] + ".txt" + ) + data = np.loadtxt(data_path, delimiter=",").astype(np.float32) + if self.num_point is not None: + if self.uniform_sampling: + with torch.no_grad(): + mask = pointops.farthest_point_sampling( + torch.tensor(data).float().cuda(), + torch.tensor([len(data)]).long().cuda(), + torch.tensor([self.num_point]).long().cuda(), + ) + data = data[mask.cpu()] + else: + data = data[: self.num_point] + coord, normal = data[:, 0:3], data[:, 3:6] + category = np.array([self.class_names[data_shape]]) + if self.if_color: + color = np.zeros_like(coord) + return dict(coord=coord, color=color, normal=normal, category=category) + return dict(coord=coord, normal=normal, category=category) + + def get_data_list(self): + assert isinstance(self.split, str) + split_path = os.path.join( + self.data_root, "modelnet40_{}.txt".format(self.split) + ) + data_list = np.loadtxt(split_path, dtype="str") + return data_list + + def get_data_name(self, idx): + data_idx = idx % len(self.data_list) + return self.data_list[data_idx] + + def __getitem__(self, idx): + if self.test_mode: + return self.prepare_test_data(idx) + else: + return self.prepare_train_data(idx) + + def __len__(self): + return len(self.data_list) * self.loop + + def prepare_train_data(self, idx): + data_dict = self.get_data(idx) + data_dict = self.transform(data_dict) + return data_dict + + def prepare_test_data(self, idx): + assert idx < len(self.data_list) + data_dict = self.get_data(idx) + category = data_dict.pop("category") + data_dict = self.transform(data_dict) + data_dict_list = [] + for aug in self.aug_transform: + data_dict_list.append(aug(copy.deepcopy(data_dict))) + for i in range(len(data_dict_list)): + data_dict_list[i] = self.post_transform(data_dict_list[i]) + data_dict = dict( + voting_list=data_dict_list, + category=category, + name=self.get_data_name(idx), + ) + return data_dict diff --git a/pointcept/datasets/nuscenes.py b/pointcept/datasets/nuscenes.py new file mode 100644 index 0000000000000000000000000000000000000000..1608f59b638f086388e3253ceef04a7d30951a93 --- /dev/null +++ b/pointcept/datasets/nuscenes.py @@ -0,0 +1,583 @@ +""" +nuScenes Dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Zheng Zhang +Please cite our work if the code is helpful to you. +""" + +import os +import numpy as np +from collections.abc import Sequence +import pickle +from PIL import Image +import open3d as o3d +import torch + +from .builder import DATASETS +from .defaults import DefaultDataset, DefaultImagePointDataset + +os.environ["OMP_NUM_THREADS"] = "1" + + +@DATASETS.register_module() +class NuScenesDataset(DefaultDataset): + def __init__(self, sweeps=10, ignore_index=-1, **kwargs): + self.sweeps = sweeps + self.ignore_index = ignore_index + self.learning_map = self.get_learning_map(ignore_index) + super().__init__(ignore_index=ignore_index, **kwargs) + + def get_info_path(self, split): + assert split in ["train", "val", "test"] + if split == "train": + return os.path.join( + self.data_root, "info", f"nuscenes_infos_{self.sweeps}sweeps_train.pkl" + ) + elif split == "val": + return os.path.join( + self.data_root, "info", f"nuscenes_infos_{self.sweeps}sweeps_val.pkl" + ) + elif split == "test": + return os.path.join( + self.data_root, "info", f"nuscenes_infos_{self.sweeps}sweeps_test.pkl" + ) + else: + raise NotImplementedError + + def get_data_list(self): + if isinstance(self.split, str): + info_paths = [self.get_info_path(self.split)] + elif isinstance(self.split, Sequence): + info_paths = [self.get_info_path(s) for s in self.split] + else: + raise NotImplementedError + data_list = [] + for info_path in info_paths: + with open(info_path, "rb") as f: + info = pickle.load(f) + data_list.extend(info) + return data_list + + def get_data(self, idx): + data = self.data_list[idx % len(self.data_list)] + lidar_path = os.path.join(self.data_root, "raw", data["lidar_path"]) + points = np.fromfile(str(lidar_path), dtype=np.float32, count=-1).reshape( + [-1, 5] + ) + coord = points[:, :3] + strength = points[:, 3].reshape([-1, 1]) / 255 # scale strength to [0, 1] + + if "gt_segment_path" in data.keys(): + gt_segment_path = os.path.join( + self.data_root, "raw", data["gt_segment_path"] + ) + segment = np.fromfile( + str(gt_segment_path), dtype=np.uint8, count=-1 + ).reshape([-1]) + segment = np.vectorize(self.learning_map.__getitem__)(segment).astype( + np.int64 + ) + else: + segment = np.ones((points.shape[0],), dtype=np.int64) * self.ignore_index + data_dict = dict( + coord=coord, + strength=strength, + segment=segment, + name=self.get_data_name(idx), + ) + return data_dict + + def get_data_name(self, idx): + # return data name for lidar seg, optimize the code when need to support detection + return self.data_list[idx % len(self.data_list)]["lidar_token"] + + @staticmethod + def get_learning_map(ignore_index): + learning_map = { + 0: ignore_index, + 1: ignore_index, + 2: 6, + 3: 6, + 4: 6, + 5: ignore_index, + 6: 6, + 7: ignore_index, + 8: ignore_index, + 9: 0, + 10: ignore_index, + 11: ignore_index, + 12: 7, + 13: ignore_index, + 14: 1, + 15: 2, + 16: 2, + 17: 3, + 18: 4, + 19: ignore_index, + 20: ignore_index, + 21: 5, + 22: 8, + 23: 9, + 24: 10, + 25: 11, + 26: 12, + 27: 13, + 28: 14, + 29: ignore_index, + 30: 15, + 31: ignore_index, + } + return learning_map + + +@DATASETS.register_module() +class NuScenesColorNormalDataset(NuScenesDataset): + @staticmethod + def estimate_normals(points, center=np.array([0, 0, 0])): + normals = points - center[None, :] + norms = np.linalg.norm(normals, axis=1, keepdims=True) + normals = normals / norms + return normals + + def get_data(self, idx): + data = self.data_list[idx % len(self.data_list)] + lidar_path = os.path.join(self.data_root, "raw", data["lidar_path"]) + points = np.fromfile(str(lidar_path), dtype=np.float32, count=-1).reshape( + [-1, 5] + ) + coord = points[:, :3] + normal = self.estimate_normals(coord) + if "gt_segment_path" in data.keys(): + gt_segment_path = os.path.join( + self.data_root, "raw", data["gt_segment_path"] + ) + segment = np.fromfile( + str(gt_segment_path), dtype=np.uint8, count=-1 + ).reshape([-1]) + segment = np.vectorize(self.learning_map.__getitem__)(segment).astype( + np.int64 + ) + else: + segment = np.ones((points.shape[0],), dtype=np.int64) * self.ignore_index + data_dict = dict( + coord=coord, + color=np.zeros_like(coord), # placeholder for color + normal=np.zeros_like(coord), # placeholder for normal + segment=segment, + name=self.get_data_name(idx), + ) + return data_dict + + +@DATASETS.register_module() +class NuScenesImagePointDataset(DefaultImagePointDataset): + CAMERA_TYPES = [ + "CAM_FRONT", + "CAM_FRONT_RIGHT", + "CAM_FRONT_LEFT", + "CAM_BACK", + "CAM_BACK_LEFT", + "CAM_BACK_RIGHT", + ] + + def __init__( + self, + if_img=False, + if_sweep=False, + sweeps_max=10, + sweeps=10, + sweep_gap=1, + ignore_index=-1, + img_num=4, + **kwargs, + ): + self.sweeps = sweeps + self.sweep_gap = sweep_gap + self.sweeps_max = sweeps_max + self.if_sweep = if_sweep + self.if_img = if_img + self.ignore_index = ignore_index + self.learning_map = self.get_learning_map(ignore_index) + frame_count = sweeps + 1 if if_sweep else 1 + self.img_ratio = img_num / (6 * frame_count) + super().__init__(ignore_index=ignore_index, if_img=if_img, **kwargs) + + @staticmethod + def project_lidar_to_image_with_color( + lidar_points, # shape: (N, 3) or (N, 4) + image, # shape: (H, W, 3), uint8 RGB + cam_intrinsic, # shape: (3, 3) + lidar_to_cam, # shape: (4, 4) + lidar_colors, + ): + """ + Projects LiDAR points to the image, fetches pixel color and pixel coordinates. + Returns: + filtered_points: (M, 3) - 3D points in camera frame that project onto the image. + colors: (M, 3) - RGB colors at projected 2D locations. + uv_coords: (M, 2) - Integer pixel coordinates (u, v) on the image. + mask: (N,) - (optional) Boolean mask indicating which lidar points are used. + """ + lidar_uv_coords = np.full( + (lidar_points.shape[0], 2), -1, dtype=int + ) # Default to (-1, -1) + lidar_points_coord = lidar_points[:, :3] + ones = np.ones((lidar_points_coord.shape[0], 1)) + lidar_hom = np.concatenate([lidar_points_coord, ones], axis=1) # (N, 4) + + points_cam = (lidar_to_cam @ lidar_hom.T).T # (N, 4) + + valid = points_cam[:, 2] > 0 + points_cam = points_cam[valid] + + pts_2d = (cam_intrinsic @ points_cam[:, :3].T).T # (N, 3) + pts_2d = pts_2d[:, :2] / pts_2d[:, 2:3] # (N, 2) - pixel (u, v) + + H, W = image.shape[:2] + u, v = pts_2d[:, 0], pts_2d[:, 1] + inside = (u >= 0) & (u < W) & (v >= 0) & (v < H) + + u = u[inside].astype(int) + v = v[inside].astype(int) + + mask = np.zeros(lidar_points.shape[0], dtype=bool) + mask[np.where(valid)[0][inside]] = True + + lidar_colors[mask] = image[v, u, :] + lidar_uv_coords[mask] = np.stack([u, v], axis=1) # (M, 2) + return lidar_colors, lidar_uv_coords, mask + + @staticmethod + def transform_normal(normal, transform): + return (transform[:3, :3] @ normal.T).T + + def get_info_path(self, split): + assert split in ["train", "val", "test"] + if split == "train": + return os.path.join( + self.data_root, + "info", + f"nuscenes_infos_{self.sweeps_max}sweeps_train.pkl", + ) + elif split == "val": + return os.path.join( + self.data_root, + "info", + f"nuscenes_infos_{self.sweeps_max}sweeps_val.pkl", + ) + elif split == "test": + return os.path.join( + self.data_root, + "info", + f"nuscenes_infos_{self.sweeps_max}sweeps_test.pkl", + ) + else: + raise NotImplementedError + + def get_data_list(self): + split_list = {} + if isinstance(self.split, str): + info_paths = [self.get_info_path(self.split)] + split = [self.split] + elif isinstance(self.split, Sequence): + split = self.split + info_paths = [self.get_info_path(s) for s in self.split] + else: + raise NotImplementedError + data_list = [] + for info_path, split_i in zip(info_paths, split): + with open(info_path, "rb") as f: + info = pickle.load(f) + data_list.extend(info) + split_list[split_i] = list([i["token"] for i in info]) + return data_list, split_list + + def get_data(self, idx): + data = self.data_list[idx % len(self.data_list)] + lidar_path = os.path.join(self.data_root, "raw", data["lidar_path"]) + points = np.fromfile(str(lidar_path), dtype=np.float32, count=-1).reshape( + [-1, 5] + ) + keyframe_point_count = points.shape[0] + + imgs = [] + cam_coords = [] + cam_normals = [] + cam_colors = [] + cam_strengths = [] + cam_correspondences = [] + correspondence_start = 0 + frame_pcd_offset = [] + lidar_colors = np.zeros((points.shape[0], 3), dtype=int) # Default to black + for id, cam_name in enumerate(self.CAMERA_TYPES): + cam_info = data["cams"][cam_name] + cam_intrinsic = cam_info["camera_intrinsics"] + cam_image = Image.open( + os.path.join(self.data_root, "raw", data["cams"][cam_name]["data_path"]) + ) + cam_image_np = np.array(cam_image) + sensor2lidar = np.eye(4) + sensor2lidar[:3, :3] = cam_info["sensor2lidar_rotation"] + sensor2lidar[:3, 3] = cam_info["sensor2lidar_translation"] + lidar2sensor = np.linalg.inv(sensor2lidar) + lidar_colors, correspondence_info, _ = ( + self.project_lidar_to_image_with_color( + points, cam_image_np, cam_intrinsic, lidar2sensor, lidar_colors + ) + ) + correspondence_point_id = ( + np.array(range(correspondence_info.shape[0])).reshape((-1, 1)) + + correspondence_start + ) + correspondence_info = np.hstack( + [correspondence_info, correspondence_point_id] + ) + if np.random.rand() < self.img_ratio: + cam_correspondences.append(correspondence_info) + imgs.append(cam_image) + correspondence_start += points.shape[0] + cam_coord = points[:, :3] + cam_center = np.array([0, 0, 0]) + cam_normal = self.get_normals(cam_center, cam_coord) + cam_normals.append(cam_normal) + cam_strength = points[:, 3].reshape([-1, 1]) / 255 + cam_coords.append(cam_coord) + cam_colors.append(lidar_colors) + cam_strengths.append(cam_strength) + + if self.if_sweep: + frame_pcd_offset.append(points.shape[0]) + for id, sweep in enumerate( + data["sweeps"][: (self.sweep_gap * self.sweeps) : self.sweep_gap] + ): + lidar_path = os.path.join(self.data_root, "raw", sweep["lidar_path"]) + points = np.fromfile( + str(lidar_path), dtype=np.float32, count=-1 + ).reshape([-1, 5]) + lidar_colors = np.zeros( + (points.shape[0], 3), dtype=int + ) # Default to black + cam_lidar_tm = ( + sweep["transform_matrix"] + if sweep["transform_matrix"] is not None + else np.eye(4) + ) + for id, cam_name in enumerate(self.CAMERA_TYPES): + cam_info = sweep["cams"][cam_name] + # cam_image_np = np.array(imgs[id]) + cam_intrinsic = cam_info["camera_intrinsics"] + cam_image = Image.open( + os.path.join(self.data_root, "raw", cam_info["data_path"]) + ) + cam_image_np = np.array(cam_image) + sensor2lidar = np.eye(4) + sensor2lidar[:3, :3] = cam_info["sensor2lidar_rotation"] + sensor2lidar[:3, 3] = cam_info["sensor2lidar_translation"] + lidar2sensor = np.linalg.inv(sensor2lidar) @ cam_lidar_tm + lidar_colors, correspondence_info, _ = ( + self.project_lidar_to_image_with_color( + points, + cam_image_np, + cam_intrinsic, + lidar2sensor, + lidar_colors, + ) + ) + correspondence_point_id = ( + np.array(range(correspondence_info.shape[0])).reshape((-1, 1)) + + correspondence_start + ) + correspondence_info = np.hstack( + [correspondence_info, correspondence_point_id] + ) + if np.random.rand() < self.img_ratio: + cam_correspondences.append(correspondence_info) + imgs.append(cam_image) + correspondence_start += correspondence_info.shape[0] + frame_pcd_offset.append(correspondence_start) + cam_coord = points[:, :3] + cam_center = np.array([0, 0, 0]) + cam_normal = self.get_normals(cam_center, cam_coord) + cam_normal = self.transform_normal(cam_normal, cam_lidar_tm) + cam_normals.append(cam_normal) + ones = np.ones((points.shape[0], 1)) + cam_coord_hom = np.concatenate([cam_coord, ones], axis=1) # (N, 4) + cam_coord = cam_coord_hom @ cam_lidar_tm.T + cam_coord = cam_coord[:, :3] + cam_strength = points[:, 3].reshape([-1, 1]) / 255 + cam_coords.append(cam_coord) + cam_colors.append(lidar_colors) + cam_strengths.append(cam_strength) + + coord = np.vstack(cam_coords) + color = np.vstack(cam_colors) + normal = np.vstack(cam_normals) + strength = np.vstack(cam_strengths) + frame_pcd_offset = np.array(frame_pcd_offset) + + car_from_ref = np.linalg.inv(data["ref_from_car"]) + coord_homo = np.hstack((coord, np.ones((coord.shape[0], 1)))) + coord_homo = coord_homo @ car_from_ref.T + coord = coord_homo[:, :3] + normal = self.transform_normal(normal, car_from_ref) + + img_assets = dict() + if self.if_img: + if len(imgs) > 0: + img_width, img_height = imgs[0].size + div_w = img_width // self.patch_w + div_h = img_height // self.patch_h + div_min = max(min(div_w, div_h), 1) + crop_img_width = div_min * self.patch_w + crop_img_height = div_min * self.patch_h + left = int((img_width - crop_img_width) / 2) + top = int((img_height - crop_img_height) / 2) + right = int((img_width + crop_img_width) / 2) + bottom = int((img_height + crop_img_height) / 2) + imgs = [img.crop((left, top, right, bottom)) for img in imgs] + imgs = [self.transform_img(img) for img in imgs] + imgs_list = torch.stack(imgs) + img_assets["images"] = imgs_list.float() + else: + img_assets["images"] = torch.empty( + ( + 0, + 3, + self.patch_h * self.patch_size, + self.patch_w * self.patch_size, + ) + ) + img_assets["img_num"] = np.array( + [img_assets["images"].shape[0]], dtype=np.int32 + ) + + correspondence_infos = np.ones( + (coord.shape[0], len(cam_correspondences), 2), dtype=np.int32 + ) * (-1) + for id, correspondence_info in enumerate(cam_correspondences): + correspondence_info = self.resize_correspondence_info( + correspondence_info, + (self.patch_h * self.patch_size, self.patch_w * self.patch_size), + (img_height, img_width), + (left, top, right, bottom), + self.patch_size, + ) + correspondence_infos[correspondence_info[:, -1], id, :] = ( + correspondence_info[:, :-1] + ) + img_assets["correspondence"] = correspondence_infos + if "gt_segment_path" in data.keys(): + gt_segment_path = os.path.join( + self.data_root, "raw", data["gt_segment_path"] + ) + segment = np.fromfile( + str(gt_segment_path), dtype=np.uint8, count=-1 + ).reshape([-1]) + segment = np.vectorize(self.learning_map.__getitem__)(segment).astype( + np.int64 + ) + else: + segment = ( + np.ones((keyframe_point_count,), dtype=np.int64) * self.ignore_index + ) + + if segment.shape[0] != keyframe_point_count: + raise ValueError( + f"nuScenes sample {self.get_data_name(idx)} has " + f"{keyframe_point_count} keyframe points but " + f"{segment.shape[0]} segment labels." + ) + + if self.if_sweep: + sweep_point_count = coord.shape[0] - keyframe_point_count + sweep_segment = np.full( + (sweep_point_count,), self.ignore_index, dtype=segment.dtype + ) + segment = np.concatenate((segment, sweep_segment), axis=0) + + color = color.astype(np.float32) + if self.if_sweep: + data_dict = dict( + coord=coord, + color=color, + normal=normal, + strength=strength, + segment=segment, + frame_pcd_offset=frame_pcd_offset, + name=self.get_data_name(idx), + ) + else: + data_dict = dict( + coord=coord, + color=color, + normal=normal, + strength=strength, + segment=segment, + name=self.get_data_name(idx), + ) + data_dict.update(img_assets) + return data_dict + + def get_data_name(self, idx): + return self.data_list[idx % len(self.data_list)]["lidar_token"] + + @staticmethod + def get_normals(cam_center, coords): + Cs = np.repeat(cam_center.reshape((1, -1)), coords.shape[0], axis=0) + view_dirs = coords - Cs + view_dirs = view_dirs / np.linalg.norm(view_dirs, axis=-1, keepdims=True) + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coords) + pcd.estimate_normals() + normals = np.asarray(pcd.normals) + dot_product = np.sum(normals * view_dirs, axis=-1) + flip_mask = dot_product > 0 + normals[flip_mask] = -normals[flip_mask] + normals = normals / np.linalg.norm(normals, axis=-1, keepdims=True) + return normals + + # or using the vector from LiDAR to points, which is faster for data loading + # @staticmethod + # def get_normals(center, points): + # normals = points - center[None, :] + # norms = np.linalg.norm(normals, axis=1, keepdims=True) + # normals = normals / norms + # return normals + + @staticmethod + def get_learning_map(ignore_index): + learning_map = { + 0: ignore_index, + 1: ignore_index, + 2: 6, + 3: 6, + 4: 6, + 5: ignore_index, + 6: 6, + 7: ignore_index, + 8: ignore_index, + 9: 0, + 10: ignore_index, + 11: ignore_index, + 12: 7, + 13: ignore_index, + 14: 1, + 15: 2, + 16: 2, + 17: 3, + 18: 4, + 19: ignore_index, + 20: ignore_index, + 21: 5, + 22: 8, + 23: 9, + 24: 10, + 25: 11, + 26: 12, + 27: 13, + 28: 14, + 29: ignore_index, + 30: 15, + 31: ignore_index, + } + return learning_map diff --git a/pointcept/datasets/partnet.py b/pointcept/datasets/partnet.py new file mode 100644 index 0000000000000000000000000000000000000000..0ab0c7881689f35d336979b718d500b7d858c1ed --- /dev/null +++ b/pointcept/datasets/partnet.py @@ -0,0 +1,79 @@ +""" +HK Remote Data Dataset + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import numpy as np +import open3d as o3d +import torch + +from .defaults import DefaultDataset +from .builder import DATASETS +from pointcept.utils.cache import shared_dict + + +@DATASETS.register_module() +class PartNetDataDataset(DefaultDataset): + def __init__(self, if_img=True, crop_h=630, crop_w=1120, patch_size=14, **kwargs): + self.crop_h = crop_h + self.crop_w = crop_w + self.patch_size = patch_size + self.patch_h = crop_h // patch_size + self.patch_w = crop_w // patch_size + self.if_img = if_img + super().__init__(**kwargs) + + def get_data(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + pc_data_path = os.path.join( + data_path, "point_sample", "sample-points-all-pts-nor-rgba-10000.ply" + ) + name = self.get_data_name(idx) + split = self.get_split_name(idx) + if self.cache: + cache_name = f"pointcept-{name}" + return shared_dict(cache_name) + + data_dict = {} + pc = o3d.io.read_point_cloud(pc_data_path) + data_dict["coord"] = np.asarray(pc.points) + data_dict["color"] = np.asarray(pc.colors) + data_dict["normal"] = np.asarray(pc.normals) + data_dict["name"] = name + data_dict["split"] = split + + if self.if_img: + data_dict["images"] = torch.empty( + (0, 3, self.patch_h * self.patch_size, self.patch_w * self.patch_size) + ) + data_dict["img_num"] = np.array([0], dtype=np.int32) + data_dict["correspondence"] = np.ones( + (data_dict["coord"].shape[0], 0, 2), dtype=np.float32 + ) * (-1) + + if "coord" in data_dict.keys(): + data_dict["coord"] = data_dict["coord"].astype(np.float32) + + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"].astype(np.float32) + + if "normal" in data_dict.keys(): + data_dict["normal"] = data_dict["normal"].astype(np.float32) + + if "segment" in data_dict.keys(): + data_dict["segment"] = data_dict["segment"].reshape([-1]).astype(np.int32) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = data_dict["instance"].reshape([-1]).astype(np.int32) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + return data_dict diff --git a/pointcept/datasets/partnete.py b/pointcept/datasets/partnete.py new file mode 100644 index 0000000000000000000000000000000000000000..23e19de09475854be573e11e4ce2827bd5c699ec --- /dev/null +++ b/pointcept/datasets/partnete.py @@ -0,0 +1,111 @@ +""" +PartNetE Datasets + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import glob +from collections.abc import Sequence +import numpy as np +import json + +from .defaults import DefaultDataset +from .builder import DATASETS + +from pointcept.utils.cache import shared_dict + + +@DATASETS.register_module() +class PartNetEDataset(DefaultDataset): + def __init__( + self, class_names, num_parts, data_root, meta_path, split, *args, **kwargs + ): + self.categories = class_names + self.num_parts = num_parts + self.num_part_offset = np.concatenate(([0], np.cumsum(self.num_parts))) + with open(meta_path, "r", encoding="utf-8") as file: + meta_data = json.load(file) + self.category2part = {} + self.parts = [] + for class_id, class_name in enumerate(self.categories): + self.category2part[class_name] = self.num_part_offset[class_id] + list( + range(self.num_parts[class_id]) + ) + category_part_name = [ + class_name + "_" + part_name + for part_name in ["other"] + meta_data[class_name] + ] + self.parts.extend(category_part_name) + super().__init__(data_root=data_root, split=split, *args, **kwargs) + + def get_data_list(self): + if isinstance(self.split, str): + data_list = glob.glob(os.path.join(self.data_root, self.split, "*/*")) + elif isinstance(self.split, Sequence): + data_list = [] + for split in self.split: + data_list += glob.glob(os.path.join(self.data_root, split, "*/*")) + else: + raise NotImplementedError + return data_list + + def get_data_name(self, idx): + file_path = self.data_list[idx % len(self.data_list)] + dir_path, id_name = os.path.split(file_path) + object_name = os.path.basename(dir_path) + data_name = f"{object_name}_{id_name}" + return data_name + + def get_data(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + name = self.get_data_name(idx) + split = self.get_split_name(idx) + if self.cache: + cache_name = f"pointcept-{name}" + return shared_dict(cache_name) + + data_dict = {} + assets = os.listdir(data_path) + for asset in assets: + if not asset.endswith(".npy"): + continue + if asset[:-4] not in self.VALID_ASSETS: + continue + data_dict[asset[:-4]] = np.load(os.path.join(data_path, asset)) + data_dict["name"] = name + data_dict["split"] = split + + object_name = name.split("_")[0] + cls_token = self.categories.index(object_name) + data_dict["cls_token"] = cls_token + + if "coord" in data_dict.keys(): + data_dict["coord"] = data_dict["coord"].astype(np.float32) + + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"].astype(np.float32) + + if "normal" in data_dict.keys(): + data_dict["normal"] = data_dict["normal"].astype(np.float32) + + if "segment" in data_dict.keys(): + data_dict["segment"] = ( + data_dict["segment"].reshape([-1]).astype(np.int32) + + self.num_part_offset[cls_token] + + 1 + ) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = data_dict["instance"].reshape([-1]).astype(np.int32) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + return data_dict diff --git a/pointcept/datasets/preprocessing/arkitscenes/preprocess_arkitscenes_mesh.py b/pointcept/datasets/preprocessing/arkitscenes/preprocess_arkitscenes_mesh.py new file mode 100644 index 0000000000000000000000000000000000000000..bf0fe79b061b2ce298dce51c0241f4395be83f69 --- /dev/null +++ b/pointcept/datasets/preprocessing/arkitscenes/preprocess_arkitscenes_mesh.py @@ -0,0 +1,93 @@ +""" +Preprocessing ArkitScenes +""" + +import os +import argparse +import glob +import plyfile +import numpy as np +import pandas as pd +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat +from pathlib import Path + + +def read_plymesh(filepath): + """Read ply file and return it as numpy array. Returns None if emtpy.""" + with open(filepath, "rb") as f: + plydata = plyfile.PlyData.read(f) + if plydata.elements: + vertices = pd.DataFrame(plydata["vertex"].data).values + faces = np.stack(plydata["face"].data["vertex_indices"], axis=0) + return vertices, faces + + +def face_normal(vertex, face): + v01 = vertex[face[:, 1]] - vertex[face[:, 0]] + v02 = vertex[face[:, 2]] - vertex[face[:, 0]] + vec = np.cross(v01, v02) + length = np.sqrt(np.sum(vec**2, axis=1, keepdims=True)) + 1.0e-8 + nf = vec / length + area = length * 0.5 + return nf, area + + +def vertex_normal(vertex, face): + nf, area = face_normal(vertex, face) + nf = nf * area + + nv = np.zeros_like(vertex) + for i in range(face.shape[0]): + nv[face[i]] += nf[i] + + length = np.sqrt(np.sum(nv**2, axis=1, keepdims=True)) + 1.0e-8 + nv = nv / length + return nv + + +def parse_scene(scene_path, output_dir): + print(f"Parsing scene {scene_path}") + split = os.path.basename(os.path.dirname(os.path.dirname(scene_path))) + scene_id = os.path.basename(os.path.dirname(scene_path)) + vertices, faces = read_plymesh(scene_path) + coords = vertices[:, :3] + colors = vertices[:, 3:6] + normals = vertex_normal(coords, faces) + data_dict = dict(coord=coords, color=colors, normal=normals) + os.makedirs(output_dir / split / scene_id, exist_ok=True) + for key in data_dict.keys(): + np.save(output_dir / split / scene_id / f"{key}.npy", data_dict[key]) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ArkitScenes dataset containing 3dod folder", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + opt = parser.parse_args() + # Create output directories + train_output_dir = os.path.join(opt.output_root, "Training") + os.makedirs(train_output_dir, exist_ok=True) + val_output_dir = os.path.join(opt.output_root, "Validation") + os.makedirs(val_output_dir, exist_ok=True) + # Load scene paths + scene_paths = sorted(glob.glob(opt.dataset_root + "/3dod/*/*/*_mesh.ply")) + # Preprocess data. + pool = ProcessPoolExecutor(max_workers=opt.num_workers) + print("Processing scenes...") + _ = list(pool.map(parse_scene, scene_paths, repeat(Path(opt.output_root)))) diff --git a/pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.py b/pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.py new file mode 100644 index 0000000000000000000000000000000000000000..08c7040a2eab44950f12f1fb625695ef1338725b --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.py @@ -0,0 +1,343 @@ +""" +Preprocessing Script for ARKitScenes + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from pathlib import Path +import argparse +import os +import pandas as pd +import camtools as ct +import numpy as np +from scipy.spatial import cKDTree +import shutil +import open3d as o3d +import cv2 +import glob +import numpy as np +import os +import plyfile +import multiprocessing as mp +from rotation import convert_angle_axis_to_matrix3 + + +def correspondenceGet(mesh, K, T, img_size, coords_gt): + height, width = img_size + if np.isnan(T).any() or np.isinf(T).any(): + return None + depth = ct.raycast.mesh_to_im_depth( + mesh=mesh, K=K, T=np.linalg.inv(T), height=height, width=width + ) + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + depth = depth.reshape((-1, 1)) + valid = ~np.isinf(depth).squeeze(-1) + coord = np.zeros_like(pixel, dtype=np.float32) + coord[valid] = depth[valid] * (np.linalg.inv(K) @ pixel[valid].T).T # coord_camera + coord[valid] = coord[valid] @ T[:3, :3].T + T[:3, 3] # column then row + pixel = pixel[valid] + coord = coord[valid] + if coord.shape[0] == 0: + return None + pixel = pixel[:, :2] + coord_dict = {"pixel": pixel, "coord": coord} + return coord_dict + + +def correspondenceSave(mesh, scene_dir, coords_gt, output_dir, img_size, Ks, Ts): + os.makedirs(output_dir, exist_ok=True) + scene_dir = Path(scene_dir) + index_gt = [ + img_name[:-4].split("_")[1] + for img_name in os.listdir(str(scene_dir / "color")) + if img_name.endswith(".png") + ] + index_gt = sorted(index_gt, key=lambda x: float(x)) + + coords_gt_ = coords_gt + pixels_ = [] + coords_ = [] + + for i, (K, T) in enumerate(zip(Ks, Ts)): + coord_dict = correspondenceGet(mesh, K, T, img_size, coords_gt) + if coord_dict is None: + correspondences = -np.ones((1, 3)) + else: + pixels_ = coord_dict["pixel"] + coords_ = coord_dict["coord"] + tree = cKDTree(coords_gt_) + dis, idx = tree.query(coords_, k=1) + idx_valid = idx[dis < 0.01] + pixels_valid = pixels_[dis < 0.01] + correspondences = np.hstack((pixels_valid, idx_valid.reshape(-1, 1))) + np.save(Path(output_dir) / "{}.npy".format(index_gt[i]), correspondences) + + +def read_plymesh(filepath): + """Read ply file and return it as numpy array. Returns None if emtpy.""" + with open(filepath, "rb") as f: + plydata = plyfile.PlyData.read(f) + if plydata.elements: + vertices = pd.DataFrame(plydata["vertex"].data).values + faces = np.stack(plydata["face"].data["vertex_indices"], axis=0) + return vertices, faces + + +def face_normal(vertex, face): + v01 = vertex[face[:, 1]] - vertex[face[:, 0]] + v02 = vertex[face[:, 2]] - vertex[face[:, 0]] + vec = np.cross(v01, v02) + length = np.sqrt(np.sum(vec**2, axis=1, keepdims=True)) + 1.0e-8 + nf = vec / length + area = length * 0.5 + return nf, area + + +def vertex_normal(vertex, face): + nf, area = face_normal(vertex, face) + nf = nf * area + + nv = np.zeros_like(vertex) + for i in range(face.shape[0]): + nv[face[i]] += nf[i] + + length = np.sqrt(np.sum(nv**2, axis=1, keepdims=True)) + 1.0e-8 + nv = nv / length + return nv + + +def TrajStringToMatrix(traj_str): + """convert traj_str into translation and rotation matrices + Args: + traj_str: A space-delimited file where each line represents a camera position at a particular timestamp. + The file has seven columns: + * Column 1: timestamp + * Columns 2-4: rotation (axis-angle representation in radians) + * Columns 5-7: translation (usually in meters) + + Returns: + ts: translation matrix + Rt: rotation matrix + """ + # line=[float(x) for x in traj_str.split()] + # ts = line[0]; + # R = cv2.Rodrigues(np.array(line[1:4]))[0]; + # t = np.array(line[4:7]); + # Rt = np.concatenate((np.concatenate((R, t[:,np.newaxis]), axis=1), [[0.0,0.0,0.0,1.0]]), axis=0) + tokens = traj_str.split() + assert len(tokens) == 7 + ts = tokens[0] + # Rotation in angle axis + angle_axis = [float(tokens[1]), float(tokens[2]), float(tokens[3])] + r_w_to_p = convert_angle_axis_to_matrix3(np.asarray(angle_axis)) + # Translation + t_w_to_p = np.asarray([float(tokens[4]), float(tokens[5]), float(tokens[6])]) + extrinsics = np.eye(4, 4) + extrinsics[:3, :3] = r_w_to_p + extrinsics[:3, -1] = t_w_to_p + Rt = np.linalg.inv(extrinsics) + return (ts, Rt) + + +def st2_camera_intrinsics(filename): + w, h, fx, fy, hw, hh = np.loadtxt(filename) + return np.asarray([[fx, 0, hw], [0, fy, hh], [0, 0, 1]]) + + +def handle_process( + scene_path, + output_path, + pointclouds_root, + frame_gap, + parse_pointclouds, + parse_depths, +): + print("loading from ", scene_path) + split = os.path.basename(os.path.dirname(scene_path)) + scene_id = os.path.basename(scene_path) + img_folder = os.path.join(scene_path, f"{scene_id}_frames", "lowres_wide") + traj_file = os.path.join(scene_path, f"{scene_id}_frames", "lowres_wide.traj") + pc_output_path = os.path.join(output_path, split, f"{scene_id}") + pc_input_path = os.path.join(pointclouds_root, split, f"{scene_id}") + im_output_path = os.path.join(output_path, "images", split, f"{scene_id}") + os.makedirs(im_output_path, exist_ok=True) + if not os.path.exists(img_folder): + frame_ids = [] + else: + images = sorted( + glob.glob(os.path.join(img_folder, "*.png")), + key=lambda x: float(x.split("/")[-1].split("_")[1][:-4]), + ) + frame_ids = [os.path.basename(x) for x in images] + frame_ids = [x.split(".png")[0].split("_")[1] for x in frame_ids] + video_id = img_folder.split("/")[-2].split("_frames")[0] + frame_ids.sort() + frame_ids = frame_ids[::frame_gap] + if len(frame_ids) > 0: + height, width = cv2.imread(images[0]).shape[:2] + if os.path.exists(traj_file): + with open(traj_file) as f: + traj = f.readlines() + # convert traj to json dict + poses_from_traj = {} + for line in traj: + traj_timestamp = line.split(" ")[0] + poses_from_traj[f"{round(float(traj_timestamp), 3):.3f}"] = ( + TrajStringToMatrix(line)[1].tolist() + ) + + if os.path.exists(traj_file): + poses = poses_from_traj + else: + poses = {} + + # get img dlc + poses_keys = list(poses.keys()) + intrinsics = [] + poses_list = [] + Ks_path = os.path.join(im_output_path, "intrinsic") + Ts_path = os.path.join(im_output_path, "pose") + img_path = os.path.join(im_output_path, "color") + + os.makedirs(Ks_path, exist_ok=True) + os.makedirs(Ts_path, exist_ok=True) + os.makedirs(img_path, exist_ok=True) + if parse_depths: + depth_path = os.path.join(im_output_path, "depth") + os.makedirs(depth_path, exist_ok=True) + for frame_id in frame_ids: + shutil.copy( + os.path.join( + scene_path, + f"{scene_id}_frames", + "lowres_wide", + f"{video_id}_{frame_id}.png", + ), + img_path, + ) + if parse_depths: + shutil.copy( + os.path.join( + scene_path, + f"{scene_id}_frames", + "lowres_depth", + f"{video_id}_{frame_id}.png", + ), + depth_path, + ) + intrinsic_fn = os.path.join( + scene_path, + f"{scene_id}_frames", + "lowres_wide_intrinsics", + f"{video_id}_{frame_id}.pincam", + ) + intrinsic = st2_camera_intrinsics(intrinsic_fn) + intrinsics.append(intrinsic) + np.save(os.path.join(Ks_path, f"{video_id}_{frame_id}.npy"), intrinsic) + frame_id = f"{round(float(frame_id), 3):.3f}" + closest_key = min( + poses_keys, key=lambda k: abs(float(k) - float(frame_id)) + ) + np.save( + os.path.join(Ts_path, f"{video_id}_{frame_id}.npy"), + poses[closest_key], + ) + poses_list.append(poses[closest_key]) + # match extrinsics + intrinsics = np.stack(intrinsics) + poses = np.stack(poses_list) + + ply_path = os.path.join(scene_path, f"{scene_id}_3dod_mesh.ply") + if parse_pointclouds: + vertices, faces = read_plymesh(ply_path) + coords = vertices[:, :3] + colors = vertices[:, 3:6] + normals = vertex_normal(coords, faces) + data_dict = dict(coord=coords, color=colors, normal=normals) + os.makedirs(pc_output_path, exist_ok=True) + for key in data_dict.keys(): + np.save(os.path.join(pc_output_path, f"{key}.npy"), data_dict[key]) + else: + coords = np.load(os.path.join(pc_input_path, "coord.npy")) + + # Save img data + mesh = o3d.io.read_triangle_mesh(ply_path) + correspondenceSave( + mesh, + im_output_path, + coords, + os.path.join(im_output_path, "correspondence"), + (height, width), + intrinsics, + poses, + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ArkitScenes dataset containing 3dod folder", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--pointclouds_root", + default="data/arkitscenes", + type=str, + help="Input path where previous pointclouds folder located", + ) + parser.add_argument( + "--frame_gap", default=50, type=int, help="Frame gap for processing" + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + parser.add_argument( + "--thread_id", + default=0, + type=int, + help="thread_id", + ) + parser.add_argument( + "--parse_pointclouds", action="store_true", help="Whether parse point clouds" + ) + parser.add_argument( + "--parse_depths", action="store_true", help="Whether parse depths" + ) + opt = parser.parse_args() + # Load scene paths + metadata_path = f"{opt.dataset_root}/3dod/metadata.csv" + metadata = pd.read_csv(metadata_path) + # Initialize the splits dictionary + splits = {} + scene_paths = [] + # Populate the splits dictionary + for index, row in metadata.iterrows(): + scene_id = str(row["video_id"]) + split = row["fold"] + splits[scene_id] = split + scene_paths.append(os.path.join(opt.dataset_root, "3dod", split, scene_id)) + # Preprocess data. + scene_paths_list = np.array_split(scene_paths, opt.num_workers) + scene_paths_ = scene_paths_list[opt.thread_id] + print("Processing scenes...") + for scene_path in scene_paths_: + handle_process( + scene_path, + opt.output_root, + opt.pointclouds_root, + opt.frame_gap, + opt.parse_pointclouds, + opt.parse_depths, + ) diff --git a/pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.sh b/pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.sh new file mode 100644 index 0000000000000000000000000000000000000000..f1a2cdd3f69cae0da2b0edad9fc33b8687dd370e --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.sh @@ -0,0 +1,42 @@ +#!/bin/bash + +dataset_root="" +output_root="" +num_workers=16 +parse_depths=false +parse_pointclouds=false + +while getopts "d:o:n:pc" opt; do + case $opt in + d) dataset_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + p) parse_depths=true ;; + c) parse_pointclouds=true ;; + *) echo "Usage: $0 -d -o [-n ] [-p] [-c]"; exit 1 ;; + esac +done + +if [ -z "$dataset_root" ] || [ -z "$output_root" ]; then + echo "Usage: $0 -d -o [-n ] [-p] [-c]" + exit 1 +fi + +for i in $(seq 0 $((num_workers - 1))); do + cmd="python pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.py --thread_id $i \ + --num_workers $num_workers \ + --dataset_root $dataset_root \ + --output_root $output_root" + + if $parse_depths; then + cmd="$cmd --parse_depths" + fi + + if $parse_pointclouds; then + cmd="$cmd --parse_pointclouds" + fi + + eval "$cmd &" +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/arkitscenes/rotation.py b/pointcept/datasets/preprocessing/concerto/arkitscenes/rotation.py new file mode 100644 index 0000000000000000000000000000000000000000..2cbc9a467bcf9bba461bdc5187bd38c8c6f0540d --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/arkitscenes/rotation.py @@ -0,0 +1,140 @@ +import cv2 +import math +import numpy as np + + +def eulerAnglesToRotationMatrix(theta): + """Euler rotation matrix with clockwise logic. + Rotation + + Args: + theta: list of float + [theta_x, theta_y, theta_z] + Returns: + R: np.array (3, 3) + rotation matrix of Rz*Ry*Rx + """ + R_x = np.array( + [ + [1, 0, 0], + [0, math.cos(theta[0]), -math.sin(theta[0])], + [0, math.sin(theta[0]), math.cos(theta[0])], + ] + ) + + R_y = np.array( + [ + [math.cos(theta[1]), 0, math.sin(theta[1])], + [0, 1, 0], + [-math.sin(theta[1]), 0, math.cos(theta[1])], + ] + ) + + R_z = np.array( + [ + [math.cos(theta[2]), -math.sin(theta[2]), 0], + [math.sin(theta[2]), math.cos(theta[2]), 0], + [0, 0, 1], + ] + ) + + R = np.dot(R_z, np.dot(R_y, R_x)) + return R + + +def upright_camera_relative_transform(pose): + """Generate pose matrix with z-dim as height + + Args: + pose: np.array (4, 4) + Returns: + urc: (4, 4) + urc_inv: (4, 4) + """ + + # take viewing direction in camera local coordiantes (which is simply unit vector along +z) + view_dir_camera = np.asarray([0, 0, 1]) + R = pose[0:3, 0:3] + t = pose[0:3, 3] + + # convert to world coordinates + view_dir_world = np.dot(R, view_dir_camera) + + # compute heading + view_dir_xy = view_dir_world[0:2] + heading = math.atan2(view_dir_xy[1], view_dir_xy[0]) + + # compute rotation around Z to align heading with +Y + zRot = -heading + math.pi / 2 + + # translation first, back to camera point + urc_t = np.identity(4) + urc_t[0:2, 3] = -1 * t[0:2] + + # compute rotation matrix + urc_r = np.identity(4) + urc_r[0:3, 0:3] = eulerAnglesToRotationMatrix([0, 0, zRot]) + + urc = np.dot(urc_r, urc_t) + urc_inv = np.linalg.inv(urc) + + return urc, urc_inv + + +def rotate_pc(pc, rotmat): + """Rotation points w.r.t. rotmat + Args: + pc: np.array (n, 3) + rotmat: np.array (4, 4) + Returns: + pc: (n, 3) + """ + pc_4 = np.ones([pc.shape[0], 4]) + pc_4[:, 0:3] = pc + pc_4 = np.dot(pc_4, np.transpose(rotmat)) + + return pc_4[:, 0:3] + + +def rotate_points_along_z(points, angle): + """Rotation clockwise + Args: + points: np.array of np.array (B, N, 3 + C) or + (N, 3 + C) for single batch + angle: np.array of np.array (B, ) + or (, ) for single batch + angle along z-axis, angle increases x ==> y + Returns: + points_rot: (B, N, 3 + C) or (N, 3 + C) + + """ + single_batch = len(points.shape) == 2 + if single_batch: + points = np.expand_dims(points, axis=0) + angle = np.expand_dims(angle, axis=0) + cosa = np.expand_dims(np.cos(angle), axis=1) + sina = np.expand_dims(np.sin(angle), axis=1) + zeros = np.zeros_like(cosa) # angle.new_zeros(points.shape[0]) + ones = np.ones_like(sina) # angle.new_ones(points.shape[0]) + + rot_matrix = np.concatenate( + (cosa, -sina, zeros, sina, cosa, zeros, zeros, zeros, ones), axis=1 + ).reshape(-1, 3, 3) + + # print(rot_matrix.view(3, 3)) + points_rot = np.matmul(points[:, :, :3], rot_matrix) + points_rot = np.concatenate((points_rot, points[:, :, 3:]), axis=-1) + + if single_batch: + points_rot = points_rot.squeeze(0) + + return points_rot + + +def convert_angle_axis_to_matrix3(angle_axis): + """Return a Matrix3 for the angle axis. + Arguments: + angle_axis {Point3} -- a rotation in angle axis form. + """ + matrix, jacobian = cv2.Rodrigues(angle_axis) + return matrix diff --git a/pointcept/datasets/preprocessing/concerto/arkitscenes/splits.py b/pointcept/datasets/preprocessing/concerto/arkitscenes/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..d25a454b9f9518c9e184752f62933600fb143e95 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/arkitscenes/splits.py @@ -0,0 +1,64 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["Training", "Validation"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split).replace( + dataset_path, "data/arkitscenes" + ) + split_names = [f.name for f in os.scandir(im_split_path) if f.is_dir()] + split_dict = {} + for name in split_names: + im_split_name_path = os.path.join(im_split_path, name, "color") + co_split_name_path = os.path.join(im_split_path, name, "correspondence") + png_files = [ + f for f in os.listdir(im_split_name_path) if f.endswith(".png") + ] + png_files = sorted(png_files, key=lambda x: float(x.split("_")[1][:-4])) + co_files = [f for f in os.listdir(co_split_name_path) if f.endswith(".npy")] + co_files = sorted(co_files, key=lambda x: float(x[:-4])) + # Get the full paths of the .png files + png_file_paths = [ + os.path.join(im_split_name_path, f).replace( + dataset_path, "data/arkitscenes" + ) + for f in png_files + ] + co_file_paths = [ + os.path.join(co_split_name_path, f).replace( + dataset_path, "data/arkitscenes" + ) + for f in co_files + ] + for i in range(0, len(png_file_paths), 4): + split_dict[f"{name}_{i//4}"] = {} + split_dict[f"{name}_{i//4}"]["pointclouds"] = os.path.join( + pc_split_path, name + ) + split_dict[f"{name}_{i//4}"]["images"] = png_file_paths[i : i + 4] + split_dict[f"{name}_{i//4}"]["correspondences"] = co_file_paths[ + i : i + 4 + ] + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the arkitscenes dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/cap3d/preprocess_cap3d.py b/pointcept/datasets/preprocessing/concerto/cap3d/preprocess_cap3d.py new file mode 100644 index 0000000000000000000000000000000000000000..263d002153095cdb4a405f0890a2714434d4b040 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/cap3d/preprocess_cap3d.py @@ -0,0 +1,261 @@ +""" +Preprocessing Script for Cap3D + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import argparse +import torch +import numpy as np +from scipy.spatial import cKDTree +from PIL import Image +from pathlib import Path +import multiprocessing as mp +import zipfile +import glob +import json +import math + +NUM_VIEWS = 4 +MAX_DEPTH = 5.0 + + +def recover_depth_from_image(depth_image_path, max_depth): + with Image.open(depth_image_path) as depth_pil: + depth_uint16 = np.array(depth_pil) + invalid_depth_mask = depth_uint16 == 65535 + depth_normalized_float = depth_uint16.astype(np.float32) / 65535.0 + depth_meters = depth_normalized_float * max_depth + return depth_meters, invalid_depth_mask + + +def normalize_point_cloud(point_cloud: np.ndarray) -> np.ndarray: + bbox_min = np.min(point_cloud, axis=0) + bbox_max = np.max(point_cloud, axis=0) + bbox_size = bbox_max - bbox_min + max_dim = np.max(bbox_size) + if max_dim < 1e-8: + return point_cloud - bbox_min + + scale = 1.0 / max_dim + scaled_point_cloud = point_cloud * scale + new_bbox_min = np.min(scaled_point_cloud, axis=0) + new_bbox_max = np.max(scaled_point_cloud, axis=0) + center = (new_bbox_min + new_bbox_max) / 2.0 + offset = -center + normalized_point_cloud = scaled_point_cloud + offset + + return normalized_point_cloud + + +def get_cam_params(cam_params_file, width, height): + with open(cam_params_file, "r") as f: + data = json.load(f) + + x_vector = data["x"] + y_vector = data["y"] + z_vector = data["z"] + origin = data["origin"] + + rotation_matrix = np.array([x_vector, y_vector, z_vector]).T + + translation_vector = np.array(origin) + + rt_matrix = np.eye(4) + rt_matrix[:3, :3] = rotation_matrix + rt_matrix[:3, 3] = translation_vector + + x_fov = data["x_fov"] + y_fov = data["y_fov"] + + cx = width / 2.0 + cy = height / 2.0 + + fx = (width / 2.0) / math.tan(x_fov / 2.0) + fy = (height / 2.0) / math.tan(y_fov / 2.0) + + s = 0 + + K = np.array([[fx, s, cx], [0, fy, cy], [0, 0, 1]]) + + return rt_matrix, K + + +def correspondenceGet(depth, invalid_depth_mask, T, K, width, height): + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + depth = depth.reshape((-1, 1)) + invalid_depth_mask = invalid_depth_mask.reshape((-1, 1)) + valid = ~np.logical_or(np.isinf(depth), invalid_depth_mask).squeeze(-1) + coord = np.zeros_like(pixel, dtype=np.float32) + coord[valid] = depth[valid] * (np.linalg.inv(K) @ pixel[valid].T).T # coord_camera + coord[valid] = coord[valid] @ T[:3, :3].T + T[:3, 3] # column then row + + pixel = pixel[valid] + coord = coord[valid] + if coord.shape[0] == 0: + return None + pixel = pixel[:, :2] + coord_dict = {"pixel": pixel, "coord": coord} + return coord_dict + + +def correspondenceSave( + depth_paths, data_name, point_cloud_gt, cam_params_paths, output_root +): + """ + Calculates and saves the correspondences for all views of a given mesh and point cloud. + """ + # Build a KDTree for the ground truth point cloud for fast querying. + tree = cKDTree(point_cloud_gt) + + # Iterate over each predefined view. + for view_idx in range(len(depth_paths)): + # Construct file paths. + cam_params_path = cam_params_paths[view_idx] + depth_path = depth_paths[view_idx] + + view_name = cam_params_path.split("/")[-1][:-5] + + # Load camera parameters and image dimensions. + depth, invalid_depth_mask = recover_depth_from_image(depth_path, MAX_DEPTH) + height, width = depth.shape + T, K = get_cam_params(cam_params_path, width, height) + + coord_dict = correspondenceGet(depth, invalid_depth_mask, T, K, width, height) + if coord_dict is None: + correspondences = -np.ones((1, 3)) + else: + pixels_ = coord_dict["pixel"] + coords_ = coord_dict["coord"] + dis, idx = tree.query(coords_, k=1) + idx_valid = idx[dis < 0.01] + pixels_valid = pixels_[dis < 0.01] + correspondences = np.hstack((pixels_valid, idx_valid.reshape(-1, 1))) + output_dir = output_root / data_name + os.makedirs(output_dir, exist_ok=True) + output_filename = output_dir / f"{view_name}.npy" + np.save(output_filename, correspondences) + + +def handle_process(pt_path, output_root, cam_root): + """ + The complete processing pipeline for a single GLB file. + """ + data_name = pt_path.stem + + if not pt_path.exists(): + return + cam_path = cam_root / f"{data_name}" + cam_path_zip = cam_root / f"{data_name}.zip" + + # Check if files exist, just in case. + if not cam_path.exists(): + if not cam_path_zip.exists(): + print(f"not exist {str(cam_path)} and its zip file") + return + with zipfile.ZipFile(cam_path_zip, "r") as zip_ref: + zip_ref.extractall(cam_path) + + # Load the point cloud. + point_cloud = torch.load(pt_path) + point_cloud_np = ( + point_cloud.cpu().numpy() + if hasattr(point_cloud, "numpy") + else np.array(point_cloud) + ) + point_cloud_np = point_cloud_np[:3, :].T + depth_paths = glob.glob(os.path.join(cam_path, "*_depth.png")) + if len(depth_paths) >= NUM_VIEWS: + depth_paths = sorted(depth_paths)[:: len(depth_paths) // NUM_VIEWS] + else: + depth_paths = sorted(depth_paths) + cam_params_paths = glob.glob(os.path.join(cam_path, "*.json")) + if os.path.isfile(os.path.join(cam_path, "transforms_train.json")): + cam_params_paths.remove(os.path.join(cam_path, "transforms_train.json")) + if os.path.isfile(os.path.join(cam_path, "info.json")): + cam_params_paths.remove(os.path.join(cam_path, "info.json")) + if len(depth_paths) >= NUM_VIEWS: + cam_params_paths = sorted(cam_params_paths)[ + :: len(cam_params_paths) // NUM_VIEWS + ] + else: + cam_params_paths = sorted(cam_params_paths) + + valid_depth_paths = { + os.path.basename(p).replace("_depth.png", ""): p for p in depth_paths + } + valid_cam_params_paths = { + os.path.basename(p).replace(".json", ""): p for p in cam_params_paths + } + common_ids = set(valid_depth_paths.keys()) & set(valid_cam_params_paths.keys()) + sorted_common_ids = sorted(list(common_ids)) + + valid_depth_paths_ = [valid_depth_paths[uid] for uid in sorted_common_ids] + valid_cam_params_paths_ = [valid_cam_params_paths[uid] for uid in sorted_common_ids] + # Calculate and save correspondences for all views. + correspondenceSave( + valid_depth_paths_, + data_name, + point_cloud_np, + valid_cam_params_paths_, + output_root, + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Generate 2D-3D correspondences for GLB models and point clouds." + ) + parser.add_argument( + "--cam_root", + type=Path, + required=True, + help="Root directory containing camera parameters.", + ) + parser.add_argument( + "--point_cloud_root", + type=Path, + required=True, + help="Directory containing .pt point cloud files.", + ) + parser.add_argument( + "--output_root", + type=Path, + required=True, + help="Output directory to save correspondence files.", + ) + parser.add_argument( + "--num_workers", + type=int, + default=mp.cpu_count(), + help="Number of processes to use for processing.", + ) + parser.add_argument( + "--thread_id", type=int, default=0, help="thread id to use for processing." + ) + config = parser.parse_args() + + # Ensure the output directory exists. + os.makedirs(config.output_root, exist_ok=True) + + print("Scanning for .pt files...") + pt_files = list(config.point_cloud_root.glob("**/*.pt")) + print(f"Found {len(pt_files)} .pt files.") + + # Load scene paths + pt_files_list = np.array_split(pt_files, config.num_workers) + pt_files_ = pt_files_list[config.thread_id] + # Preprocess data. + print("Processing scenes...") + for pt_files_i in pt_files_: + handle_process( + pt_files_i, + config.output_root, + config.cam_root, + ) + + print("\nAll files processed.") diff --git a/pointcept/datasets/preprocessing/concerto/cap3d/preprocess_cap3d.sh b/pointcept/datasets/preprocessing/concerto/cap3d/preprocess_cap3d.sh new file mode 100644 index 0000000000000000000000000000000000000000..44e38262c676923c6e42df47938d520ba2a90848 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/cap3d/preprocess_cap3d.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +cam_root="" +point_cloud_root="" +output_root="" +num_workers=16 + +while getopts "c:p:o:n" opt; do + case $opt in + c) cam_root=$OPTARG ;; + p) point_cloud_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + *) echo "Usage: $0 -c -p -o [-n ]"; exit 1 ;; + esac +done + +if [ -z "$cam_root" ] || [ -z "$point_cloud_root" ] || [ -z "$output_root" ]; then + echo "Error: Missing required arguments. Use -h for help." + echo "Usage: $0 -c -p -o [-n ]" + exit 1 +fi + +for i in $(seq 0 $((num_workers - 1))); do + cmd="python pointcept/datasets/preprocessing/concerto/cap3d/preprocess_cap3d.py \ + --cam_root $cam_root \ + --point_cloud_root $point_cloud_root \ + --output_root $output_root \ + --num_workers $num_workers \ + --thread_id $i" + + eval "$cmd &" +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/cap3d/splits.py b/pointcept/datasets/preprocessing/concerto/cap3d/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..d7e801e2fcdbe960675d0fea034981edfaba71ee --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/cap3d/splits.py @@ -0,0 +1,76 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + co_path = os.path.join(dataset_path, "correspondences") + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["train"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + co_split_path = os.path.join(co_path, split) + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split) + split_names = [f.name for f in os.scandir(co_split_path) if f.is_dir()] + split_dict = {} + for name in split_names: + print(f"processing {name}") + co_name_split_path = os.path.join(co_split_path, name) + img_names = [ + f.name.removesuffix(".npy") + for f in os.scandir(co_name_split_path) + if f.is_file() and f.name.endswith(".npy") + ] + img_names = sorted(img_names) + png_file_paths = [ + os.path.join(im_split_path, name, img_name + ".png").replace( + dataset_path, "data/cap3d" + ) + for img_name in img_names + ] + # Get the full paths of the .png files + co_file_paths = [ + os.path.join(co_name_split_path, img_name + ".npy").replace( + dataset_path, "data/cap3d" + ) + for img_name in img_names + ] + pc_file_path = os.path.join(pc_split_path, name + ".pt") + file_check = True + for png_file_path in png_file_paths: + if not os.path.isfile(png_file_path): + file_check = False + for co_file_path in co_file_paths: + if not os.path.isfile(co_file_path): + file_check = False + if not os.path.isfile(pc_file_path): + file_check = False + continue + if file_check: + split_dict[f"{name}"] = {} + split_dict[f"{name}"]["pointclouds"] = pc_file_path + split_dict[f"{name}"]["images"] = png_file_paths + split_dict[f"{name}"]["correspondences"] = co_file_paths + else: + split_dict[f"{name}"] = {} + split_dict[f"{name}"]["pointclouds"] = pc_file_path + split_dict[f"{name}"]["images"] = [] + split_dict[f"{name}"]["correspondences"] = [] + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/graspnet/preprocess_graspnet_poses.py b/pointcept/datasets/preprocessing/concerto/graspnet/preprocess_graspnet_poses.py new file mode 100644 index 0000000000000000000000000000000000000000..610f5cba7d3ffb818baf5b59df417bcfd660b20f --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/graspnet/preprocess_graspnet_poses.py @@ -0,0 +1,56 @@ +""" +Preprocessing Script for GraspNet + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import numpy as np +import argparse +from tqdm import tqdm + + +def split_poses(graspnet_root): + """ + Splits the camera_poses.npy file into individual pose files for each scene. + """ + scenes_path = os.path.join(graspnet_root, "scenes") + scene_names = sorted( + [ + d + for d in os.listdir(scenes_path) + if os.path.isdir(os.path.join(scenes_path, d)) + ] + ) + + print(f"Found {len(scene_names)} scenes. Starting preprocessing...") + + for scene_name in tqdm(scene_names, desc="Processing Scenes"): + kinect_path = os.path.join(scenes_path, scene_name, "kinect") + poses_file = os.path.join(kinect_path, "camera_poses.npy") + if not os.path.exists(poses_file): + print(f"Warning: camera_poses.npy not found in {kinect_path}, skipping.") + continue + + all_poses = np.load(poses_file) # Shape: (256, 4, 4) + output_pose_dir = os.path.join(kinect_path, "pose") + os.makedirs(output_pose_dir, exist_ok=True) + + for i in range(all_poses.shape[0]): + pose = all_poses[i] + output_file = os.path.join(output_pose_dir, f"{i:04d}.npy") + np.save(output_file, pose) + + print("Preprocessing finished. Individual pose files have been created.") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Preprocess GraspNet camera poses.") + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the root of the GraspNet dataset (the one containing the 'scenes' folder).", + ) + args = parser.parse_args() + split_poses(args.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/graspnet/splits.py b/pointcept/datasets/preprocessing/concerto/graspnet/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..08e3eb22d3927ac1994c7962567cdceee383dfb5 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/graspnet/splits.py @@ -0,0 +1,91 @@ +# GraspNet : train: 0-99, test_seen: 100-189 +# train: scene_0000 to scene_0100 (100 scenes) +# val: scene_0100 to scene_0189 (90 scenes) +import os +import json +import argparse +from tqdm import tqdm + + +def generate_graspnet_splits(dataset_path, group_num=4): + """ + Generates train/val/test JSON split files for the GraspNet dataset. + """ + scenes_dir = os.path.join(dataset_path, "scenes") + all_scenes = sorted( + [f for f in os.scandir(scenes_dir) if f.is_dir()], key=lambda f: f.name + ) + all_scene_names = [f.name for f in all_scenes] + + split_definitions = { + "train": [s for s in all_scene_names if 100 > int(s.split("_")[1]) >= 0], + "val": [s for s in all_scene_names if int(s.split("_")[1]) >= 100], + } + + output_split_path = os.path.join(dataset_path, "splits") + os.makedirs(output_split_path, exist_ok=True) + print(f"JSON files will be saved in: {output_split_path}") + + for split, scene_names in split_definitions.items(): + print(f"Processing '{split}' split with {len(scene_names)} scenes...") + split_dict = {} + for scene_name in tqdm(scene_names, desc=f"Generating {split}.json"): + kinect_path = os.path.join(scenes_dir, scene_name, "kinect") + + rgb_path = os.path.join(kinect_path, "rgb") + depth_path = os.path.join(kinect_path, "depth") + pose_path = os.path.join(kinect_path, "pose") + K_file_path_abs = os.path.join(kinect_path, "camK.npy") + K_file_path_rel = K_file_path_abs.replace(dataset_path, "data/graspnet") + + rgb_files = [f for f in os.listdir(rgb_path) if f.endswith(".png")] + rgb_files = sorted(rgb_files, key=lambda x: int(os.path.splitext(x)[0])) + + rgb_file_paths = [ + os.path.join(rgb_path, f).replace(dataset_path, "data/graspnet") + for f in rgb_files + ] + depth_file_paths = [ + os.path.join(depth_path, f).replace(dataset_path, "data/graspnet") + for f in rgb_files + ] + pose_file_paths = [ + os.path.join(pose_path, f.replace(".png", ".npy")).replace( + dataset_path, "data/graspnet" + ) + for f in rgb_files + ] + + for i in range(0, len(rgb_file_paths), group_num): + if i + 4 > len(rgb_file_paths): + continue + + chunk_key = f"{scene_name}_{i//group_num}" + split_dict[chunk_key] = { + "images": rgb_file_paths[i : i + group_num], + "depths": depth_file_paths[i : i + group_num], + "Ts": pose_file_paths[i : i + group_num], + "Ks": [K_file_path_rel] * group_num, + } + + output_json_file = os.path.join(output_split_path, f"{split}.json") + with open(output_json_file, "w") as f: + json.dump(split_dict, f, indent=4) + print(f"Successfully created {output_json_file}") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the GraspNet dataset root (containing the 'scenes' folder).", + ) + parser.add_argument( + "--group_num", + default=4, + type=int, + help="Group Num.", + ) + config = parser.parse_args() + generate_graspnet_splits(config.dataset_root, config.group_num) diff --git a/pointcept/datasets/preprocessing/concerto/hk/hkremote_download.sh b/pointcept/datasets/preprocessing/concerto/hk/hkremote_download.sh new file mode 100644 index 0000000000000000000000000000000000000000..8b82d35f6225da40998555cb7e214f741ef52ac9 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/hk/hkremote_download.sh @@ -0,0 +1,60 @@ +#!/bin/bash + +# Export variables so they can be accessed by subshells spawned by xargs +export BASE_URL="https://data11.map.gov.hk/api/3d-zip/OBJ/" +# To get API_KEY, visit https://3d.map.gov.hk/download-api. After you agree the license, your API_KEY will show in Example URL: https://download.map.gov.hk/api/3d-zip/[format]/[tile_sheet_num].zip?key=[API_KEY] +export API_KEY=${API_KEY} +export OUTPUT_DIR="hk_3d_maps" +export MAX_THREADS=8 # Set the number of concurrent downloads here + +mkdir -p "$OUTPUT_DIR" +echo "Files will be downloaded to: $OUTPUT_DIR" +echo "Using $MAX_THREADS concurrent threads..." + +# Define the download logic as a function +download_map() { + local MAP_INDEX=$1 + local ZIP_FILENAME="${MAP_INDEX}.zip" + local OUTPUT_PATH="${OUTPUT_DIR}/${ZIP_FILENAME}" + local DOWNLOAD_URL="${BASE_URL}${ZIP_FILENAME}?key=${API_KEY}" + + # Check existence + if [ -f "$OUTPUT_PATH" ]; then + echo "[SKIP] File already exists: ${ZIP_FILENAME}" + return 0 + fi + + echo "[START] Downloading: ${MAP_INDEX}" + + # Use wget to download + # Added --timeout=10 and --tries=2 to prevent threads from hanging forever + wget --no-check-certificate -q --timeout=10 --tries=2 -O "$OUTPUT_PATH" "$DOWNLOAD_URL" + + # Check wget exit code; $? == 0 means success + if [ $? -eq 0 ]; then + echo "[SUCCESS] Downloaded: ${ZIP_FILENAME}" + else + echo "[FAILED] Not found or failed: ${ZIP_FILENAME}" + # Delete the potentially empty file created by a failed download + rm -f "$OUTPUT_PATH" + fi +} + +# Export the function so xargs subshells can use it +export -f download_map + +# Generate the list of map indices and pipe them to xargs for parallel execution +# We use standard space-separated lists for the string variables for cleaner execution +( + for base in {1..15}; do + for quad_10k in NW NE SW SE; do + for grid_2k in {1..25}; do + for sub_grid_1k in A B C D; do + echo "${base}-${quad_10k}-${grid_2k}${sub_grid_1k}" + done + done + done + done +) | xargs -P "$MAX_THREADS" -I {} bash -c 'download_map "$@"' _ {} + +echo "All download tasks completed!" \ No newline at end of file diff --git a/pointcept/datasets/preprocessing/concerto/hk/preprocess_hk.py b/pointcept/datasets/preprocessing/concerto/hk/preprocess_hk.py new file mode 100644 index 0000000000000000000000000000000000000000..2552c8a09594469054b4d6862d7c863ab677aaae --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/hk/preprocess_hk.py @@ -0,0 +1,110 @@ +""" +Preprocessing Script for HK Remote + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import open3d as o3d +import numpy as np +from pathlib import Path +import argparse +import multiprocessing as mp + + +def handle_process(obj_file_paths, output_directory): + num_points_to_sample = 10000000 + for obj_file_path in obj_file_paths: + try: + print(f"Loading mesh from: {obj_file_path}") + parent_name = obj_file_path.parent.name + save_path = output_directory / parent_name + if save_path.exists(): + print(f"exist {str(parent_name)}") + continue + mesh = o3d.io.read_triangle_mesh(obj_file_path, enable_post_processing=True) + + if not mesh.has_vertices(): + print("Error: Mesh could not be loaded.") + exit() + + if not mesh.has_textures(): + print( + "Warning: Mesh does not have textures. Sampled colors will be black or default." + ) + + mesh.compute_vertex_normals() + print("Mesh loaded successfully. Proceeding to sample points...") + + pcd = mesh.sample_points_uniformly(number_of_points=num_points_to_sample) + points = np.asarray(pcd.points) + colors = np.asarray(pcd.colors) + if colors.shape[0] == 0: + colors = np.zeros_like(points) + normals = np.asarray(pcd.normals) + if normals.shape[0] == 0: + normals = np.zeros_like(points) + + print(f"\nSuccessfully sampled {len(points)} points.") + print(f"Points array shape: {points.shape}") # (N, 3) for XYZ coordinates + print( + f"Colors array shape: {colors.shape}" + ) # (N, 3) for RGB values (0-1 range) + print(f"Normals array shape: {normals.shape}") # (N, 3) for normal vectors + + output_coord_path = output_directory / parent_name / "coord.npy" + output_color_path = output_directory / parent_name / "color.npy" + output_normal_path = output_directory / parent_name / "normal.npy" + (output_directory / parent_name).mkdir(exist_ok=True) + + np.save(output_coord_path, points) + np.save(output_color_path, (colors * 255).astype(np.int32)) + np.save(output_normal_path, normals) + + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(points) + pcd.colors = o3d.utility.Vector3dVector(colors) + pcd.normals = o3d.utility.Vector3dVector(normals) + print( + f"\nSampled point cloud saved to: {str(output_directory/parent_name)}" + ) + except: + print(f"fail {parent_name}") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + parser.add_argument( + "--thread_id", + default=0, + type=int, + help="Thread id for parallel processing", + ) + config = parser.parse_args() + output_directory = Path(config.output_root) / "train" + root_directory = Path(config.dataset_root) + output_directory.mkdir(exist_ok=True) + obj_file_paths_generator = root_directory.rglob("*.obj") + obj_file_paths = list(obj_file_paths_generator) + print(f"Found {len(obj_file_paths)} .obj files.") + + obj_file_paths_list = np.array_split(obj_file_paths, config.num_workers) + obj_file_paths_ = obj_file_paths_list[config.thread_id] + + handle_process(obj_file_paths_, output_directory) diff --git a/pointcept/datasets/preprocessing/concerto/hk/preprocess_hk.sh b/pointcept/datasets/preprocessing/concerto/hk/preprocess_hk.sh new file mode 100644 index 0000000000000000000000000000000000000000..52869ec7831787f6c9d45fae245226ff0240d240 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/hk/preprocess_hk.sh @@ -0,0 +1,30 @@ +#!/bin/bash + +dataset_root="" +output_root="" +num_workers=16 +parse_depths=false + +while getopts "d:o:n:p" opt; do + case $opt in + d) dataset_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + *) echo "Usage: $0 -d -o [-n ]"; exit 1 ;; + esac +done + +if [ -z "$dataset_root" ] || [ -z "$output_root" ]; then + echo "Usage: $0 -d -o [-n ]" + exit 1 +fi + +for i in $(seq 0 $((num_workers - 1))); do + cmd="python pointcept/datasets/preprocessing/concerto/hk/preprocess_hk.py --thread_id $i \ + --num_workers $num_workers \ + --dataset_root $dataset_root \ + --output_root $output_root" + eval "$cmd &" +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/hk/splits.py b/pointcept/datasets/preprocessing/concerto/hk/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..6bde2a004c7e7cc1bae474c4965e753f042f8279 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/hk/splits.py @@ -0,0 +1,35 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + pc_path = dataset_path + splits = ["train"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + pc_split_path = os.path.join(pc_path, split) + split_names = [f.name for f in os.scandir(pc_split_path) if f.is_dir()] + pc_split_path = pc_split_path.replace(dataset_path, "data/hk") + split_dict = {} + for name in split_names: + split_dict[f"{name}"] = {} + split_dict[f"{name}"]["pointclouds"] = os.path.join(pc_split_path, name) + split_dict[f"{name}"]["images"] = [] + split_dict[f"{name}"]["correspondences"] = [] + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/hm3d/__init__.py b/pointcept/datasets/preprocessing/concerto/hm3d/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/pointcept/datasets/preprocessing/concerto/hm3d/common/utils.py b/pointcept/datasets/preprocessing/concerto/hm3d/common/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..63077efcf11bafd0eb9010299c330867f8ea5ed8 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/hm3d/common/utils.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import math +import random +from typing import List, Optional + +import habitat_sim +import numpy as np +import quaternion as qt +from habitat_sim.utils.common import quat_from_angle_axis + + +def get_random_quaternion() -> qt.quaternion: + return quat_from_angle_axis( + math.radians(random.uniform(-180.0, 180.0)), np.array([0, 1.0, 0]) + ) + + +def convert_heading_to_quaternion(heading: float) -> qt.quaternion: + # heading angle in degrees + return quat_from_angle_axis(math.radians(heading), np.array([0, 1.0, 0])) + + +def make_habitat_configuration(scene_path: str, use_sensor: bool = False): + # simulator configuration + backend_cfg = habitat_sim.SimulatorConfiguration() + backend_cfg.scene_id = scene_path + + # agent configuration + sensor_cfg = habitat_sim.CameraSensorSpec() + sensor_cfg.resolution = [1080, 960] + sensor_cfg.sensor_type = habitat_sim.SensorType.DEPTH + agent_cfg = habitat_sim.agent.AgentConfiguration() + agent_cfg.sensor_specifications = [sensor_cfg] if use_sensor else [] + + return habitat_sim.Configuration(backend_cfg, [agent_cfg]) + + +def robust_load_sim(scene_path: str) -> habitat_sim.Simulator: + sim_cfg = make_habitat_configuration(scene_path, use_sensor=False) + hsim = habitat_sim.Simulator(sim_cfg) + if not hsim.pathfinder.is_loaded: + hsim.close() + sim_cfg = make_habitat_configuration(scene_path, use_sensor=True) + hsim = habitat_sim.Simulator(sim_cfg) + navmesh_settings = habitat_sim.NavMeshSettings() + navmesh_settings.set_defaults() + hsim.recompute_navmesh(hsim.pathfinder, navmesh_settings) + return hsim + + +def get_filtered_scenes(scenes: List[str], filter_scenes_path: str) -> List[str]: + """ + Filter scenes to only include valid scenes. + """ + with open(filter_scenes_path, "r") as fp: + filter_scenes = fp.readlines() + filter_scenes = [f.strip("\n") for f in filter_scenes] + filtered_scenes = [] + for scene in scenes: + scene_name = scene.split("/")[-1][: -len(".glb")] + if scene_name in filter_scenes: + filtered_scenes.append(scene) + return filtered_scenes + + +def quaternion_to_list(q: qt.quaternion): + return q.imag.tolist() + [q.real] + + +# ------------------------------------------------------------------------------- +# Functionality from habitat-lab +# ------------------------------------------------------------------------------- +def calculate_meters_per_pixel(map_resolution: int, pathfinder=None): + r"""Calculate the meters_per_pixel for a given map resolution""" + lower_bound, upper_bound = pathfinder.get_bounds() + return min( + abs(upper_bound[coord] - lower_bound[coord]) / map_resolution + for coord in [0, 2] + ) + + +def get_topdown_map( + pathfinder, + height: float, + map_resolution: int = 1024, + meters_per_pixel: Optional[float] = None, +) -> np.ndarray: + r"""Return a top-down occupancy map for a sim. Note, this only returns valid + values for whatever floor the agent is currently on. + :param pathfinder: A habitat-sim pathfinder instances to get the map from + :param height: The height in the environment to make the topdown map + :param map_resolution: Length of the longest side of the map. Used to calculate :p:`meters_per_pixel` + :param draw_border: Whether or not to draw a border + :param meters_per_pixel: Overrides map_resolution an + :return: Image containing 0 if occupied, 1 if unoccupied, and 2 if border (if + the flag is set). + """ + + if meters_per_pixel is None: + meters_per_pixel = calculate_meters_per_pixel( + map_resolution, pathfinder=pathfinder + ) + + top_down_map = pathfinder.get_topdown_view( + meters_per_pixel=meters_per_pixel, height=height + ).astype(np.uint8) + + return np.ascontiguousarray(top_down_map) diff --git a/pointcept/datasets/preprocessing/concerto/hm3d/preprocessed_hm3d.py b/pointcept/datasets/preprocessing/concerto/hm3d/preprocessed_hm3d.py new file mode 100644 index 0000000000000000000000000000000000000000..090006c1992238d07c92064a92a2031f691ac68f --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/hm3d/preprocessed_hm3d.py @@ -0,0 +1,617 @@ +""" +Preprocessing Script for hm3d using habitat-sim +https://github.com/facebookresearch/habitat-sim/blob/main/examples/tutorials/notebooks/ECCV_2020_Navigation.ipynb + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +#!/usr/bin/env python3 + +# Copyright (c) Facebook, Inc. and its affiliates. +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import argparse +import glob +import itertools +import multiprocessing as mp +import os +from pathlib import Path +import random +from typing import Any, Dict, List, Optional, Tuple +import habitat_sim +import imageio +import numpy as np +from sklearn.cluster import DBSCAN +from scipy.spatial import cKDTree +import trimesh +import sys + +sys.path.append("../../hm3d/") +from hm3d_constants import CLASS_LABELS_40 + + +from common.utils import ( + convert_heading_to_quaternion, + get_topdown_map, +) + +MPCAT2INDEX = dict([(CLASS_LABELS_40[i], i) for i in range(40)]) +MPCAT2INDEX["unlabeled"] = -1 +CAT2INDEX = dict() +with open(Path("../../hm3d") / "hm3dsem_category_mappings.tsv") as f: + f.readline() # raw_category category mpcat40 + lines = f.readlines() +for line in lines: + line = line.strip().split("\t") + if len(line) == 2: + # L2164: "\tunknown\tunlabeled" + line.insert(0, "") + CAT2INDEX[line[1]] = MPCAT2INDEX[line[2]] + + +def correspondenceGet(depth, K, T, img_size): + height, width = img_size + if np.isnan(T).any() or np.isinf(T).any(): + return None + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + depth = depth.reshape((-1, 1)) + valid = ~np.isinf(depth).squeeze(-1) + coord = np.zeros_like(pixel, dtype=np.float32) + coord[valid] = depth[valid] * (np.linalg.inv(K) @ pixel[valid].T).T # coord_camera + coord[valid] = coord[valid] @ T[:3, :3].T + T[:3, 3] # column then row + pixel = pixel[valid] + coord = coord[valid] + if coord.shape[0] == 0: + return None + pixel = pixel[:, :2] + coord_dict = {"pixel": pixel, "coord": coord} + return coord_dict + + +def uv_to_texture_color(uv, texture): + width, height = texture.size + u = int(uv[0] * width) + v = int((1 - uv[1]) * height) # Flip y-axis for image coordinates + if 0 <= u < width and 0 <= v < height: + return texture.getpixel((u, v)) + else: + return 0, 0, 0 + + +def load_hex_mapping(mapping_path): + hex2label = {} + with open(mapping_path) as f: + f.readline() # remove 'HM3D Semantic Annotations\n' + lines = f.readlines() # get the left + for line in lines: + line = line.strip().split(",") + cat = line[2].strip('"') + if cat == "trashcan": + cat = "trash can" + elif cat == "fridge": + cat = "refrigerator" + hex2label[line[1]] = dict(instance=line[0], segment=CAT2INDEX[cat]) + return hex2label + + +def make_habitat_configuration( + scene_path: str, + hfov: int = 90, + resolution: Tuple[int] = (300, 300), + stage_json_path: Optional[str] = None, +): + # simulator configuration + backend_cfg = habitat_sim.SimulatorConfiguration() + if stage_json_path is not None: + backend_cfg.scene_dataset_config_file = stage_json_path + backend_cfg.scene_id = "habitat/" + scene_path.split("/")[-1] + else: + backend_cfg.scene_id = scene_path + + # agent configuration + rgb_sensor_cfg = habitat_sim.CameraSensorSpec() + rgb_sensor_cfg.uuid = "rgba" + rgb_sensor_cfg.sensor_type = habitat_sim.SensorType.COLOR + rgb_sensor_cfg.resolution = resolution + rgb_sensor_cfg.hfov = hfov + rgb_sensor_cfg.position = [0.0, 0.0, 0.0] + depth_sensor_cfg = habitat_sim.CameraSensorSpec() + depth_sensor_cfg.uuid = "depth" + depth_sensor_cfg.sensor_type = habitat_sim.SensorType.DEPTH + depth_sensor_cfg.resolution = resolution + depth_sensor_cfg.hfov = hfov + depth_sensor_cfg.position = [0.0, 0.0, 0.0] + + agent_cfg = habitat_sim.agent.AgentConfiguration() + agent_cfg.sensor_specifications = [rgb_sensor_cfg, depth_sensor_cfg] + + return habitat_sim.Configuration(backend_cfg, [agent_cfg]) + + +def robust_load_sim(scene_path: str, **kwargs: Any) -> habitat_sim.Simulator: + sim_cfg = make_habitat_configuration(scene_path, **kwargs) + hsim = habitat_sim.Simulator(sim_cfg) + if not hsim.pathfinder.is_loaded: + navmesh_settings = habitat_sim.NavMeshSettings() + navmesh_settings.set_defaults() + hsim.recompute_navmesh(hsim.pathfinder, navmesh_settings) + return hsim + + +def get_floor_heights( + sim: habitat_sim.Simulator, max_points_to_sample: int = 20000 +) -> List[Dict[str, float]]: + """Get heights of different floors in a scene. This is done in two steps. + (1) Randomly samples navigable points in the scene. + (2) Cluster the points based on discretized y coordinates to get floors. + + Args: + sim: habitat simulator instance + max_points_to_sample: number of navigable points to randomly sample + """ + nav_points = [] + for _ in range(max_points_to_sample): + nav_points.append(sim.pathfinder.get_random_navigable_point()) + nav_points = np.stack(nav_points, axis=0) + y_coors = np.around(nav_points[:, 1], decimals=1) + # cluster Y coordinates + clustering = DBSCAN(eps=0.2, min_samples=2000).fit(y_coors[:, np.newaxis]) + c_labels = clustering.labels_ + n_clusters = len(set(c_labels)) - (1 if -1 in c_labels else 0) + # get floor extents in Y + # each cluster corresponds to points from 1 floor + floor_extents = [] + core_sample_y = y_coors[clustering.core_sample_indices_] + core_sample_labels = c_labels[clustering.core_sample_indices_] + for i in range(n_clusters): + floor_min = core_sample_y[core_sample_labels == i].min().item() + floor_max = core_sample_y[core_sample_labels == i].max().item() + floor_mean = core_sample_y[core_sample_labels == i].mean().item() + floor_extents.append({"min": floor_min, "max": floor_max, "mean": floor_mean}) + floor_extents = sorted(floor_extents, key=lambda x: x["mean"]) + + # reject floors that have too few points + max_points = 0 + for fext in floor_extents: + top_down_map = get_topdown_map(sim.pathfinder, fext["mean"]) + max_points = max(np.count_nonzero(top_down_map), max_points) + clean_floor_extents = [] + for fext in floor_extents: + top_down_map = get_topdown_map(sim.pathfinder, fext["mean"]) + num_points = np.count_nonzero(top_down_map) + if num_points < 0.2 * max_points: + continue + clean_floor_extents.append(fext) + + return clean_floor_extents + + +def get_navmesh_extents_at_y( + sim: habitat_sim.Simulator, y_bounds: Optional[Tuple[float]] = None +) -> Tuple[float]: + if y_bounds is None: + lower_bound, upper_bound = sim.pathfinder.get_bounds() + else: + assert len(y_bounds) == 2 + assert y_bounds[0] < y_bounds[1] + navmesh_vertices = np.array(sim.pathfinder.build_navmesh_vertices()) + navmesh_vertices = navmesh_vertices[ + (y_bounds[0] <= navmesh_vertices[:, 1]) + & (navmesh_vertices[:, 1] <= y_bounds[1]) + ] + lower_bound = navmesh_vertices.min(axis=0) + upper_bound = navmesh_vertices.max(axis=0) + return (lower_bound, upper_bound) + + +def get_dense_navmesh_vertices( + sim: habitat_sim.Simulator, sampling_resolution: float = 0.5 +) -> np.ndarray: + + navmesh_vertices = [] + floor_extents = get_floor_heights(sim) + for fext in floor_extents: + l_bound, u_bound = get_navmesh_extents_at_y( + sim, y_bounds=(fext["min"] - 0.5, fext["max"] + 0.5) + ) + x_range = np.arange(l_bound[0].item(), u_bound[0].item(), sampling_resolution) + y = fext["mean"] + z_range = np.arange(l_bound[2].item(), u_bound[2].item(), sampling_resolution) + for x, z in itertools.product(x_range, z_range): + if sim.pathfinder.is_navigable(np.array([x, y, z])): + navmesh_vertices.append((np.array([x, y, z]))) + if len(navmesh_vertices) > 0: + navmesh_vertices = np.stack(navmesh_vertices, axis=0) + else: + navmesh_vertices = np.zeros((0, 3)) + return navmesh_vertices + + +def get_dense_mesh_vertices( + sim: habitat_sim.Simulator, room_bounds: dict, sampling_resolution: float = 0.5 +) -> np.ndarray: + navmesh_vertices = [] + for room_id, room_bound in room_bounds.items(): + x_range = np.arange( + room_bound["min_x"] + 0.5, room_bound["max_x"] - 0.5, sampling_resolution + ) + y_range = np.arange( + room_bound["min_y"] + 0.5, room_bound["max_y"] - 0.5, sampling_resolution + ) + z_range = np.arange( + room_bound["min_z"] + 0.5, room_bound["max_z"] - 0.5, sampling_resolution + ) + for x, y, z in itertools.product(x_range, y_range, z_range): + navmesh_vertices.append((np.array([x, y, z]))) + if len(navmesh_vertices) > 0: + navmesh_vertices = np.stack(navmesh_vertices, axis=0) + else: + navmesh_vertices = np.zeros((0, 3)) + return navmesh_vertices + + +def get_scene_name(scene_path, dataset): + if dataset == "replica": + scene_name = scene_path.split("/")[-2].split(".")[0] + else: + scene_name = scene_path.split("/")[-1].split(".")[0] + return scene_name + + +def plySave(coords, colors, save_path): + ply_path = save_path / "point_cloud.ply" + trimesh.points.PointCloud(coords, colors=colors).export(ply_path) + + +def _aux_fn(args): + ( + density, + scene_path, + save_prefix, + hfov, + resolution, + sampling_resolution, + num_rotations, + sim_kwargs, + parse_pointclouds, + parse_depths, + ) = args + if sim_kwargs is None: + sim_kwargs = {} + sim = robust_load_sim(scene_path, hfov=hfov, resolution=resolution, **sim_kwargs) + save_prefix = Path(save_prefix) + rgb_save_prefix = save_prefix / "images" + pc_save_prefix = save_prefix + intrinsic_save_path = Path(rgb_save_prefix) / "intrinsic" + intrinsic_save_path.mkdir(parents=True, exist_ok=True) + np.save(intrinsic_save_path / "K.npy", K) + scene_path = Path(scene_path) + scene_label_path = scene_path.with_suffix(".semantic.glb") + scene_mapping_path = scene_path.with_suffix(".semantic.txt") + scene_name = scene_path.parent.name.replace("-", "_") + scene_id = scene_name.split("_")[0] + labeled = True if scene_label_path.is_file() else False + + # test split (900-1000) is reserved by official + if 0 <= int(scene_id) < 800: + split = "train" + else: + split = "val" + print(f"Parsing scene {scene_name} in {split} split...") + scene = trimesh.load(scene_path) + if labeled: + labeled_scene_ = trimesh.load(scene_label_path) + labeled_scene = trimesh.Scene() + for name, mesh in labeled_scene_.geometry.items(): + # some case, name in labeled scene and scene is not matched, so only use chunk id + labeled_scene.add_geometry(mesh, geom_name=name.split("_")[0]) + del labeled_scene_ + hex2label = load_hex_mapping(scene_mapping_path) + + room_dict = {} + room_bound = {} + for name, mesh in scene.geometry.items(): + room_id = "_".join(name.split("_")[1:3]).replace("group", "").replace("sub", "") + if room_id not in room_dict.keys(): + room_dict[room_id] = trimesh.Scene() + room_bound[room_id] = { + "min_x": np.inf, + "max_x": -np.inf, + "min_y": np.inf, + "max_y": -np.inf, + "min_z": np.inf, + "max_z": -np.inf, + } + room_dict[room_id].add_geometry(mesh, geom_name=name.split("_")[0]) + bounding_box = mesh.bounds + room_bound[room_id]["min_x"] = min( + room_bound[room_id]["min_x"], bounding_box[0][0] + ) + room_bound[room_id]["max_x"] = max( + room_bound[room_id]["max_x"], bounding_box[1][0] + ) + room_bound[room_id]["min_y"] = min( + room_bound[room_id]["min_y"], bounding_box[0][1] + ) + room_bound[room_id]["max_y"] = max( + room_bound[room_id]["max_y"], bounding_box[1][1] + ) + room_bound[room_id]["min_z"] = min( + room_bound[room_id]["min_z"], bounding_box[0][2] + ) + room_bound[room_id]["max_z"] = max( + room_bound[room_id]["max_z"], bounding_box[1][2] + ) + + # Get dense navmesh vertices + navmesh_vertices = get_dense_mesh_vertices( + sim, room_bound, sampling_resolution=sampling_resolution + ) + room_vertices = dict() + for navmesh_vertex in navmesh_vertices: + for room_id, bound in room_bound.items(): + if ( + bound["min_x"] < navmesh_vertex[0] < bound["max_x"] + and bound["min_z"] < navmesh_vertex[2] < bound["max_z"] + and bound["min_y"] < navmesh_vertex[1] < bound["max_y"] + ): + if room_id not in room_vertices.keys(): + room_vertices[room_id] = [] + room_vertices[room_id].append(navmesh_vertex) + break + + for room_id, scene in room_dict.items(): + if room_id not in room_vertices.keys(): + continue + np.random.seed(int(scene_id + room_id.replace("_", ""))) + room_coord = [] + room_color = [] + room_normal = [] + if labeled: + room_label_color = [] + + for name in scene.geometry.keys(): + mesh = scene.geometry[name] + num_points = int(np.sum(mesh.area_faces) / density**2) + if num_points == 0: + continue + coords, face_indices = mesh.sample(num_points, return_index=True) + faces = mesh.faces[face_indices] + triangles = mesh.vertices[faces] + bary_coords = trimesh.triangles.points_to_barycentric(triangles, coords) + uv_coords = mesh.visual.uv[faces] + sampled_uvs = np.einsum("ijk,ij->ik", uv_coords, bary_coords) + + pbr_material = mesh.visual.material + texture_image = pbr_material.baseColorTexture + if texture_image is None: + continue + colors = np.array( + [uv_to_texture_color(uv, texture_image) for uv in sampled_uvs] + ) + normals = mesh.vertex_normals[faces] + normals = np.einsum("ijk,ij->ik", normals, bary_coords) + room_coord.append(coords) + room_color.append(colors) + room_normal.append(normals) + + if labeled: + labeled_mash = labeled_scene.geometry[name] + label_texture_image = labeled_mash.visual.material.baseColorTexture + label_color = np.array( + [uv_to_texture_color(uv, label_texture_image) for uv in sampled_uvs] + ) + room_label_color.append(label_color) + if len(room_coord) == 0: + continue + + room_coord = np.concatenate(room_coord, axis=0).astype("float32") + if parse_pointclouds: + room_color = np.concatenate(room_color, axis=0).astype("uint8") + room_normal = np.concatenate(room_normal, axis=0).astype("float32") + data_dict = dict(coord=room_coord, color=room_color, normal=room_normal) + + if labeled: + room_label_color = np.concatenate(room_label_color, axis=0) + instance_label_color = np.unique(room_label_color, axis=0) + room_instance = -np.ones(len(room_label_color), dtype="int16") + room_segment = -np.ones(len(room_label_color), dtype="int16") + for i in range(len(instance_label_color)): + label_color = instance_label_color[i] + label_hex = "{c[0]:02x}{c[1]:02x}{c[2]:02x}".format(c=label_color) + mask = np.all(room_label_color == label_color, axis=-1) + room_instance[mask] = i + if label_hex.upper() in hex2label.keys(): + room_segment[mask] = hex2label[label_hex.upper()]["segment"] + data_dict["instance"] = room_instance + data_dict["segment"] = room_segment + pc_save_path_single = ( + Path(pc_save_prefix) / split / "_".join([scene_name, room_id]) + ) + os.makedirs(pc_save_path_single, exist_ok=True) + print(f"Saving data to {pc_save_path_single}...") + for key, value in data_dict.items(): + np.save(pc_save_path_single / f"{key}.npy", value) + + agent = sim.get_agent(0) + rgb_paths = [] + depth_paths = [] + if not sim.pathfinder.is_loaded: + sim.close() + return rgb_paths, depth_paths + + vertex = sum(room_vertices[room_id]) / len(room_vertices[room_id]) + + T0 = np.array([[1, 0, 0, 0], [0, 0, 1, 0], [0, -1, 0, 0], [0, 0, 0, 1]]) + Ts = [] + loc_ = vertex + np.random.uniform(-0.25, 0.25, size=vertex.shape) + count = 0 + loc = loc_ @ T0[:3, :3].T + init_angle = random.uniform(0, 90) + pose_save_path = ( + Path(rgb_save_prefix) / split / "_".join([scene_name, room_id]) / "pose" + ) + pose_save_path.mkdir(parents=True, exist_ok=True) + rgb_path = ( + Path(rgb_save_prefix) / split / "_".join([scene_name, room_id]) / "color" + ) + os.makedirs(rgb_path, exist_ok=True) + if parse_depths: + depth_path = ( + Path(rgb_save_prefix) + / split + / "_".join([scene_name, room_id]) + / "depth" + ) + os.makedirs(depth_path, exist_ok=True) + for heading in np.linspace(init_angle, init_angle + 360, num_rotations + 1)[ + :-1 + ]: + rot = convert_heading_to_quaternion(heading) + angle = np.radians(-heading) # Replace 45 with the desired angle in degrees + R_y = np.array( + [ + [np.cos(angle), 0, np.sin(angle)], + [0, 1, 0], + [-np.sin(angle), 0, np.cos(angle)], + ] + ) + T = np.eye(4) + T[:3, :3] = T0[:3, :3] @ R_y + T[:3, 3] = loc_ + Ts.append(T) + np.save(pose_save_path / f"{count}.npy", T) + agent_state = agent.get_state() + agent_state.position = loc + agent_state.rotation = rot + agent.set_state(agent_state, reset_sensors=True) + obs = sim.get_sensor_observations() + rgb = obs["rgba"][..., :3] + depth = obs["depth"] + rgb_path_ = rgb_path / f"{count}.png" + imageio.imwrite(rgb_path_, rgb) + if parse_depths: + depth_path_ = depth_path / f"{count}.png" + depth_img = (depth * 1000).astype(np.uint16) # Convert to mm + imageio.imwrite(depth_path_, depth_img) + coord_dict = correspondenceGet(depth, K, T, RESOLUTION) + if coord_dict is None: + correspondences = np.ones((1, 3)) + else: + pixels_ = coord_dict["pixel"] + coords_ = coord_dict["coord"] + tree = cKDTree(room_coord) + dis, idx = tree.query(coords_, k=1) + idx_valid = idx[dis < 0.01] + pixels_valid = pixels_[dis < 0.01] + co_save_path = ( + Path(rgb_save_prefix) + / split + / "_".join([scene_name, room_id]) + / "correspondence" + ) + co_save_path.mkdir(parents=True, exist_ok=True) + correspondences = np.hstack((pixels_valid, idx_valid.reshape(-1, 1))) + np.save(co_save_path / "{}.npy".format(count), correspondences) + count += 1 + sim.close() + + +HFOV = 90 +HFOV_rad = np.deg2rad(HFOV) +RESOLUTION = [720, 720] +NUM_ROTATIONS = 4 +F = RESOLUTION[1] / (2 * np.tan(HFOV_rad / 2)) +CX, CY = RESOLUTION[1] / 2, RESOLUTION[0] / 2 +K = np.array([[F, 0, CX], [0, F, CY], [0, 0, 1]]) + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--parse_pointclouds", + action="store_true", + help="Parse point clouds from the scenes", + ) + parser.add_argument( + "--parse_depths", + action="store_true", + help="Parse depths from the scenes", + ) + parser.add_argument( + "--density", + default=0.02, + type=float, + help="Sampling density on mesh surface (m)", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + parser.add_argument( + "--worker_id", + default=0, + type=int, + help="Num workers for preprocessing.", + ) + args = parser.parse_args() + + scene_list = glob.glob(os.path.join(args.dataset_root, "*", "*.glb")) + scene_list = [scene for scene in scene_list if not scene.endswith("semantic.glb")] + scene_list = [scene for scene in scene_list if not scene.endswith("basis.glb")] + scene_list = sorted(scene_list, key=lambda x: int(x.split("/")[-2].split("-")[0])) + assert len(scene_list) == 900 + print( + f"Found {len(scene_list)} scenes in {args.dataset_root},using {args.num_workers} workers." + ) + scene_list = [scene_list[i :: args.num_workers] for i in range(args.num_workers)][ + args.worker_id + ] + + pc_train_dir = os.path.join(args.output_root, "pointclouds", "train") + pc_val_dir = os.path.join(args.output_root, "pointclouds", "val") + existing_scene_ids = set() + + for pc_dir in [pc_train_dir, pc_val_dir]: + if os.path.exists(pc_dir): + for folder in os.listdir(pc_dir): + if "_" in folder: + scene_id = folder.split("_")[0] + existing_scene_ids.add(scene_id) + + scene_list = [ + scene + for scene in scene_list + if scene.split("/")[-2].split("-")[0] not in existing_scene_ids + ] + print(f"Processing {len(scene_list)} scenes...") + + for scene_path in scene_list: + inputs = ( + args.density, + scene_path, + args.output_root, + HFOV, + RESOLUTION, + 0.5, + NUM_ROTATIONS, + None, + args.parse_pointclouds, + args.parse_depths, + ) + _aux_fn(inputs) diff --git a/pointcept/datasets/preprocessing/concerto/hm3d/preprocessed_hm3d.sh b/pointcept/datasets/preprocessing/concerto/hm3d/preprocessed_hm3d.sh new file mode 100644 index 0000000000000000000000000000000000000000..eb81aa40d7e129cc3be28308f15eac37db848b44 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/hm3d/preprocessed_hm3d.sh @@ -0,0 +1,42 @@ +#!/bin/bash + +dataset_root="" +output_root="" +num_workers=8 +parse_depths=false +parse_pointclouds=false + +while getopts "d:o:n:pc" opt; do + case $opt in + d) dataset_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + p) parse_depths=true ;; + c) parse_pointclouds=true ;; + *) echo "Usage: $0 -d -o [-n ] [-p] [-c]"; exit 1 ;; + esac +done + +if [ -z "$dataset_root" ] || [ -z "$output_root" ]; then + echo "Usage: $0 -d -o [-n ] [-p] [-c]" + exit 1 +fi + +for i in $(seq 0 $((num_workers - 1))); do + cmd="python pointcept/datasets/preprocessing/concerto/hm3d/preprocess_hm3d.py --worker_id $i \ + --num_workers $num_workers \ + --dataset_root $dataset_root \ + --output_root $output_root" + + if $parse_depths; then + cmd="$cmd --parse_depths" + fi + + if $parse_pointclouds; then + cmd="$cmd --parse_pointclouds" + fi + + eval "$cmd &" +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/hm3d/splits.py b/pointcept/datasets/preprocessing/concerto/hm3d/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..4577685cafa8ad116ea623a513710b72c316b34f --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/hm3d/splits.py @@ -0,0 +1,60 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["train", "val"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split).replace(dataset_path, "data/hm3d") + split_names = [f.name for f in os.scandir(im_split_path) if f.is_dir()] + split_dict = {} + for name in split_names: + im_split_name_path = os.path.join(im_split_path, name, "color") + co_split_name_path = os.path.join(im_split_path, name, "correspondence") + png_files = [ + f + for f in os.listdir(im_split_name_path) + if f.endswith(".png") and (not f.startswith(".")) + ] + png_files = sorted(png_files, key=lambda x: int(x.split(".")[0])) + # Get the full paths of the .png files + png_file_paths = [ + os.path.join(im_split_name_path, f).replace(dataset_path, "data/hm3d") + for f in png_files + ] + co_file_paths = [ + os.path.join(co_split_name_path, f.replace(".png", ".npy")).replace( + dataset_path, "data/hm3d" + ) + for f in png_files + ] + for i in range(0, len(png_file_paths), 4): + split_dict[f"{name}_{i//4}"] = {} + split_dict[f"{name}_{i//4}"]["pointclouds"] = os.path.join( + pc_split_path, name + ) + split_dict[f"{name}_{i//4}"]["images"] = png_file_paths[i : i + 4] + split_dict[f"{name}_{i//4}"]["correspondences"] = co_file_paths[ + i : i + 4 + ] + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/nuscenes/preprocess_nuscenes_info.py b/pointcept/datasets/preprocessing/concerto/nuscenes/preprocess_nuscenes_info.py new file mode 100644 index 0000000000000000000000000000000000000000..6bc3ab31cf8ac21ed97be15c6851f70913f2b58a --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/nuscenes/preprocess_nuscenes_info.py @@ -0,0 +1,687 @@ +""" +Preprocessing Script for nuScenes Informantion +modified from OpenPCDet (https://github.com/open-mmlab/OpenPCDet) + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +from pathlib import Path +import numpy as np +import argparse +import tqdm +import pickle +from functools import reduce +from pyquaternion import Quaternion +from nuscenes.nuscenes import NuScenes +from nuscenes.utils import splits +from nuscenes.utils.geometry_utils import transform_matrix + +map_name_from_general_to_detection = { + "human.pedestrian.adult": "pedestrian", + "human.pedestrian.child": "pedestrian", + "human.pedestrian.wheelchair": "ignore", + "human.pedestrian.stroller": "ignore", + "human.pedestrian.personal_mobility": "ignore", + "human.pedestrian.police_officer": "pedestrian", + "human.pedestrian.construction_worker": "pedestrian", + "animal": "ignore", + "vehicle.car": "car", + "vehicle.motorcycle": "motorcycle", + "vehicle.bicycle": "bicycle", + "vehicle.bus.bendy": "bus", + "vehicle.bus.rigid": "bus", + "vehicle.truck": "truck", + "vehicle.construction": "construction_vehicle", + "vehicle.emergency.ambulance": "ignore", + "vehicle.emergency.police": "ignore", + "vehicle.trailer": "trailer", + "movable_object.barrier": "barrier", + "movable_object.trafficcone": "traffic_cone", + "movable_object.pushable_pullable": "ignore", + "movable_object.debris": "ignore", + "static_object.bicycle_rack": "ignore", +} + + +cls_attr_dist = { + "barrier": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "bicycle": { + "cycle.with_rider": 2791, + "cycle.without_rider": 8946, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "bus": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 9092, + "vehicle.parked": 3294, + "vehicle.stopped": 3881, + }, + "car": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 114304, + "vehicle.parked": 330133, + "vehicle.stopped": 46898, + }, + "construction_vehicle": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 882, + "vehicle.parked": 11549, + "vehicle.stopped": 2102, + }, + "ignore": { + "cycle.with_rider": 307, + "cycle.without_rider": 73, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 165, + "vehicle.parked": 400, + "vehicle.stopped": 102, + }, + "motorcycle": { + "cycle.with_rider": 4233, + "cycle.without_rider": 8326, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "pedestrian": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 157444, + "pedestrian.sitting_lying_down": 13939, + "pedestrian.standing": 46530, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "traffic_cone": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "trailer": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 3421, + "vehicle.parked": 19224, + "vehicle.stopped": 1895, + }, + "truck": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 21339, + "vehicle.parked": 55626, + "vehicle.stopped": 11097, + }, +} + + +def get_available_scenes(nusc): + available_scenes = [] + for scene in nusc.scene: + scene_token = scene["token"] + scene_rec = nusc.get("scene", scene_token) + sample_rec = nusc.get("sample", scene_rec["first_sample_token"]) + sd_rec = nusc.get("sample_data", sample_rec["data"]["LIDAR_TOP"]) + has_more_frames = True + scene_not_exist = False + while has_more_frames: + lidar_path, boxes, _ = nusc.get_sample_data(sd_rec["token"]) + if not Path(lidar_path).exists(): + scene_not_exist = True + break + else: + break + if scene_not_exist: + continue + available_scenes.append(scene) + return available_scenes + + +def get_sample_data(nusc, sample_data_token, selected_anntokens=None): + """ + Returns the data path as well as all annotations related to that sample_data. + Note that the boxes are transformed into the current sensor"s coordinate frame. + Args: + nusc: + sample_data_token: Sample_data token. + selected_anntokens: If provided only return the selected annotation. + + Returns: + + """ + # Retrieve sensor & pose records + sd_record = nusc.get("sample_data", sample_data_token) + cs_record = nusc.get("calibrated_sensor", sd_record["calibrated_sensor_token"]) + sensor_record = nusc.get("sensor", cs_record["sensor_token"]) + pose_record = nusc.get("ego_pose", sd_record["ego_pose_token"]) + + data_path = nusc.get_sample_data_path(sample_data_token) + + if sensor_record["modality"] == "camera": + cam_intrinsic = np.array(cs_record["camera_intrinsic"]) + else: + cam_intrinsic = None + + # Retrieve all sample annotations and map to sensor coordinate system. + if selected_anntokens is not None: + boxes = list(map(nusc.get_box, selected_anntokens)) + else: + boxes = nusc.get_boxes(sample_data_token) + + # Make list of Box objects including coord system transforms. + box_list = [] + for box in boxes: + box.velocity = nusc.box_velocity(box.token) + # Move box to ego vehicle coord system + box.translate(-np.array(pose_record["translation"])) + box.rotate(Quaternion(pose_record["rotation"]).inverse) + + # Move box to sensor coord system + box.translate(-np.array(cs_record["translation"])) + box.rotate(Quaternion(cs_record["rotation"]).inverse) + + box_list.append(box) + + return data_path, box_list, cam_intrinsic + + +def quaternion_yaw(q: Quaternion) -> float: + """ + Calculate the yaw angle from a quaternion. + Note that this only works for a quaternion that represents a box in lidar or global coordinate frame. + It does not work for a box in the camera frame. + :param q: Quaternion of interest. + :return: Yaw angle in radians. + """ + + # Project into xy plane. + v = np.dot(q.rotation_matrix, np.array([1, 0, 0])) + + # Measure yaw using arctan. + yaw = np.arctan2(v[1], v[0]) + + return yaw + + +def obtain_sensor2top( + nusc, sensor_token, l2e_t, l2e_r_mat, e2g_t, e2g_r_mat, sensor_type="lidar" +): + """Obtain the info with RT matric from general sensor to Top LiDAR. + + Args: + nusc (class): Dataset class in the nuScenes dataset. + sensor_token (str): Sample data token corresponding to the + specific sensor type. + l2e_t (np.ndarray): Translation from lidar to ego in shape (1, 3). + l2e_r_mat (np.ndarray): Rotation matrix from lidar to ego + in shape (3, 3). + e2g_t (np.ndarray): Translation from ego to global in shape (1, 3). + e2g_r_mat (np.ndarray): Rotation matrix from ego to global + in shape (3, 3). + sensor_type (str): Sensor to calibrate. Default: "lidar". + + Returns: + sweep (dict): Sweep information after transformation. + """ + sd_rec = nusc.get("sample_data", sensor_token) + cs_record = nusc.get("calibrated_sensor", sd_rec["calibrated_sensor_token"]) + pose_record = nusc.get("ego_pose", sd_rec["ego_pose_token"]) + data_path = str(nusc.get_sample_data_path(sd_rec["token"])) + # if os.getcwd() in data_path: # path from lyftdataset is absolute path + # data_path = data_path.split(f"{os.getcwd()}/")[-1] # relative path + sweep = { + "data_path": data_path, + "type": sensor_type, + "sample_data_token": sd_rec["token"], + "sensor2ego_translation": cs_record["translation"], + "sensor2ego_rotation": cs_record["rotation"], + "ego2global_translation": pose_record["translation"], + "ego2global_rotation": pose_record["rotation"], + "timestamp": sd_rec["timestamp"], + } + l2e_r_s = sweep["sensor2ego_rotation"] + l2e_t_s = sweep["sensor2ego_translation"] + e2g_r_s = sweep["ego2global_rotation"] + e2g_t_s = sweep["ego2global_translation"] + + # obtain the RT from sensor to Top LiDAR + # sweep->ego->global->ego'->lidar + l2e_r_s_mat = Quaternion(l2e_r_s).rotation_matrix + e2g_r_s_mat = Quaternion(e2g_r_s).rotation_matrix + R = (l2e_r_s_mat.T @ e2g_r_s_mat.T) @ ( + np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T + ) + T = (l2e_t_s @ e2g_r_s_mat.T + e2g_t_s) @ ( + np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T + ) + T -= ( + e2g_t @ (np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T) + + l2e_t @ np.linalg.inv(l2e_r_mat).T + ).squeeze(0) + sweep["sensor2lidar_rotation"] = R.T # points @ R.T + T + sweep["sensor2lidar_translation"] = T + return sweep + + +def find_closest_cam_sd_token(nusc, start_cam_sd_token, target_timestamp): + """Find the camera sample_data closest in time to a LiDAR sweep.""" + start_cam_sd = nusc.get("sample_data", start_cam_sd_token) + closest_cam_sd = start_cam_sd + closest_time_diff = abs(start_cam_sd["timestamp"] - target_timestamp) + + current_cam_sd = start_cam_sd + while current_cam_sd["next"]: + next_cam_sd = nusc.get("sample_data", current_cam_sd["next"]) + time_diff = abs(next_cam_sd["timestamp"] - target_timestamp) + if time_diff >= closest_time_diff: + break + closest_cam_sd = next_cam_sd + closest_time_diff = time_diff + current_cam_sd = next_cam_sd + + current_cam_sd = start_cam_sd + while current_cam_sd["prev"]: + prev_cam_sd = nusc.get("sample_data", current_cam_sd["prev"]) + time_diff = abs(prev_cam_sd["timestamp"] - target_timestamp) + if time_diff >= closest_time_diff: + break + closest_cam_sd = prev_cam_sd + closest_time_diff = time_diff + current_cam_sd = prev_cam_sd + + return closest_cam_sd["token"] + + +def fill_trainval_infos( + data_path, nusc, train_scenes, test=False, max_sweeps=10, with_camera=False +): + train_nusc_infos = [] + val_nusc_infos = [] + progress_bar = tqdm.tqdm( + total=len(nusc.sample), desc="create_info", dynamic_ncols=True + ) + + ref_chan = "LIDAR_TOP" # The radar channel from which we track back n sweeps to aggregate the point cloud. + chan = "LIDAR_TOP" # The reference channel of the current sample_rec that the point clouds are mapped to. + + camera_types = [ + "CAM_FRONT", + "CAM_FRONT_RIGHT", + "CAM_FRONT_LEFT", + "CAM_BACK", + "CAM_BACK_LEFT", + "CAM_BACK_RIGHT", + ] + + for index, sample in enumerate(nusc.sample): + progress_bar.update() + + ref_sd_token = sample["data"][ref_chan] + ref_sd_rec = nusc.get("sample_data", ref_sd_token) + ref_cs_rec = nusc.get( + "calibrated_sensor", ref_sd_rec["calibrated_sensor_token"] + ) + ref_pose_rec = nusc.get("ego_pose", ref_sd_rec["ego_pose_token"]) + ref_time = 1e-6 * ref_sd_rec["timestamp"] + + ref_lidar_path, ref_boxes, _ = get_sample_data(nusc, ref_sd_token) + + ref_cam_front_token = sample["data"]["CAM_FRONT"] + ref_cam_path, _, ref_cam_intrinsic = nusc.get_sample_data(ref_cam_front_token) + + # Homogeneous transform from ego car frame to reference frame + ref_from_car = transform_matrix( + ref_cs_rec["translation"], Quaternion(ref_cs_rec["rotation"]), inverse=True + ) + + # Homogeneous transformation matrix from global to _current_ ego car frame + car_from_global = transform_matrix( + ref_pose_rec["translation"], + Quaternion(ref_pose_rec["rotation"]), + inverse=True, + ) + info = { + "lidar_path": Path(ref_lidar_path).relative_to(data_path).__str__(), + "lidar_token": ref_sd_token, + "cam_front_path": Path(ref_cam_path).relative_to(data_path).__str__(), + "cam_intrinsic": ref_cam_intrinsic, + "token": sample["token"], + "sweeps": [], + "ref_from_car": ref_from_car, + "car_from_global": car_from_global, + "timestamp": ref_time, + } + if with_camera: + info["cams"] = dict() + l2e_r = ref_cs_rec["rotation"] + l2e_t = (ref_cs_rec["translation"],) + e2g_r = ref_pose_rec["rotation"] + e2g_t = ref_pose_rec["translation"] + l2e_r_mat = Quaternion(l2e_r).rotation_matrix + e2g_r_mat = Quaternion(e2g_r).rotation_matrix + + for cam in camera_types: + cam_token = sample["data"][cam] + cam_path, _, camera_intrinsics = nusc.get_sample_data(cam_token) + cam_info = obtain_sensor2top( + nusc, cam_token, l2e_t, l2e_r_mat, e2g_t, e2g_r_mat, cam + ) + cam_info["data_path"] = ( + Path(cam_info["data_path"]).relative_to(data_path).__str__() + ) + cam_info.update(camera_intrinsics=camera_intrinsics) + info["cams"].update({cam: cam_info}) + + sample_data_token = sample["data"][chan] + curr_sd_rec = nusc.get("sample_data", sample_data_token) + + sweeps = [] + while len(sweeps) < max_sweeps - 1: + if curr_sd_rec["prev"] == "": + if len(sweeps) == 0: + sweep = { + "lidar_path": Path(ref_lidar_path) + .relative_to(data_path) + .__str__(), + "sample_data_token": curr_sd_rec["token"], + "transform_matrix": None, + "time_lag": curr_sd_rec["timestamp"] * 0, + "is_key_frame": curr_sd_rec["is_key_frame"], + } + if with_camera: + sweep_sample = nusc.get("sample", curr_sd_rec["sample_token"]) + cam_infos = dict() + for cam in camera_types: + cam_token = find_closest_cam_sd_token( + nusc, + sweep_sample["data"][cam], + curr_sd_rec["timestamp"], + ) + cam_path, _, camera_intrinsics = nusc.get_sample_data( + cam_token + ) + cam_info = obtain_sensor2top( + nusc, + cam_token, + l2e_t, + l2e_r_mat, + e2g_t, + e2g_r_mat, + cam, + ) + cam_info["data_path"] = ( + Path(cam_info["data_path"]) + .relative_to(data_path) + .__str__() + ) + cam_info.update(camera_intrinsics=camera_intrinsics) + cam_infos.update({cam: cam_info}) + sweep.update({"cams": cam_infos}) + sweeps.append(sweep) + else: + sweeps.append(sweeps[-1]) + else: + curr_sd_rec = nusc.get("sample_data", curr_sd_rec["prev"]) + if with_camera: + sweep_sample = nusc.get("sample", curr_sd_rec["sample_token"]) + cam_infos = dict() + for cam in camera_types: + cam_token = find_closest_cam_sd_token( + nusc, + sweep_sample["data"][cam], + curr_sd_rec["timestamp"], + ) + cam_path, _, camera_intrinsics = nusc.get_sample_data(cam_token) + cam_info = obtain_sensor2top( + nusc, cam_token, l2e_t, l2e_r_mat, e2g_t, e2g_r_mat, cam + ) + cam_info["data_path"] = ( + Path(cam_info["data_path"]).relative_to(data_path).__str__() + ) + cam_info.update(camera_intrinsics=camera_intrinsics) + cam_infos.update({cam: cam_info}) + # Get past pose + current_pose_rec = nusc.get("ego_pose", curr_sd_rec["ego_pose_token"]) + global_from_car = transform_matrix( + current_pose_rec["translation"], + Quaternion(current_pose_rec["rotation"]), + inverse=False, + ) + + # Homogeneous transformation matrix from sensor coordinate frame to ego car frame. + current_cs_rec = nusc.get( + "calibrated_sensor", curr_sd_rec["calibrated_sensor_token"] + ) + car_from_current = transform_matrix( + current_cs_rec["translation"], + Quaternion(current_cs_rec["rotation"]), + inverse=False, + ) + + tm = reduce( + np.dot, + [ref_from_car, car_from_global, global_from_car, car_from_current], + ) + + lidar_path = nusc.get_sample_data_path(curr_sd_rec["token"]) + + time_lag = ref_time - 1e-6 * curr_sd_rec["timestamp"] + + sweep = { + "lidar_path": Path(lidar_path).relative_to(data_path).__str__(), + "sample_data_token": curr_sd_rec["token"], + "transform_matrix": tm, + "global_from_car": global_from_car, + "car_from_current": car_from_current, + "time_lag": time_lag, + "is_key_frame": curr_sd_rec["is_key_frame"], + } + if with_camera: + sweep.update({"cams": cam_infos}) + + sweeps.append(sweep) + + info["sweeps"] = sweeps + + assert len(info["sweeps"]) == max_sweeps - 1, ( + f"sweep {curr_sd_rec['token']} only has {len(info['sweeps'])} sweeps, " + f"you should duplicate to sweep num {max_sweeps - 1}" + ) + + if not test: + # processing gt bbox + annotations = [ + nusc.get("sample_annotation", token) for token in sample["anns"] + ] + + # the filtering gives 0.5~1 map improvement + num_lidar_pts = np.array([anno["num_lidar_pts"] for anno in annotations]) + num_radar_pts = np.array([anno["num_radar_pts"] for anno in annotations]) + mask = num_lidar_pts + num_radar_pts > 0 + + locs = np.array([b.center for b in ref_boxes]).reshape(-1, 3) + dims = np.array([b.wlh for b in ref_boxes]).reshape(-1, 3)[ + :, [1, 0, 2] + ] # wlh == > dxdydz (lwh) + velocity = np.array([b.velocity for b in ref_boxes]).reshape(-1, 3) + rots = np.array([quaternion_yaw(b.orientation) for b in ref_boxes]).reshape( + -1, 1 + ) + names = np.array([b.name for b in ref_boxes]) + tokens = np.array([b.token for b in ref_boxes]) + gt_boxes = np.concatenate([locs, dims, rots, velocity[:, :2]], axis=1) + + assert len(annotations) == len(gt_boxes) == len(velocity) + + info["gt_boxes"] = gt_boxes[mask, :] + info["gt_boxes_velocity"] = velocity[mask, :] + info["gt_names"] = np.array( + [map_name_from_general_to_detection[name] for name in names] + )[mask] + info["gt_boxes_token"] = tokens[mask] + info["num_lidar_pts"] = num_lidar_pts[mask] + info["num_radar_pts"] = num_radar_pts[mask] + + # processing gt segment + segment_path = nusc.get("lidarseg", ref_sd_token)["filename"] + info["gt_segment_path"] = segment_path + + if sample["scene_token"] in train_scenes: + train_nusc_infos.append(info) + else: + val_nusc_infos.append(info) + + progress_bar.close() + return train_nusc_infos, val_nusc_infos + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", required=True, help="Path to the nuScenes dataset." + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where processed information located.", + ) + parser.add_argument( + "--max_sweeps", default=10, type=int, help="Max number of sweeps. Default: 10." + ) + parser.add_argument( + "--with_camera", + action="store_true", + default=False, + help="Whether use camera or not.", + ) + config = parser.parse_args() + + print(f"Loading nuScenes tables for version v1.0-trainval...") + nusc_trainval = NuScenes( + version="v1.0-trainval", dataroot=config.dataset_root, verbose=False + ) + available_scenes_trainval = get_available_scenes(nusc_trainval) + available_scene_names_trainval = [s["name"] for s in available_scenes_trainval] + print("total scene num:", len(nusc_trainval.scene)) + print("exist scene num:", len(available_scenes_trainval)) + assert len(available_scenes_trainval) == len(nusc_trainval.scene) == 850 + + print(f"Loading nuScenes tables for version v1.0-test...") + nusc_test = NuScenes( + version="v1.0-test", dataroot=config.dataset_root, verbose=False + ) + available_scenes_test = get_available_scenes(nusc_test) + available_scene_names_test = [s["name"] for s in available_scenes_test] + print("total scene num:", len(nusc_test.scene)) + print("exist scene num:", len(available_scenes_test)) + assert len(available_scenes_test) == len(nusc_test.scene) == 150 + + train_scenes = splits.train + train_scenes = set( + [ + available_scenes_trainval[available_scene_names_trainval.index(s)]["token"] + for s in train_scenes + ] + ) + test_scenes = splits.test + test_scenes = set( + [ + available_scenes_test[available_scene_names_test.index(s)]["token"] + for s in test_scenes + ] + ) + print(f"Filling trainval information...") + train_nusc_infos, val_nusc_infos = fill_trainval_infos( + config.dataset_root, + nusc_trainval, + train_scenes, + test=False, + max_sweeps=config.max_sweeps, + with_camera=config.with_camera, + ) + print(f"Filling test information...") + test_nusc_infos, _ = fill_trainval_infos( + config.dataset_root, + nusc_test, + test_scenes, + test=True, + max_sweeps=config.max_sweeps, + with_camera=config.with_camera, + ) + + print(f"Saving nuScenes information...") + os.makedirs(os.path.join(config.output_root, "info"), exist_ok=True) + print( + f"train sample: {len(train_nusc_infos)}, val sample: {len(val_nusc_infos)}, test sample: {len(test_nusc_infos)}" + ) + with open( + os.path.join( + config.output_root, + "info", + f"nuscenes_infos_{config.max_sweeps}sweeps_train.pkl", + ), + "wb", + ) as f: + pickle.dump(train_nusc_infos, f) + with open( + os.path.join( + config.output_root, + "info", + f"nuscenes_infos_{config.max_sweeps}sweeps_val.pkl", + ), + "wb", + ) as f: + pickle.dump(val_nusc_infos, f) + with open( + os.path.join( + config.output_root, + "info", + f"nuscenes_infos_{config.max_sweeps}sweeps_test.pkl", + ), + "wb", + ) as f: + pickle.dump(test_nusc_infos, f) diff --git a/pointcept/datasets/preprocessing/concerto/re10k/preprocess_re10k.py b/pointcept/datasets/preprocessing/concerto/re10k/preprocess_re10k.py new file mode 100644 index 0000000000000000000000000000000000000000..2f6028e9c5ec11d75df49e55775ab95f558740ae --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/re10k/preprocess_re10k.py @@ -0,0 +1,556 @@ +""" +Preprocessing Script for RE10K using VGGT(https://github.com/facebookresearch/vggt) + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from pathlib import Path +from torchvision.utils import save_image +import torch +from io import BytesIO +from einops import rearrange, repeat +from PIL import Image +from tqdm import tqdm +import torchvision.transforms as tf +import argparse +import os +import cv2 +import numpy as np +from scipy.spatial.transform import Rotation as R +import open3d as o3d +from torchvision.utils import save_image + +from vggt.models.vggt import VGGT +from vggt.utils.pose_enc import pose_encoding_to_extri_intri +import camtools as ct +from torchvision.transforms import transforms as T + +_alignment = 14 +target_size = 518 + +generator = torch.Generator() +generator.manual_seed(123) + + +def extract_and_align_ground_plane( + pcd, + height_percentile=20, + ransac_distance_threshold=0.01, + ransac_n=3, + ransac_iterations=1000, + max_angle_degree=40, + max_trials=6, +): + points = np.asarray(pcd.points) + z_vals = points[:, 2] + z_thresh = np.percentile(z_vals, height_percentile) + low_indices = np.where(z_vals <= z_thresh)[0] + + remaining_indices = low_indices.copy() + + for trial in range(max_trials): + if len(remaining_indices) < ransac_n: + raise ValueError("Not enough points left to fit a plane.") + + low_pcd = pcd.select_by_index(remaining_indices) + + plane_model, inliers = low_pcd.segment_plane( + distance_threshold=ransac_distance_threshold, + ransac_n=ransac_n, + num_iterations=ransac_iterations, + ) + a, b, c, d = plane_model + normal = np.array([a, b, c]) + normal /= np.linalg.norm(normal) + + angle = np.arccos(np.clip(np.dot(normal, [0, 0, 1]), -1.0, 1.0)) * 180 / np.pi + if angle <= max_angle_degree: + inliers_global = remaining_indices[inliers] + + target = np.array([0, 0, 1]) + axis = np.cross(normal, target) + axis_norm = np.linalg.norm(axis) + + if axis_norm < 1e-6: + rotation_matrix = np.eye(3) + else: + axis /= axis_norm + rot_angle = np.arccos(np.clip(np.dot(normal, target), -1.0, 1.0)) + rotation = R.from_rotvec(axis * rot_angle) + rotation_matrix = rotation.as_matrix() + + rotated_points = points @ rotation_matrix.T + ground_points_z = rotated_points[inliers_global, 2] + offset = np.mean(ground_points_z) + rotated_points[:, 2] -= offset + + aligned_pcd = o3d.geometry.PointCloud() + aligned_pcd.points = o3d.utility.Vector3dVector(rotated_points) + if pcd.has_colors(): + aligned_pcd.colors = pcd.colors + if pcd.has_normals(): + rotated_normals = np.asarray(pcd.normals) @ rotation_matrix.T + aligned_pcd.normals = o3d.utility.Vector3dVector(rotated_normals) + + return aligned_pcd, inliers_global, rotation_matrix, offset + + else: + rejected_indices = remaining_indices[inliers] + remaining_indices = np.setdiff1d(remaining_indices, rejected_indices) + + raise ValueError("Failed to find a valid ground plane within max trials.") + + +def rotx(x, theta=90): + """ + Rotate x by theta degrees around the x-axis + """ + theta = np.deg2rad(theta) + rot_matrix = np.array( + [ + [1, 0, 0, 0], + [0, np.cos(theta), -np.sin(theta), 0], + [0, np.sin(theta), np.cos(theta), 0], + [0, 0, 0, 1], + ] + ) + return rot_matrix @ x + + +def Coord2zup(points, extrinsics): + """ + Convert the dust3r coordinate system to the z-up coordinate system + """ + points = np.concatenate([points, np.ones([points.shape[0], 1])], axis=1).T + points = rotx(points, -90)[:3].T + t = np.min(points, axis=0) + points -= t + extrinsics = rotx(extrinsics, -90) + extrinsics[:, :3, 3] -= t.T + return points, extrinsics + + +def resize_images_intrinsic(images, intrinsics, size, crop_size): + h, w = size + crop_h, crop_w = crop_size + intrinsics[:, 0, 0] = crop_w + intrinsics[:, 1, 1] = crop_h + cam_trans = torch.tensor( + [[w / crop_w, 0, 0], [0, h / crop_h, 0], [0, 0, 1]], dtype=torch.float32 + ) + intrinsics = torch.stack([cam_trans @ intrinsics_i for intrinsics_i in intrinsics]) + return images, intrinsics + + +def calDelta(ang1, ang2, dist1, dist2): + alpha = 20 + angD = ang1 - ang2 + distD = dist1 - dist2 + ang_s = np.linalg.norm(angD) + dist_s = np.linalg.norm(distD) + return ang_s + alpha * dist_s, ang_s, dist_s + + +def convert_poses(poses): + b, _ = poses.shape + + # Convert the intrinsics to a 3x3 normalized K matrix. + intrinsics = torch.eye(3, dtype=torch.float32) + intrinsics = repeat(intrinsics, "h w -> b h w", b=b).clone() + fx, fy, cx, cy = poses[:, :4].T + intrinsics[:, 0, 0] = fx + intrinsics[:, 1, 1] = fy + intrinsics[:, 0, 2] = cx + intrinsics[:, 1, 2] = cy + + # Convert the extrinsics to a 4x4 OpenCV-style W2C matrix. + w2c = repeat(torch.eye(4, dtype=torch.float32), "h w -> b h w", b=b).clone() + w2c[:, :3] = rearrange(poses[:, 6:], "b (h w) -> b h w", h=3, w=4) + return w2c.inverse(), intrinsics + + +def convert_images(images): + torch_images = [] + for image in images: + image = Image.open(BytesIO(image.numpy().tobytes())) + torch_images.append(tf.ToTensor()(image)) + return torch.stack(torch_images) + + +def plyGet( + images, + scene_dir, + pc_outputdir, + im_outputdir, + device, + dtype, + vggt_model, + depth_points="points", + conf=0.0, + parse_depths=False, +): + # T0 = T0.numpy() + print(f"Processing scene: {scene_dir}") + height, width = images.shape[-2:] + + # Skip every n frames + print(f"Processing {images.shape[0]} images") + + image_num = images.shape[0] + # Load and preprocess images + images = images.to(device) + + with torch.no_grad(): + with torch.cuda.amp.autocast(dtype=dtype): + # Predict attributes including cameras, depth maps, and point maps. + predictions = vggt_model(images) + + # Convert pose encoding to extrinsic and intrinsic matrices + Ts, Ks = pose_encoding_to_extri_intri(predictions["pose_enc"], images.shape[-2:]) + Ts = Ts[0].detach().cpu().numpy() + Ks = Ks[0].detach().cpu().numpy() + Ts = ct.convert.pad_0001(Ts) + Ts_inv = np.linalg.inv(Ts) + Cs = np.array([ct.convert.T_to_C(T) for T in Ts]) # (n, 3) + + # correspondance + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel_ = [] + for i in range(image_num): + pixel_id = np.hstack((pixel, i * np.ones((pixel.shape[0], 1)))) + pixel_.append(pixel_id) + pixel = np.concatenate(pixel_, axis=0) + + if depth_points == "points": + world_points = predictions["world_points"].detach().cpu().numpy() + world_points_conf = predictions["world_points_conf"].detach().cpu().numpy() + img_depths = predictions["depth"].detach().cpu().numpy().squeeze(-1) + else: + img_depths = predictions["depth"].detach().cpu().numpy().squeeze(-1) + img_depths_conf = predictions["depth_conf"].detach().cpu().numpy() + world_points = [ + ct.project.im_depth_to_point_cloud(img_depth, K, T) + for img_depth, K, T in zip(img_depths[0], Ks, Ts) + ] + world_points = np.stack(world_points).reshape( + 1, image_num, images.shape[-2], images.shape[-1], -1 + ) + world_points_conf = img_depths_conf + + masks = world_points_conf > conf + points_masks = masks.reshape((image_num * images.shape[-2] * images.shape[-1],)) + + # Compute view direction for each pixel + # (b n h w c) - (n, 3) + view_dirs = world_points - rearrange(Cs, "n c -> 1 n 1 1 c") + view_dirs = rearrange(view_dirs, "b n h w c -> (b n h w) c") + view_dirs = view_dirs / np.linalg.norm(view_dirs, axis=-1, keepdims=True) + + # Extract points and colors + # [1, 8, 3, 294, 518] + images = images.detach().cpu().unsqueeze(0).numpy() + points = rearrange(world_points, "b n h w c -> (b n h w) c") + colors = rearrange(images, "b n c h w -> (b n h w) c") + + # masks + points = points[points_masks] + colors = colors[points_masks] + pixel = pixel[points_masks] + pixel = np.concatenate((pixel, np.arange(points.shape[0]).reshape(-1, 1)), axis=-1) + + points, Ts_inv = Coord2zup(points, Ts_inv) + scale = 3 / (points[:, 2].max() - points[:, 2].min()) + points *= scale + Ts_inv[:, :3, 3] *= scale + + # Create a point cloud + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(points) + pcd.colors = o3d.utility.Vector3dVector(colors) + pcd.estimate_normals() + try: + pcd, inliers, rotation_matrix, offset = extract_and_align_ground_plane(pcd) + except: + print("cannot find ground") + return + T_pcd = np.eye(4) + T_pcd[:3, :3] = rotation_matrix + T_pcd[2, 3] = -offset + Ts_inv = T_pcd @ Ts_inv + os.makedirs(pc_outputdir, exist_ok=True) + os.makedirs(im_outputdir, exist_ok=True) + + # Filp normals such that normals always point to camera + # Compute the dot product between the normal and the view direction + # If the dot product is less than 0, flip the normal + normals = np.asarray(pcd.normals) + view_dirs = np.asarray(view_dirs) + view_dirs = view_dirs[points_masks] + dot_product = np.sum(normals * view_dirs, axis=-1) + flip_mask = dot_product > 0 + normals[flip_mask] = -normals[flip_mask] + + # Normalize normals a nd m + normals = normals / np.linalg.norm(normals, axis=-1, keepdims=True) + colors = (colors * 255).astype(np.uint8) + + np.save(Path(pc_outputdir) / "coord.npy", points) + np.save(Path(pc_outputdir) / "color.npy", colors) + np.save(Path(pc_outputdir) / "normal.npy", normals) + + correspondence_path = os.path.join(im_outputdir, "correspondence") + img_output_path = os.path.join(im_outputdir, "color") + Ts_output_path = os.path.join(im_outputdir, "pose") + Ks_output_path = os.path.join(im_outputdir, "intrinsic") + os.makedirs(img_output_path, exist_ok=True) + os.makedirs(Ts_output_path, exist_ok=True) + os.makedirs(Ks_output_path, exist_ok=True) + os.makedirs(correspondence_path, exist_ok=True) + for id in range(image_num): + pixels_correspondence = pixel[pixel[:, 2] == id] + pixels_correspondence = pixels_correspondence[:, [0, 1, 3]] + np.save( + os.path.join(im_outputdir, "correspondence", f"{id}.npy"), + pixels_correspondence, + ) + images = torch.tensor(images.squeeze(0)) + if parse_depths: + depth_output_path = os.path.join(im_outputdir, "depth") + os.makedirs(depth_output_path, exist_ok=True) + img_depths = img_depths.squeeze(0)[..., np.newaxis] * scale * 1000 + for id in range(images.shape[0]): + save_image(images[id], os.path.join(img_output_path, f"{id}.png")) + np.save(os.path.join(Ts_output_path, f"{id}.npy"), Ts_inv[id]) + np.save(os.path.join(Ks_output_path, f"{id}.npy"), Ks[id]) + if parse_depths: + cv2.imwrite( + os.path.join(depth_output_path, f"{id}.png"), + img_depths[id].astype(np.uint16), + ) + + +def parse_scene( + chunk_path, + vggt_model, + num_context_views, + outputdir, + device, + frame_gap, + overlap_range, + dtype, + conf, + parse_depths, +): + print("loading from ", chunk_path) + split = chunk_path.parts[-2] + # Load the chunk. + chunk = torch.load(chunk_path) + for run_idx in range(len(chunk)): + example = chunk[run_idx] + extrinsics, intrinsics = convert_poses(example["cameras"]) + scene = example["key"] + + # Load the images. + context_images = example["images"] + context_images = convert_images(context_images) + + v, _, _ = extrinsics.shape + ang_list = np.array( + [ + R.from_matrix(ex_i[:3, :3].cpu().numpy()).as_euler("xyz", degrees=True) + for ex_i in extrinsics + ] + ) + dist_list = np.array(extrinsics[:, :3, 3].reshape((-1, 3)).cpu()) + context_indices = torch.randperm(v, generator=generator) + for context_index in tqdm(context_indices, "Finding context pair"): + choose_index = [context_index.item()] + while len(choose_index) < num_context_views: + # Step away from context view until the minimum overlap threshold is met. + valid_indices = [] + for step in (1, -1): + if step == 1: + context_index = max(choose_index) + else: + context_index = min(choose_index) + min_distance = frame_gap[0] + max_distance = frame_gap[1] + current_index = context_index + step * min_distance + + while 0 <= current_index < v: + # Compute overlap. + overlap, overlap_a, overlap_b = calDelta( + ang_list[context_index], + ang_list[current_index], + dist_list[context_index], + dist_list[current_index], + ) + # print("similarity",overlap_a,overlap_b) + min_overlap = overlap_range[0] + max_overlap = overlap_range[1] + if min_overlap <= overlap <= max_overlap: + valid_indices.append((current_index, overlap_a, overlap_b)) + delta = np.abs(current_index - context_index) + + # Stop once the camera has panned away too much. + if overlap < min_overlap or delta > max_distance: + break + + current_index += step + if valid_indices: + # Pick a random valid view. Index the resulting views. + num_options = len(valid_indices) + chosen = torch.randint( + 0, num_options, size=tuple(), generator=generator + ) + chosen, overlap_a, overlap_b = valid_indices[chosen] + choose_index.append(chosen) + else: + break + if len(choose_index) < num_context_views: + continue + context_left = min(choose_index) + context_right = max(choose_index) + delta = context_right - context_left + choose_index.sort() + # Pick non-repeated random target views. + + extrinsics = extrinsics[choose_index] + intrinsics = intrinsics[choose_index] + img_height, img_width = context_images.shape[-2:] + img_width_new = target_size + img_height_new = ( + round(img_height * (img_width_new / img_width) / _alignment) + * _alignment + ) + transform = T.Compose( + [ + T.Resize((img_height_new, img_width_new)), + ] + ) + imgs_list = transform(context_images) + if img_height_new > target_size: + start_y = (img_height_new - target_size) // 2 + context_images = context_images[ + :, :, start_y : start_y + target_size, : + ] + imgs_list, intrinsics_list = resize_images_intrinsic( + imgs_list, + intrinsics, + (img_height_new, img_width_new), + (target_size, target_size), + ) + pc_outputdir = os.path.join(outputdir, split, scene) + im_outputdir = os.path.join(outputdir, "images", split, scene) + plyGet( + imgs_list[choose_index], + scene, + pc_outputdir, + im_outputdir, + device, + dtype, + vggt_model, + depth_points="points", + conf=conf, + parse_depths=parse_depths, + ) + break + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--num_context_views", + default=4, + type=int, + help="num_context_views", + ) + parser.add_argument( + "--frame_gap", + default=[15, 135], + type=int, + help="frame_gap", + ) + parser.add_argument( + "--overlap_range", + default=[5, 40], + type=int, + help="overlap_range", + ) + parser.add_argument( + "--conf", + default=0.0, + type=float, + help="overlap_range", + ) + parser.add_argument( + "--num_workers", + default=1, + type=int, + help="num_workers", + ) + parser.add_argument( + "--thread_id", + default=0, + type=int, + help="thread_id", + ) + parser.add_argument( + "--device", + default="cuda", + type=str, + help="device", + ) + parser.add_argument( + "--splits", + default=["train", "test"], + nargs="+", + choices=["train", "test"], + help="Splits need to process.", + ) + parser.add_argument( + "--parse_depths", action="store_true", help="Whether parse depths" + ) + cfg = parser.parse_args() + VGGT_model = VGGT().to(cfg.device) + _URL = "https://huggingface.co/facebook/VGGT-1B/resolve/main/model.pt" + state_dict = torch.hub.load_state_dict_from_url(_URL) + VGGT_model.load_state_dict(state_dict) + dtype = ( + torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16 + ) + + chunk_paths = [] + for split in cfg.splits: + root = Path(cfg.dataset_root) / split + root_chunks = [entry for entry in root.iterdir() if ".json" not in str(entry)] + root_chunks = sorted(root_chunks) + chunk_paths.extend(root_chunks) + chunk_paths_list = np.array_split(chunk_paths, cfg.num_workers) + chunk_paths_ = chunk_paths_list[cfg.thread_id] + + for chunk_path in chunk_paths_: + parse_scene( + chunk_path, + VGGT_model, + cfg.num_context_views, + cfg.output_root, + cfg.device, + cfg.frame_gap, + cfg.overlap_range, + dtype, + cfg.conf, + cfg.parse_depths, + ) diff --git a/pointcept/datasets/preprocessing/concerto/re10k/splits.py b/pointcept/datasets/preprocessing/concerto/re10k/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..830964c446eec27e9d87ff1c03a79544afb72c70 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/re10k/splits.py @@ -0,0 +1,57 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["train", "test"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split).replace( + dataset_path, "data/re10k_align" + ) + split_names = [f.name for f in os.scandir(im_split_path) if f.is_dir()] + split_dict = {} + for name in split_names: + im_split_name_path = os.path.join(im_split_path, name, "color") + co_split_name_path = os.path.join(im_split_path, name, "correspondence") + png_files = [ + f for f in os.listdir(im_split_name_path) if f.endswith(".png") + ] + png_files = sorted(png_files, key=lambda x: int(x.split(".")[0])) + # Get the full paths of the .png files + png_file_paths = [ + os.path.join(im_split_name_path, f).replace( + dataset_path, "data/re10k_align" + ) + for f in png_files + ] + co_file_paths = [ + os.path.join(co_split_name_path, f.replace(".png", ".npy")).replace( + dataset_path, "data/re10k_align" + ) + for f in png_files + ] + split_dict[f"{name}"] = {} + split_dict[f"{name}"]["pointclouds"] = os.path.join(pc_split_path, name) + split_dict[f"{name}"]["images"] = png_file_paths + split_dict[f"{name}"]["correspondences"] = co_file_paths + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the re10k dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.py b/pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.py new file mode 100644 index 0000000000000000000000000000000000000000..072ee53916ae805ce46a3e6e80e07e5f198cadd4 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.py @@ -0,0 +1,472 @@ +""" +Preprocessing Script for S3DIS + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import argparse +import glob +import numpy as np +import re +import json +from PIL import Image +from scipy.spatial import cKDTree +from pathlib import Path +import camtools as ct +import shutil + +try: + import open3d +except ImportError: + import warnings + + warnings.warn("Please install open3d for parsing normal") + +try: + import trimesh +except ImportError: + import warnings + + warnings.warn("Please install trimesh for parsing normal") + +area_mesh_dict = {} + + +def correspondenceGet(mesh, K, T, img_size, coords_gt): + height, width = img_size + if np.isnan(T).any() or np.isinf(T).any(): + return None + depth = ct.raycast.mesh_to_im_depth( + mesh=mesh, K=K, T=np.linalg.inv(T), height=height, width=width + ) + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + depth = depth.reshape((-1, 1)) + valid = ~np.isinf(depth).squeeze(-1) + coord = np.zeros_like(pixel, dtype=np.float32) + coord[valid] = depth[valid] * (np.linalg.inv(K) @ pixel[valid].T).T # coord_camera + coord[valid] = coord[valid] @ T[:3, :3].T + T[:3, 3] # column then row + pixel = pixel[valid] + coord = coord[valid] + if coord.shape[0] == 0: + return None + pixel = pixel[:, :2] + coord_dict = {"pixel": pixel, "coord": coord} + return coord_dict + + +def correspondenceSave(mesh, output_dir, img_size, coords_gt, Ks, Ts): + output_dir = Path(output_dir) + (output_dir / "correspondence").mkdir(exist_ok=True) + Ks = Ks[:, :3, :3] + coords_gt_ = coords_gt + pixels_ = [] + coords_ = [] + index_gt = [ + img_name.split(".")[0] + for img_name in os.listdir(str(output_dir / "color")) + if img_name.endswith(".png") + ] + index_gt.sort(key=lambda x: int(x.split("_frame_")[1].split("_domain_")[0])) + for i, (K, T) in enumerate(zip(Ks, Ts)): + coord_dict = correspondenceGet(mesh, K, T, img_size, coords_gt) + if coord_dict is None: + correspondences = -np.ones((1, 3)) + else: + pixels_ = coord_dict["pixel"] + coords_ = coord_dict["coord"] + + tree = cKDTree(coords_gt_) + dis, idx = tree.query(coords_, k=1) + idx_valid = idx[dis < 0.01] + pixels_valid = pixels_[dis < 0.01] + correspondences = np.hstack((pixels_valid, idx_valid.reshape(-1, 1))) + np.save( + output_dir / "correspondence" / "{}.npy".format(index_gt[i]), + correspondences, + ) + + +def parse_room( + room, + angle, + rgb_gap, + raw_root, + dataset_root, + output_root, + align_angle=True, + parse_normal=False, + parse_pointclouds=False, + parse_depths=False, +): + print("Parsing: {}".format(room)) + classes = [ + "ceiling", + "floor", + "wall", + "beam", + "column", + "window", + "door", + "table", + "chair", + "sofa", + "bookcase", + "board", + "clutter", + ] + class2label = {cls: i for i, cls in enumerate(classes)} + source_dir = os.path.join(dataset_root, room) + output_root_pointclouds = output_root + output_root_imgs = os.path.join(output_root, "images") + os.makedirs(output_root_imgs, exist_ok=True) + save_path_imgs = os.path.join(output_root_imgs, room) + os.makedirs(save_path_imgs, exist_ok=True) + object_path_list = sorted(glob.glob(os.path.join(source_dir, "Annotations/*.txt"))) + + room_coords = [] + room_colors = [] + room_normals = [] + room_semantic_gt = [] + room_instance_gt = [] + + for object_id, object_path in enumerate(object_path_list): + object_name = os.path.basename(object_path).split("_")[0] + obj = np.loadtxt(object_path) + coords = obj[:, :3] + colors = obj[:, 3:6] + # note: in some room there is 'stairs' class + class_name = object_name if object_name in classes else "clutter" + semantic_gt = np.repeat(class2label[class_name], coords.shape[0]) + semantic_gt = semantic_gt.reshape([-1, 1]) + instance_gt = np.repeat(object_id, coords.shape[0]) + instance_gt = instance_gt.reshape([-1, 1]) + + room_coords.append(coords) + room_colors.append(colors) + room_semantic_gt.append(semantic_gt) + room_instance_gt.append(instance_gt) + + room_coords = np.ascontiguousarray(np.vstack(room_coords)) + room_colors = np.ascontiguousarray(np.vstack(room_colors)) + + if parse_normal: + x_min, z_max, y_min = np.min(room_coords, axis=0) + x_max, z_min, y_max = np.max(room_coords, axis=0) + z_max = -z_max + z_min = -z_min + max_bound = np.array([x_max, y_max, z_max]) + 0.1 + min_bound = np.array([x_min, y_min, z_min]) - 0.1 + bbox = open3d.geometry.AxisAlignedBoundingBox( + min_bound=min_bound, max_bound=max_bound + ) + # crop room + room_mesh = ( + area_mesh_dict[os.path.dirname(room)] + .crop(bbox) + .transform( + np.array([[1, 0, 0, 0], [0, 0, -1, 0], [0, 1, 0, 0], [0, 0, 0, 1]]) + ) + ) + vertices = np.array(room_mesh.vertices) + faces = np.array(room_mesh.triangles) + vertex_normals = np.array(room_mesh.vertex_normals) + o3d_room_mesh = room_mesh + room_mesh = trimesh.Trimesh( + vertices=vertices, faces=faces, vertex_normals=vertex_normals + ) + closest_points, distances, face_id = room_mesh.nearest.on_surface(room_coords) + room_normals = room_mesh.face_normals[face_id] + Ks = [] + Ts = [] + + area = room.split("/")[0].lower() + part = room.split("/")[1] + if area == "area_5": + camera_pose_paths = sorted( + glob.glob( + os.path.join(raw_root, f"{area}*", "data", "pose", f"*_{part}_*.json") + ) + ) + camera_rgb_paths = sorted( + glob.glob( + os.path.join(raw_root, f"{area}*", "data", "rgb", f"*_{part}_*.png") + ) + ) + camera_depth_paths = sorted( + glob.glob( + os.path.join(raw_root, f"{area}*", "data", "depth", f"*_{part}_*.png") + ) + ) + else: + camera_pose_paths = sorted( + glob.glob(os.path.join(raw_root, area, "data", "pose", f"*_{part}_*.json")) + ) + camera_rgb_paths = sorted( + glob.glob(os.path.join(raw_root, area, "data", "rgb", f"*_{part}_*.png")) + ) + camera_depth_paths = sorted( + glob.glob(os.path.join(raw_root, area, "data", "depth", f"*_{part}_*.png")) + ) + camera_pose_names = set( + [i.split(f"_{part}_")[0].split("camera_")[1] for i in camera_pose_paths] + ) + camera_rgb_names = set( + [i.split(f"_{part}_")[0].split("camera_")[1] for i in camera_rgb_paths] + ) + camera_names = camera_pose_names.intersection(camera_rgb_names) + for camera_name in camera_names: + camera_pose_paths_withname = [ + path for path in camera_pose_paths if camera_name in path + ] + camera_rgb_paths_withname = [ + path for path in camera_rgb_paths if camera_name in path + ] + camera_depth_paths_withname = [ + path for path in camera_depth_paths if camera_name in path + ] + pose_frame_ids = set( + [re.search(r"frame_(\d+)", s).group(1) for s in camera_pose_paths_withname] + ) + rgb_frame_ids = set( + [re.search(r"frame_(\d+)", s).group(1) for s in camera_rgb_paths_withname] + ) + frame_ids = pose_frame_ids.intersection(rgb_frame_ids) + frame_ids_selected = sorted(list(map(int, list(frame_ids))))[::rgb_gap] + camera_pose_prefix = camera_pose_paths_withname[0].split("frame_")[0] + "frame_" + camera_pose_postfix = ( + "_domain" + camera_pose_paths_withname[0].split("_domain")[1] + ) + camera_pose_path_selected = [ + camera_pose_prefix + str(id) + camera_pose_postfix + for id in frame_ids_selected + ] + camera_rgb_prefix = camera_rgb_paths_withname[0].split("frame_")[0] + "frame_" + camera_rgb_postfix = ( + "_domain" + camera_rgb_paths_withname[0].split("_domain")[1] + ) + camera_rgb_path_selected = [ + camera_rgb_prefix + str(id) + camera_rgb_postfix + for id in frame_ids_selected + ] + camera_depth_prefix = ( + camera_depth_paths_withname[0].split("frame_")[0] + "frame_" + ) + camera_depth_postfix = ( + "_domain" + camera_depth_paths_withname[0].split("_domain")[1] + ) + camera_depth_path_selected = [ + camera_depth_prefix + str(id) + camera_depth_postfix + for id in frame_ids_selected + ] + image_shape = Image.open(camera_rgb_path_selected[0]).size + save_path_imgs_camera = os.path.join(save_path_imgs, camera_name) + imgs_output_path = os.path.join(save_path_imgs_camera, "color") + os.makedirs(imgs_output_path, exist_ok=True) + for img_path in camera_rgb_path_selected: + shutil.copy2(img_path, imgs_output_path) + if parse_depths: + depths_output_path = os.path.join(save_path_imgs_camera, "depth") + os.makedirs(depths_output_path, exist_ok=True) + for depth_path in camera_depth_path_selected: + shutil.copy2(depth_path, depths_output_path) + Ks = [] + Ts = [] + Ks_output_path = os.path.join(save_path_imgs_camera, "intrinsic") + Ts_output_path = os.path.join(save_path_imgs_camera, "pose") + os.makedirs(Ks_output_path, exist_ok=True) + os.makedirs(Ts_output_path, exist_ok=True) + for id, posepath in enumerate(camera_pose_path_selected): + with open(posepath, "r") as f: + data = json.load(f) + k_matrix = np.array(data["camera_k_matrix"]) + rt_matrix = np.array(data["camera_rt_matrix"]) + T_matrix = np.eye(4) + T_matrix[:3, :] = rt_matrix + Ks.append(k_matrix) + Ts.append(np.linalg.inv(T_matrix)) + np.save(os.path.join(Ks_output_path, f"{id}.npy"), k_matrix) + np.save(os.path.join(Ts_output_path, f"{id}.npy"), T_matrix) + Ks = np.array(Ks) + Ts = np.array(Ts) + correspondenceSave( + o3d_room_mesh, + save_path_imgs_camera, + (image_shape[1], image_shape[0]), + room_coords, + Ks, + Ts, + ) + + if align_angle: + angle = (2 - angle / 180) * np.pi + rot_cos, rot_sin = np.cos(angle), np.sin(angle) + rot_t = np.array([[rot_cos, -rot_sin, 0], [rot_sin, rot_cos, 0], [0, 0, 1]]) + room_center = (np.max(room_coords, axis=0) + np.min(room_coords, axis=0)) / 2 + room_coords = (room_coords - room_center) @ np.transpose(rot_t) + room_center + if parse_normal: + room_normals = room_normals @ np.transpose(rot_t) + + if parse_pointclouds: + os.makedirs(output_root_pointclouds, exist_ok=True) + save_path_pointclouds = os.path.join(output_root_pointclouds, room) + os.makedirs(save_path_pointclouds, exist_ok=True) + room_semantic_gt = np.ascontiguousarray(np.vstack(room_semantic_gt)) + room_instance_gt = np.ascontiguousarray(np.vstack(room_instance_gt)) + np.save( + os.path.join(save_path_pointclouds, "coord.npy"), + room_coords.astype(np.float32), + ) + np.save( + os.path.join(save_path_pointclouds, "color.npy"), + room_colors.astype(np.uint8), + ) + np.save( + os.path.join(save_path_pointclouds, "segment.npy"), + room_semantic_gt.astype(np.int16), + ) + np.save( + os.path.join(save_path_pointclouds, "instance.npy"), + room_instance_gt.astype(np.int16), + ) + + if parse_normal: + np.save( + os.path.join(save_path_pointclouds, "normal.npy"), + room_normals.astype(np.float32), + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--splits", + required=True, + nargs="+", + choices=["Area_1", "Area_2", "Area_3", "Area_4", "Area_5", "Area_6"], + help="Splits need to process ([Area_1, Area_2, Area_3, Area_4, Area_5, Area_6]).", + ) + parser.add_argument( + "--dataset_root", required=True, help="Path to Stanford3dDataset_v1.2 dataset" + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where area folders will be located", + ) + parser.add_argument( + "--pointclouds_root", + default="data/s3dis", + type=str, + help="Input path where previous pointclouds folder located", + ) + parser.add_argument( + "--raw_root", + default=None, + help="Path to Stanford2d3dDataset_noXYZ dataset (optional)", + ) + parser.add_argument( + "--rgb_gap", + default=5, + help="gap of rgb (optional)", + ) + parser.add_argument( + "--align_angle", action="store_true", help="Whether align room angles" + ) + parser.add_argument( + "--parse_normal", action="store_true", help="Whether process normal" + ) + parser.add_argument( + "--parse_pointclouds", action="store_true", help="Whether parse point clouds" + ) + parser.add_argument( + "--parse_depths", action="store_true", help="Whether parse depths" + ) + parser.add_argument( + "--num_workers", default=1, type=int, help="Num workers for preprocessing." + ) + parser.add_argument( + "--thread_id", + default=0, + type=int, + help="Thread id for parallel processing", + ) + args = parser.parse_args() + + if args.parse_normal: + assert args.raw_root is not None + + room_list = [] + angle_list = [] + + # Load room information + print("Loading room information ...") + for split in args.splits: + area_info = np.loadtxt( + os.path.join( + args.dataset_root, + split, + f"{split}_alignmentAngle.txt", + ), + dtype=str, + ) + room_list += [os.path.join(split, room_info[0]) for room_info in area_info] + angle_list += [int(room_info[1]) for room_info in area_info] + + if args.parse_normal: + # load raw mesh file to extract normal + print("Loading raw mesh file ...") + for split in args.splits: + if split != "Area_5": + mesh_dir = os.path.join(args.raw_root, split.lower(), "3d", "rgb.obj") + mesh = open3d.io.read_triangle_mesh(mesh_dir) + mesh.triangle_uvs.clear() + # trimesh_mesh = trimesh.load(mesh_dir, process=False) + else: + mesh_a_dir = os.path.join( + args.raw_root, f"{split.lower()}a", "3d", "rgb.obj" + ) + mesh_b_dir = os.path.join( + args.raw_root, f"{split.lower()}b", "3d", "rgb.obj" + ) + mesh_a = open3d.io.read_triangle_mesh(mesh_a_dir) + mesh_a.triangle_uvs.clear() + # trimesh_mesh_a = trimesh.load(mesh_a_dir, process=False) + mesh_b = open3d.io.read_triangle_mesh(mesh_b_dir) + mesh_b.triangle_uvs.clear() + # trimesh_mesh_b = trimesh.load(mesh_b_dir, process=False) + mesh_b = mesh_b.transform( + np.array( + [ + [0, 0, -1, -4.09703582], + [0, 1, 0, 0], + [1, 0, 0, -6.22617759], + [0, 0, 0, 1], + ] + ) + ) + mesh = mesh_a + mesh_b + # trimesh_mesh = trimesh_mesh_a +trimesh_mesh_b + area_mesh_dict[split] = mesh + # trimesh_area_mesh_dict[split] = trimesh_mesh + print(f"{split} mesh is loaded") + + room_list_list = np.array_split(room_list, args.num_workers) + room_list_ = room_list_list[args.thread_id] + + for i in range(len(room_list_)): + parse_room( + room_list_[i], + angle_list[i], + args.rgb_gap, + args.raw_root, + args.dataset_root, + args.output_root, + args.align_angle, + args.parse_normal, + args.parse_pointclouds, + args.parse_depths, + ) diff --git a/pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.sh b/pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.sh new file mode 100644 index 0000000000000000000000000000000000000000..8e54eebf82a8d0e38d8a37e18203ace79a7c3c8f --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.sh @@ -0,0 +1,45 @@ +#!/bin/bash + +dataset_root="" +output_root="" +num_workers=6 +parse_depths=false +parse_pointclouds=false + +while getopts "d:o:n:r:pc" opt; do + case $opt in + d) dataset_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + r) raw_root=$OPTARG ;; + p) parse_depths=true ;; + c) parse_pointclouds=true ;; + *) echo "Usage: $0 -d -o -r [-n ] [-p] [-c]"; exit 1 ;; + esac +done + +if [ -z "$dataset_root" ] || [ -z "$output_root" ] || [ -z "$raw_root" ]; then + echo "Usage: $0 -d -o -r [-n ] [-p] [-c]" + exit 1 +fi + +for i in $(seq 1 $((num_workers))); do + cmd="python pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.py --splits Area_$i \ + --num_workers 1 \ + --thread_id 0 \ + --dataset_root $dataset_root \ + --output_root $output_root \ + --raw_root $raw_root \ + --parse_normal" + if $parse_depths; then + cmd="$cmd --parse_depths" + fi + + if $parse_pointclouds; then + cmd="$cmd --parse_pointclouds" + fi + + eval "$cmd &" +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/s3dis/splits.py b/pointcept/datasets/preprocessing/concerto/s3dis/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..056585e1a2de721310a87070e0a6b021f85f6d66 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/s3dis/splits.py @@ -0,0 +1,74 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["Area_1", "Area_2", "Area_3", "Area_4", "Area_5", "Area_6"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split).replace(dataset_path, "data/s3dis") + split_names = [f.name for f in os.scandir(im_split_path) if f.is_dir()] + split_dict = {} + for name in split_names: + cam_split_name_path = os.path.join(im_split_path, name) + cam_names = [f.name for f in os.scandir(cam_split_name_path) if f.is_dir()] + for cam_name in cam_names: + im_split_name_path = os.path.join( + cam_split_name_path, cam_name, "color" + ) + co_split_name_path = os.path.join( + cam_split_name_path, cam_name, "correspondence" + ) + png_files = [ + f for f in os.listdir(im_split_name_path) if f.endswith(".png") + ] + png_files = sorted( + png_files, + key=lambda x: int( + x.split(".")[0].split("_frame_")[1].split("_domain_")[0] + ), + ) + # Get the full paths of the .png files + png_file_paths = [ + os.path.join(im_split_name_path, f).replace( + dataset_path, "data/s3dis" + ) + for f in png_files + ] + co_file_paths = [ + os.path.join(co_split_name_path, f.replace(".png", ".npy")).replace( + dataset_path, "data/s3dis" + ) + for f in png_files + ] + for i in range(0, len(png_file_paths), 4): + split_dict[f"{name}_{cam_name}_{i//4}"] = {} + split_dict[f"{name}_{cam_name}_{i//4}"]["pointclouds"] = ( + os.path.join(pc_split_path, name) + ) + split_dict[f"{name}_{cam_name}_{i//4}"]["images"] = png_file_paths[ + i : i + 4 + ] + split_dict[f"{name}_{cam_name}_{i//4}"]["correspondences"] = ( + co_file_paths[i : i + 4] + ) + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the S3DIS dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.py b/pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.py new file mode 100644 index 0000000000000000000000000000000000000000..95c6f769259b98c6a4619d8c7114015e6af7aee2 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.py @@ -0,0 +1,574 @@ +""" +Preprocessing Script for ScanNet 20/200 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import warnings + +warnings.filterwarnings("ignore", category=DeprecationWarning) + +import os +import camtools as ct +import open3d as o3d +from scipy.spatial import cKDTree +import struct +import zlib +import imageio +import cv2 +import argparse +import glob +import json +import plyfile +import numpy as np +import pandas as pd +import multiprocessing as mp +from pathlib import Path + +# Load external constants +import sys + +sys.path.append("pointcept/datasets/preprocessing/scannet/meta_data") +from scannet200_constants import VALID_CLASS_IDS_200, VALID_CLASS_IDS_20 + +CLOUD_FILE_PFIX = "_vh_clean_2" +SEGMENTS_FILE_PFIX = ".0.010000.segs.json" +AGGREGATIONS_FILE_PFIX = ".aggregation.json" +CLASS_IDS200 = VALID_CLASS_IDS_200 +CLASS_IDS20 = VALID_CLASS_IDS_20 +IGNORE_INDEX = -1 + +COMPRESSION_TYPE_COLOR = {-1: "unknown", 0: "raw", 1: "png", 2: "jpeg"} +COMPRESSION_TYPE_DEPTH = { + -1: "unknown", + 0: "raw_ushort", + 1: "zlib_ushort", + 2: "occi_ushort", +} + + +class RGBDFrame: + def load(self, file_handle): + self.camera_to_world = np.asarray( + struct.unpack("f" * 16, file_handle.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.timestamp_color = struct.unpack("Q", file_handle.read(8))[0] + self.timestamp_depth = struct.unpack("Q", file_handle.read(8))[0] + self.color_size_bytes = struct.unpack("Q", file_handle.read(8))[0] + self.depth_size_bytes = struct.unpack("Q", file_handle.read(8))[0] + self.color_data = b"".join( + struct.unpack( + "c" * self.color_size_bytes, file_handle.read(self.color_size_bytes) + ) + ) + self.depth_data = b"".join( + struct.unpack( + "c" * self.depth_size_bytes, file_handle.read(self.depth_size_bytes) + ) + ) + + def decompress_depth(self, compression_type): + if compression_type == "zlib_ushort": + return self.decompress_depth_zlib() + else: + raise + + def decompress_depth_zlib(self): + return zlib.decompress(self.depth_data) + + def decompress_color(self, compression_type): + if compression_type == "jpeg": + return self.decompress_color_jpeg() + else: + raise + + def decompress_color_jpeg(self): + return imageio.imread(self.color_data) + + +class SensorData: + def __init__(self, filename): + self.version = 4 + self.load(filename) + + def load(self, filename): + with open(filename, "rb") as f: + version = struct.unpack("I", f.read(4))[0] + assert self.version == version + strlen = struct.unpack("Q", f.read(8))[0] + self.sensor_name = b"".join(struct.unpack("c" * strlen, f.read(strlen))) + self.intrinsic_color = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.extrinsic_color = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.intrinsic_depth = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.extrinsic_depth = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.color_compression_type = COMPRESSION_TYPE_COLOR[ + struct.unpack("i", f.read(4))[0] + ] + self.depth_compression_type = COMPRESSION_TYPE_DEPTH[ + struct.unpack("i", f.read(4))[0] + ] + self.color_width = struct.unpack("I", f.read(4))[0] + self.color_height = struct.unpack("I", f.read(4))[0] + self.depth_width = struct.unpack("I", f.read(4))[0] + self.depth_height = struct.unpack("I", f.read(4))[0] + self.depth_shift = struct.unpack("f", f.read(4))[0] + num_frames = struct.unpack("Q", f.read(8))[0] + self.frames = [] + for i in range(num_frames): + frame = RGBDFrame() + frame.load(f) + self.frames.append(frame) + + def export_depth_images(self, output_path, image_size=None, frame_skip=1): + if not os.path.exists(output_path): + os.makedirs(output_path) + print( + "exporting", len(self.frames) // frame_skip, " depth frames to", output_path + ) + for f in range(0, len(self.frames), frame_skip): + if os.path.exists((os.path.join(output_path, str(f) + ".png"))): + continue + if f % 100 == 0: + print( + "exporting", + f, + "th depth frames to", + os.path.join(output_path, str(f) + ".png"), + ) + + depth_data = self.frames[f].decompress_depth(self.depth_compression_type) + depth = np.fromstring(depth_data, dtype=np.uint16).reshape( + self.depth_height, self.depth_width + ) + if image_size is not None: + depth = cv2.resize( + depth, + (image_size[1], image_size[0]), + interpolation=cv2.INTER_NEAREST, + ) + imageio.imwrite(os.path.join(output_path, str(f) + ".png"), depth) + + def export_color_images(self, output_path, image_size=None, frame_skip=1): + if not os.path.exists(output_path): + os.makedirs(output_path) + print( + "exporting", len(self.frames) // frame_skip, "color frames to", output_path + ) + for f in range(0, len(self.frames), frame_skip): + if os.path.exists((os.path.join(output_path, str(f) + ".png"))): + continue + if f % 100 == 0: + print( + "exporting", + f, + "th color frames to", + os.path.join(output_path, str(f) + ".png"), + ) + color = self.frames[f].decompress_color(self.color_compression_type) + if image_size is not None: + color = cv2.resize( + color, + (image_size[1], image_size[0]), + interpolation=cv2.INTER_NEAREST, + ) + # imageio.imwrite(os.path.join(output_path, str(f) + '.jpg'), color) + imageio.imwrite(os.path.join(output_path, str(f) + ".png"), color) + + def save_mat_to_file(self, matrix, filename): + with open(filename, "w") as f: + for line in matrix: + np.savetxt(f, line[np.newaxis], fmt="%f") + + def export_poses(self, output_path, frame_skip=1): + if not os.path.exists(output_path): + os.makedirs(output_path) + print( + "exporting", len(self.frames) // frame_skip, "camera poses to", output_path + ) + for f in range(0, len(self.frames), frame_skip): + np.save( + os.path.join(output_path, str(f) + ".npy"), + self.frames[f].camera_to_world, + ) + + def export_intrinsics(self, output_path): + if not os.path.exists(output_path): + os.makedirs(output_path) + print("exporting camera intrinsics to", output_path) + np.save(os.path.join(output_path, "intrinsic.npy"), self.intrinsic_color) + + +def reader( + filename, + output_path, + frame_skip, + export_color_images=False, + export_depth_images=False, + export_poses=False, + export_intrinsics=False, +): + if not os.path.exists(output_path): + os.makedirs(output_path) + + # load the data + print("loading %s..." % filename) + sd = SensorData(filename) + if export_depth_images: + sd.export_depth_images( + os.path.join(output_path, "depth"), frame_skip=frame_skip + ) + if export_color_images: + sd.export_color_images( + os.path.join(output_path, "color"), frame_skip=frame_skip + ) + if export_poses: + sd.export_poses(os.path.join(output_path, "pose"), frame_skip=frame_skip) + if export_intrinsics: + sd.export_intrinsics(os.path.join(output_path, "intrinsic")) + return sd.color_height, sd.color_width + + +def read_plymesh(filepath): + """Read ply file and return it as numpy array. Returns None if emtpy.""" + with open(filepath, "rb") as f: + plydata = plyfile.PlyData.read(f) + if plydata.elements: + vertices = pd.DataFrame(plydata["vertex"].data).values + faces = np.stack(plydata["face"].data["vertex_indices"], axis=0) + return vertices, faces + + +# Map the raw category id to the point cloud +def point_indices_from_group(seg_indices, group, labels_pd): + group_segments = np.array(group["segments"]) + label = group["label"] + + # Map the category name to id + label_id20 = labels_pd[labels_pd["raw_category"] == label]["nyu40id"] + label_id20 = int(label_id20.iloc[0]) if len(label_id20) > 0 else 0 + label_id200 = labels_pd[labels_pd["raw_category"] == label]["id"] + label_id200 = int(label_id200.iloc[0]) if len(label_id200) > 0 else 0 + + # Only store for the valid categories + if label_id20 in CLASS_IDS20: + label_id20 = CLASS_IDS20.index(label_id20) + else: + label_id20 = IGNORE_INDEX + + if label_id200 in CLASS_IDS200: + label_id200 = CLASS_IDS200.index(label_id200) + else: + label_id200 = IGNORE_INDEX + + # get points, where segment indices (points labelled with segment ids) are in the group segment list + point_idx = np.where(np.isin(seg_indices, group_segments))[0] + return point_idx, label_id20, label_id200 + + +def face_normal(vertex, face): + v01 = vertex[face[:, 1]] - vertex[face[:, 0]] + v02 = vertex[face[:, 2]] - vertex[face[:, 0]] + vec = np.cross(v01, v02) + length = np.sqrt(np.sum(vec**2, axis=1, keepdims=True)) + 1.0e-8 + nf = vec / length + area = length * 0.5 + return nf, area + + +def vertex_normal(vertex, face): + nf, area = face_normal(vertex, face) + nf = nf * area + + nv = np.zeros_like(vertex) + for i in range(face.shape[0]): + nv[face[i]] += nf[i] + + length = np.sqrt(np.sum(nv**2, axis=1, keepdims=True)) + 1.0e-8 + nv = nv / length + return nv + + +def correspondenceGet(mesh, K, T, img_size, coords_gt): + height, width = img_size + if np.isnan(T).any() or np.isinf(T).any(): + return None + + depth = ct.raycast.mesh_to_im_depth( + mesh=mesh, K=K, T=np.linalg.inv(T), height=height, width=width + ) + + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + depth = depth.reshape((-1, 1)) + valid = ~np.isinf(depth).squeeze(-1) + coord = np.zeros_like(pixel, dtype=np.float32) + coord[valid] = depth[valid] * (np.linalg.inv(K) @ pixel[valid].T).T # coord_camera + coord[valid] = coord[valid] @ T[:3, :3].T + T[:3, 3] # column then row + + pixel = pixel[valid] + coord = coord[valid] + if coord.shape[0] == 0: + return None + pixel = pixel[:, :2] + coord_dict = {"pixel": pixel, "coord": coord} + return coord_dict + + +def correspondenceSave(mesh, scene_dir, coords_gt, output_dir, img_size): + os.makedirs(output_dir, exist_ok=True) + scene_dir = Path(scene_dir) + index_gt = [ + img_name.split(".")[0] + for img_name in os.listdir(str(scene_dir / "color")) + if img_name.endswith(".png") + ] + index_gt = sorted(index_gt, key=lambda x: int(x)) + + Ks_path = str(scene_dir / "intrinsic" / "intrinsic.npy") + Ts_path = str(scene_dir / "pose") + Ts_files = sorted( + [f for f in os.listdir(Ts_path) if f.endswith(".npy")], + key=lambda x: int(x.split(".")[0]), + ) + + print(f"total pose num:{len(Ts_files)}") + Ts = [] + for Ts_file in Ts_files: + file_path = os.path.join(Ts_path, Ts_file) + Ts_ = np.load(file_path) + Ts.append(Ts_) + Ts = np.stack(Ts) + Ks = np.load(Ks_path) + + Ks = np.tile(Ks, (Ts.shape[0], 1, 1)) + Ks = Ks[:, :3, :3] + coords_gt_ = coords_gt + pixels_ = [] + coords_ = [] + + for i, (K, T) in enumerate(zip(Ks, Ts)): + coord_dict = correspondenceGet(mesh, K, T, img_size, coords_gt) + if coord_dict is None: + correspondences = -np.ones((1, 3)) + else: + pixels_ = coord_dict["pixel"] + coords_ = coord_dict["coord"] + tree = cKDTree(coords_gt_) + dis, idx = tree.query(coords_, k=1) + idx_valid = idx[dis < 0.01] + pixels_valid = pixels_[dis < 0.01] + correspondences = np.hstack((pixels_valid, idx_valid.reshape(-1, 1))) + np.save(Path(output_dir) / "{}.npy".format(index_gt[i]), correspondences) + + +def handle_process( + scene_path, + output_path, + pointclouds_root, + labels_pd, + train_scenes, + val_scenes, + frame_gap=75, + parse_pointclouds=True, + parse_normals=True, + export_depth_images=True, +): + pc_output_path = output_path + im_output_path = os.path.join(output_path, "images") + scene_id = os.path.basename(scene_path) + mesh_path = os.path.join(scene_path, f"{scene_id}{CLOUD_FILE_PFIX}.ply") + segments_file = os.path.join( + scene_path, f"{scene_id}{CLOUD_FILE_PFIX}{SEGMENTS_FILE_PFIX}" + ) + aggregations_file = os.path.join(scene_path, f"{scene_id}{AGGREGATIONS_FILE_PFIX}") + + if scene_id in train_scenes: + pc_output_path = os.path.join(pc_output_path, "train", f"{scene_id}") + pc_input_path = os.path.join(pointclouds_root, "train", f"{scene_id}") + im_output_path = os.path.join(im_output_path, "train", f"{scene_id}") + split_name = "train" + elif scene_id in val_scenes: + pc_output_path = os.path.join(pc_output_path, "val", f"{scene_id}") + pc_input_path = os.path.join(pointclouds_root, "val", f"{scene_id}") + im_output_path = os.path.join(im_output_path, "val", f"{scene_id}") + split_name = "val" + else: + pc_output_path = os.path.join(pc_output_path, "test", f"{scene_id}") + pc_input_path = os.path.join(pointclouds_root, "test", f"{scene_id}") + im_output_path = os.path.join(im_output_path, "test", f"{scene_id}") + split_name = "test" + + print(f"Processing: {scene_id} in {split_name}") + + if parse_pointclouds: + vertices, faces = read_plymesh(mesh_path) + coords = vertices[:, :3] + colors = vertices[:, 3:6] + save_dict = dict( + coord=coords.astype(np.float32), + color=colors.astype(np.uint8), + ) + # Parse Normals + if parse_normals: + save_dict["normal"] = vertex_normal(coords, faces).astype(np.float32) + + # Load segments file + if split_name != "test": + with open(segments_file) as f: + segments = json.load(f) + seg_indices = np.array(segments["segIndices"]) + + # Load Aggregations file + with open(aggregations_file) as f: + aggregation = json.load(f) + seg_groups = np.array(aggregation["segGroups"]) + + # Generate new labels + semantic_gt20 = np.ones((vertices.shape[0]), dtype=np.int16) * IGNORE_INDEX + semantic_gt200 = np.ones((vertices.shape[0]), dtype=np.int16) * IGNORE_INDEX + instance_ids = np.ones((vertices.shape[0]), dtype=np.int16) * IGNORE_INDEX + for group in seg_groups: + point_idx, label_id20, label_id200 = point_indices_from_group( + seg_indices, group, labels_pd + ) + + semantic_gt20[point_idx] = label_id20 + semantic_gt200[point_idx] = label_id200 + instance_ids[point_idx] = group["id"] + + semantic_gt20 = semantic_gt20.astype(int) + semantic_gt200 = semantic_gt200.astype(int) + instance_ids = instance_ids.astype(int) + + save_dict["segment20"] = semantic_gt20 + save_dict["segment200"] = semantic_gt200 + save_dict["instance"] = instance_ids + + # Concatenate with original cloud + processed_vertices = np.hstack((semantic_gt200, instance_ids)) + + if np.any(np.isnan(processed_vertices)) or not np.all( + np.isfinite(processed_vertices) + ): + raise ValueError(f"Find NaN in Scene: {scene_id}") + + # Save pointcloud data + os.makedirs(pc_output_path, exist_ok=True) + for key in save_dict.keys(): + np.save(os.path.join(pc_output_path, f"{key}.npy"), save_dict[key]) + else: + coords = np.load(os.path.join(pc_input_path, "coord.npy")) + save_dict = dict( + coord=coords.astype(np.float32), + ) + + # Save img data + os.makedirs(im_output_path, exist_ok=True) + sens_dir = os.path.join(scene_path, scene_id + ".sens") + print(f"Parsing sens data{sens_dir}") + h, w = reader( + sens_dir, + im_output_path, + frame_gap, + export_color_images=True, + export_depth_images=export_depth_images, + export_poses=True, + export_intrinsics=True, + ) + mesh = o3d.io.read_triangle_mesh(mesh_path) + correspondenceSave( + mesh, + im_output_path, + save_dict["coord"], + os.path.join(im_output_path, "correspondence"), + (h, w), + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--pointclouds_root", + default="data/scannet", + type=str, + help="Input path where previous pointclouds folder located", + ) + parser.add_argument( + "--frame_gap", default=75, type=int, help="Frame gap for processing" + ) + parser.add_argument( + "--parse_pointclouds", action="store_true", help="Whether parse point clouds" + ) + parser.add_argument( + "--parse_normals", action="store_true", help="Whether parse point normals" + ) + parser.add_argument( + "--parse_depths", action="store_true", help="Whether parse depths" + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + parser.add_argument( + "--thread_id", + default=0, + type=int, + help="Thread id for parallel processing", + ) + config = parser.parse_args() + meta_root = Path("pointcept/datasets/preprocessing/scannet") / "meta_data" + + # Load label map + labels_pd = pd.read_csv( + meta_root / "scannetv2-labels.combined.tsv", + sep="\t", + header=0, + ) + + # Load train/val splits + with open(meta_root / "scannetv2_train.txt") as train_file: + train_scenes = train_file.read().splitlines() + with open(meta_root / "scannetv2_val.txt") as val_file: + val_scenes = val_file.read().splitlines() + + # Load scene paths + scene_paths = sorted(glob.glob(config.dataset_root + "/scans*/scene*")) + scene_paths_list = np.array_split(scene_paths, config.num_workers) + scene_paths_ = scene_paths_list[config.thread_id] + # Preprocess data. + print("Processing scenes...") + for scene_paths_i in scene_paths_: + handle_process( + scene_paths_i, + config.output_root, + config.pointclouds_root, + labels_pd, + train_scenes, + val_scenes, + config.frame_gap, + config.parse_pointclouds, + config.parse_normals, + config.parse_depths, + ) diff --git a/pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.sh b/pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.sh new file mode 100644 index 0000000000000000000000000000000000000000..00b21127933c503663291dec909797f777aaa8f8 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.sh @@ -0,0 +1,42 @@ +#!/bin/bash + +dataset_root="" +output_root="" +num_workers=16 +parse_depths=false +parse_pointclouds=false + +while getopts "d:o:n:pc" opt; do + case $opt in + d) dataset_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + p) parse_depths=true ;; + c) parse_pointclouds=true ;; + *) echo "Usage: $0 -d -o [-n ] [-p] [-c]"; exit 1 ;; + esac +done + +if [ -z "$dataset_root" ] || [ -z "$output_root" ]; then + echo "Usage: $0 -d -o [-n ] [-p] [-c]" + exit 1 +fi + +for i in $(seq 0 $((num_workers - 1))); do + cmd="python pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.py --thread_id $i \ + --num_workers $num_workers \ + --dataset_root $dataset_root \ + --output_root $output_root \ + --parse_normal" + if $parse_depths; then + cmd="$cmd --parse_depths" + fi + + if $parse_pointclouds; then + cmd="$cmd --parse_pointclouds" + fi + + eval "$cmd &" +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/scannet/splits.py b/pointcept/datasets/preprocessing/concerto/scannet/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..5d4e5c614afb6885ce04cc6f85f2ed9d2ff4be47 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/scannet/splits.py @@ -0,0 +1,62 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["train", "val", "test"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split).replace( + dataset_path, "data/scannet" + ) + split_names = [f.name for f in os.scandir(im_split_path) if f.is_dir()] + split_dict = {} + for name in split_names: + im_split_name_path = os.path.join(im_split_path, name, "color") + co_split_name_path = os.path.join(im_split_path, name, "correspondence") + png_files = [ + f for f in os.listdir(im_split_name_path) if f.endswith(".png") + ] + png_files = sorted(png_files, key=lambda x: int(x.split(".")[0])) + # Get the full paths of the .png files + png_file_paths = [ + os.path.join(im_split_name_path, f).replace( + dataset_path, "data/scannet" + ) + for f in png_files + ] + co_file_paths = [ + os.path.join(co_split_name_path, f.replace(".png", ".npy")).replace( + dataset_path, "data/scannet" + ) + for f in png_files + ] + for i in range(0, len(png_file_paths), 4): + split_dict[f"{name}_{i//4}"] = {} + split_dict[f"{name}_{i//4}"]["pointclouds"] = os.path.join( + pc_split_path, name + ) + split_dict[f"{name}_{i//4}"]["images"] = png_file_paths[i : i + 4] + split_dict[f"{name}_{i//4}"]["correspondences"] = co_file_paths[ + i : i + 4 + ] + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.py b/pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.py new file mode 100644 index 0000000000000000000000000000000000000000..d9ab3a8493b757218b4e8f22c1f314571af8967b --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.py @@ -0,0 +1,557 @@ +""" +Preprocessing Script for ScanNet 20/200 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import warnings + +warnings.filterwarnings("ignore", category=DeprecationWarning) +import argparse +import json +import torch +import shutil +import os +import torch +import camtools as ct +import open3d as o3d +from scipy.spatial import cKDTree +import cv2 +import numpy as np +import pandas as pd +import multiprocessing as mp +from collections import OrderedDict +from scipy.spatial.transform import Rotation +from pathlib import Path +import re + +try: + import pointseg +except: + # Pointseg is located in libs/pointseg + warnings.warn("Pointseg is not installed, superpoint segmentation will be skipped.") + pointseg = None + +REGEXPR_DSLR = re.compile(r".*DSC(?P\d+).JPG$") +REGEXPR_IPHONE = re.compile(r"frame_(?P\d+).jpg$") + + +def pose_from_qwxyz_txyz(elems): + qw, qx, qy, qz, tx, ty, tz = map(float, elems) + pose = np.eye(4) + pose[:3, :3] = Rotation.from_quat((qx, qy, qz, qw)).as_matrix() + pose[:3, 3] = (tx, ty, tz) + c2w = np.linalg.inv(pose) # returns cam2world + return torch.tensor(c2w, dtype=torch.float32) + + +def get_frame_number(name, cam_type="dslr"): + if cam_type == "dslr": + regex_expr = REGEXPR_DSLR + elif cam_type == "iphone": + regex_expr = REGEXPR_IPHONE + else: + raise NotImplementedError(f"wrong {cam_type=} for get_frame_number") + matches = re.match(regex_expr, name) + return matches["frameid"] + + +def colmap_to_opencv_intrinsics(K): + """ + Modify camera intrinsics to follow a different convention. + Coordinates of the center of the top-left pixels are by default: + - (0.5, 0.5) in Colmap + - (0,0) in OpenCV + """ + K = K.copy() + K[0, 2] -= 0.5 + K[1, 2] -= 0.5 + return K + + +def opencv_to_colmap_intrinsics(K): + """ + Modify camera intrinsics to follow a different convention. + Coordinates of the center of the top-left pixels are by default: + - (0.5, 0.5) in Colmap + - (0,0) in OpenCV + """ + K = K.copy() + K[0, 2] += 0.5 + K[1, 2] += 0.5 + return K + + +def load_sfm(sfm_dir, cam_type="dslr"): + # load cameras + with open(os.path.join(sfm_dir, "cameras.txt"), "r") as f: + raw = f.read().splitlines()[3:] # skip header + + intrinsics = {} + for camera in raw: + camera = camera.split(" ") + intrinsics[int(camera[0])] = [camera[1]] + [float(cam) for cam in camera[2:]] + ins = intrinsics[int(camera[0])] + w, h, ins = undistort_ins(ins) + # load images + with open(os.path.join(sfm_dir, "images.txt"), "r") as f: + raw = f.read().splitlines() + raw = [line for line in raw if not line.startswith("#")] # skip header + + img_idx = {} + img_infos = {} + for image, points in zip(raw[0::2], raw[1::2]): + image = image.split(" ") + points = points.split(" ") + + idx = image[0] + img_name = image[-1] + assert img_name not in img_idx, "duplicate db image: " + img_name + img_idx[img_name] = idx # register image name + + # current_points2D = {int(i): (float(x), float(y)) + # for i, x, y in zip(points[2::3], points[0::3], points[1::3]) if i != '-1'} + img_infos[idx] = dict( + intrinsics=ins, + path=img_name, + frame_id=get_frame_number(img_name, cam_type), + cam_to_world=pose_from_qwxyz_txyz(image[1:-2]), + width=w, + height=h, + ) + + return img_idx, img_infos + + +def undistort_ins(intrinsics): + camera_type = intrinsics[0] + + width = int(intrinsics[1]) + height = int(intrinsics[2]) + fx = intrinsics[3] + fy = intrinsics[4] + cx = intrinsics[5] + cy = intrinsics[6] + distortion = np.array(intrinsics[7:]) + + K = np.zeros([3, 3]) + K[0, 0] = fx + K[0, 2] = cx + K[1, 1] = fy + K[1, 2] = cy + K[2, 2] = 1 + + K = colmap_to_opencv_intrinsics(K) + if camera_type == "OPENCV_FISHEYE": + assert len(distortion) == 4 + + new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify( + K, + distortion, + (width, height), + np.eye(3), + balance=0.0, + ) + # Make the cx and cy to be the center of the image + new_K[0, 2] = width / 2.0 + new_K[1, 2] = height / 2.0 + + map1, map2 = cv2.fisheye.initUndistortRectifyMap( + K, distortion, np.eye(3), new_K, (width, height), cv2.CV_32FC1 + ) + else: + new_K, _ = cv2.getOptimalNewCameraMatrix( + K, distortion, (width, height), 1, (width, height), True + ) + map1, map2 = cv2.initUndistortRectifyMap( + K, distortion, np.eye(3), new_K, (width, height), cv2.CV_32FC1 + ) + + # undistorted_image = cv2.remap(rgb, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REFLECT_101) + # undistorted_mask = cv2.remap(mask, map1, map2, interpolation=cv2.INTER_LINEAR, + # borderMode=cv2.BORDER_CONSTANT, borderValue=255) + new_K = opencv_to_colmap_intrinsics(new_K) + new_K = torch.tensor(new_K, dtype=torch.float32) + return width, height, new_K + + +def correspondenceGet(mesh, K, T, img_size, coords_gt): + height, width = img_size + if np.isnan(T).any() or np.isinf(T).any(): + return None + depth = ct.raycast.mesh_to_im_depth( + mesh=mesh, K=K, T=np.linalg.inv(T), height=height, width=width + ) + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + depth = depth.reshape((-1, 1)) + valid = ~np.isinf(depth).squeeze(-1) + coord = np.zeros_like(pixel, dtype=np.float32) + coord[valid] = depth[valid] * (np.linalg.inv(K) @ pixel[valid].T).T # coord_camera + coord[valid] = coord[valid] @ T[:3, :3].T + T[:3, 3] # column then row + pixel = pixel[valid] + coord = coord[valid] + if coord.shape[0] == 0: + return None + pixel = pixel[:, :2] + coord_dict = {"pixel": pixel, "coord": coord} + return coord_dict, depth + + +def correspondenceSave( + mesh, scene_dir, coords_gt, Ks, Ts, output_dir, img_size, parse_depths +): + os.makedirs(output_dir / "correspondence", exist_ok=True) + scene_dir = Path(scene_dir) + index_gt = [ + img_name.split(".")[0] + for img_name in os.listdir(str(scene_dir / "color")) + if img_name.endswith(".JPG") + ] + index_gt = sorted(index_gt, key=lambda x: int(x.split("DSC")[1])) + + Ks = Ks[:, :3, :3] + coords_gt_ = coords_gt + pixels_ = [] + coords_ = [] + + for i, (K, T) in enumerate(zip(Ks, Ts)): + try: + coord_dict, depth = correspondenceGet(mesh, K, T, img_size, coords_gt) + except: + print(output_dir) + if coord_dict is None: + correspondences = -np.ones((1, 3)) + else: + pixels_ = coord_dict["pixel"] + coords_ = coord_dict["coord"] + tree = cKDTree(coords_gt_) + dis, idx = tree.query(coords_, k=1) + idx_valid = idx[dis < 0.01] + pixels_valid = pixels_[dis < 0.01] + correspondences = np.hstack((pixels_valid, idx_valid.reshape(-1, 1))) + np.save( + output_dir / "correspondence" / "{}.npy".format(index_gt[i]), + correspondences, + ) + if parse_depths: + os.makedirs(output_dir / "depth", exist_ok=True) + depth = depth * 1000 + depth = np.where(np.isinf(depth), 65535, depth).astype(np.uint16) + depth = depth.reshape(img_size) + cv2.imwrite(output_dir / "depth" / "{}.png".format(index_gt[i]), depth) + + +def handle_process( + name, + split, + dataset_path, + output_path, + pointclouds_root, + label_mapping, + class2idx, + ignore_index=-1, + frame_gap=20, + parse_pointclouds=False, + parse_depths=False, +): + pc_output_path = output_path / split + im_output_path = output_path / "images" / split + scene_path = dataset_path / "data" / name + mesh_path = scene_path / "scans" / "mesh_aligned_0.05.ply" + segs_path = scene_path / "scans" / "segments.json" + anno_path = scene_path / "scans" / "segments_anno.json" + # load mesh vertices and colors + mesh = o3d.io.read_triangle_mesh(str(mesh_path)) + + if parse_pointclouds: + # extract mesh information + mesh.compute_vertex_normals(normalized=True) + coord = np.array(mesh.vertices).astype(np.float32) + color = (np.array(mesh.vertex_colors) * 255).astype(np.uint8) + normal = np.array(mesh.vertex_normals).astype(np.float32) + # extract superpoint information + if pointseg is not None: + vertices_sp = torch.from_numpy(np.array(mesh.vertices).astype(np.float32)) + faces_sp = torch.from_numpy(np.array(mesh.triangles).astype(np.int64)) + superpoint = pointseg.segment_mesh(vertices_sp, faces_sp).numpy() + else: + superpoint = None + + # Save pointcloud data + os.makedirs(pc_output_path / name, exist_ok=True) + np.save(pc_output_path / name / "coord.npy", coord) + np.save(pc_output_path / name / "color.npy", color) + np.save(pc_output_path / name / "normal.npy", normal) + if superpoint is not None: + np.save(pc_output_path / "superpoint.npy", superpoint) + + if split != "test": + # get label on vertices + # load segments = vertices per segment ID + with open(segs_path) as f: + segments = json.load(f) + # load anno = (instance, groups of segments) + with open(anno_path) as f: + anno = json.load(f) + seg_indices = np.array(segments["segIndices"], dtype=np.uint32) + num_vertices = len(seg_indices) + assert num_vertices == len(coord) + semantic_gt = np.ones((num_vertices, 3), dtype=np.int16) * ignore_index + instance_gt = np.ones((num_vertices, 3), dtype=np.int16) * ignore_index + + # number of labels are used per vertex. initially 0 + # increment each time a new label is added + instance_size = np.ones((num_vertices, 3), dtype=np.int16) * np.inf + + # keep track of the size of the instance (#vertices) assigned to each vertex + # later, keep the label of the smallest instance for major label of vertices + # store inf initially so that we can pick the smallest instance + labels_used = np.zeros(num_vertices, dtype=np.int16) + + for idx, instance in enumerate(anno["segGroups"]): + label = instance["label"] + instance["label_orig"] = label + # remap label + instance["label"] = label_mapping.get(label, None) + instance["label_index"] = class2idx.get(instance["label"], ignore_index) + + if instance["label_index"] == ignore_index: + continue + # get all the vertices with segment index in this instance + # and max number of labels not yet applied + mask = np.isin(seg_indices, instance["segments"]) & (labels_used < 3) + size = mask.sum() + if size == 0: + continue + + # get the position to add the label - 0, 1, 2 + label_position = labels_used[mask] + semantic_gt[mask, label_position] = instance["label_index"] + # store all valid instance (include ignored instance) + instance_gt[mask, label_position] = instance["objectId"] + instance_size[mask, label_position] = size + labels_used[mask] += 1 + + # major label is the label of smallest instance for each vertex + # use major label for single class segmentation + # shift major label to the first column + mask = labels_used > 1 + if mask.sum() > 0: + major_label_position = np.argmin(instance_size[mask], axis=1) + + major_semantic_label = semantic_gt[mask, major_label_position] + semantic_gt[mask, major_label_position] = semantic_gt[:, 0][mask] + semantic_gt[:, 0][mask] = major_semantic_label + + major_instance_label = instance_gt[mask, major_label_position] + instance_gt[mask, major_label_position] = instance_gt[:, 0][mask] + instance_gt[:, 0][mask] = major_instance_label + + np.save(pc_output_path / name / "segment.npy", semantic_gt) + np.save(pc_output_path / name / "instance.npy", instance_gt) + else: + coord = np.load(pointclouds_root / split / name / "coord.npy") + rgb_path = scene_path / "dslr" / "undistorted_images" + colmap_path = scene_path / "dslr" / "colmap" + img_idxs, img_infos = load_sfm(str(colmap_path), cam_type="dslr") + rgb_path = sorted( + rgb_path.glob("*.JPG"), key=lambda x: int(str(x)[:-4].split("DSC")[1]) + )[::frame_gap] + rgb_path = [str(path) for path in rgb_path] + color_output_path = im_output_path / name / "color" + os.makedirs(color_output_path, exist_ok=True) + for rgb_path_i in rgb_path: + shutil.copy(rgb_path_i, color_output_path) + imgnames = [path.split("/")[-1] for path in rgb_path] + img_idxs = [img_idxs[imgname] for imgname in imgnames] + img_infos = [img_infos[img_idx] for img_idx in img_idxs] + intrinsic_save_path = im_output_path / name / "intrinsic" + pose_save_path = im_output_path / name / "pose" + os.makedirs(intrinsic_save_path, exist_ok=True) + os.makedirs(pose_save_path, exist_ok=True) + Ks = [] + Ts = [] + for id, img_info in enumerate(img_infos): + np.save(intrinsic_save_path / f"{id}.npy", img_info["intrinsics"]) + np.save(pose_save_path / f"{id}.npy", img_info["cam_to_world"]) + Ks.append(img_info["intrinsics"]) + Ts.append(img_info["cam_to_world"]) + Ks = np.stack(Ks) + Ts = np.stack(Ts) + + if len(rgb_path) == 0 or len(img_infos) == 0: + return + h = img_infos[0]["height"] + w = img_infos[0]["width"] + correspondenceSave( + mesh, + im_output_path / name, + coord, + Ks, + Ts, + im_output_path / name, + (h, w), + parse_depths, + ) + + +def filter_map_classes(mapping, mapping_type): + if mapping_type == "semantic": + map_key = "semantic_map_to" + elif mapping_type == "instance": + map_key = "instance_map_to" + else: + raise NotImplementedError + # create a dict with classes to be mapped + # classes that don't have mapping are entered as x->x + # otherwise x->y + map_dict = OrderedDict() + + for i in range(mapping.shape[0]): + row = mapping.iloc[i] + class_name = row["class"] + map_target = row[map_key] + + # map to None or some other label -> don't add this class to the label list + try: + if len(map_target) > 0: + # map to None -> don't use this class + if map_target == "None": + pass + else: + # map to something else -> use this class + map_dict[class_name] = map_target + except TypeError: + # nan values -> no mapping, keep label as is + if class_name not in map_dict: + map_dict[class_name] = class_name + + return map_dict + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--pointclouds_root", + default="data/scannetpp", + type=str, + help="Input path where previous pointclouds folder located", + ) + parser.add_argument( + "--ignore_index", + default=-1, + type=int, + help="Default ignore index.", + ) + parser.add_argument( + "--frame_gap", default=20, type=int, help="Frame gap for processing" + ) + parser.add_argument( + "--parse_pointclouds", action="store_true", help="Whether parse point clouds" + ) + parser.add_argument( + "--parse_depths", action="store_true", help="Whether parse depths" + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + parser.add_argument( + "--thread_id", + default=0, + type=int, + help="Thread id for parallel processing", + ) + config = parser.parse_args() + + print("Loading meta data...") + config.dataset_root = Path(config.dataset_root) + config.output_root = Path(config.output_root) + config.pointclouds_root = Path(config.pointclouds_root) + + train_list = np.loadtxt( + config.dataset_root / "splits" / "nvs_sem_train.txt", + dtype=str, + ) + print("Num samples in training split:", len(train_list)) + + val_list = np.loadtxt( + config.dataset_root / "splits" / "nvs_sem_val.txt", + dtype=str, + ) + print("Num samples in validation split:", len(val_list)) + + test_list = np.loadtxt( + config.dataset_root / "splits" / "sem_test.txt", + dtype=str, + ) + print("Num samples in testing split:", len(test_list)) + + data_list = np.concatenate([train_list, val_list, test_list]) + split_list = np.concatenate( + [ + np.full_like(train_list, "train"), + np.full_like(val_list, "val"), + np.full_like(test_list, "test"), + ] + ) + + # Parsing label information and mapping + segment_class_names = np.loadtxt( + config.dataset_root / "metadata" / "semantic_benchmark" / "top100.txt", + dtype=str, + delimiter=".", # dummy delimiter to replace " " + ) + print("Num classes in segment class list:", len(segment_class_names)) + + instance_class_names = np.loadtxt( + config.dataset_root / "metadata" / "semantic_benchmark" / "top100_instance.txt", + dtype=str, + delimiter=".", # dummy delimiter to replace " " + ) + print("Num classes in instance class list:", len(instance_class_names)) + + label_mapping = pd.read_csv( + config.dataset_root / "metadata" / "semantic_benchmark" / "map_benchmark.csv" + ) + label_mapping = filter_map_classes(label_mapping, mapping_type="semantic") + class2idx = { + class_name: idx for (idx, class_name) in enumerate(segment_class_names) + } + # Load scene paths + + data_list_list = np.array_split(data_list, config.num_workers) + data_list_ = data_list_list[config.thread_id] + split_list_list = np.array_split(split_list, config.num_workers) + split_list_ = split_list_list[config.thread_id] + # Preprocess data. + print("Processing scenes...") + for data_list_i, split_list_i in zip(data_list_, split_list_): + handle_process( + data_list_i, + split_list_i, + config.dataset_root, + config.output_root, + config.pointclouds_root, + label_mapping, + class2idx, + config.ignore_index, + config.frame_gap, + config.parse_pointclouds, + config.parse_depths, + ) diff --git a/pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.sh b/pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.sh new file mode 100644 index 0000000000000000000000000000000000000000..13fe42e113d78d0b002cee8be9edcb466f99f16d --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.sh @@ -0,0 +1,42 @@ +#!/bin/bash + +dataset_root="" +output_root="" +num_workers=16 +parse_depths=false +parse_pointclouds=false + +while getopts "d:o:n:pc" opt; do + case $opt in + d) dataset_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + p) parse_depths=true ;; + c) parse_pointclouds=true ;; + *) echo "Usage: $0 -d -o [-n ] [-p] [-c]"; exit 1 ;; + esac +done + +if [ -z "$dataset_root" ] || [ -z "$output_root" ]; then + echo "Usage: $0 -d -o [-n ] [-p] [-c]" + exit 1 +fi + +for i in $(seq 0 $((num_workers - 1))); do + cmd="python pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.py --thread_id $i \ + --num_workers $num_workers \ + --dataset_root $dataset_root \ + --output_root $output_root" + + if $parse_depths; then + cmd="$cmd --parse_depths" + fi + + if $parse_pointclouds; then + cmd="$cmd --parse_pointclouds" + fi + + eval "$cmd &" +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/scannetpp/splits.py b/pointcept/datasets/preprocessing/concerto/scannetpp/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..3789af74a6e3b7224b784fff0b183d4d2554fa23 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/scannetpp/splits.py @@ -0,0 +1,64 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["train", "val", "test"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split).replace( + dataset_path, "data/scannetpp" + ) + split_names = [f.name for f in os.scandir(im_split_path) if f.is_dir()] + split_dict = {} + for name in split_names: + im_split_name_path = os.path.join(im_split_path, name, "color") + co_split_name_path = os.path.join(im_split_path, name, "correspondence") + png_files = [ + f for f in os.listdir(im_split_name_path) if f.endswith(".JPG") + ] + png_files = sorted( + png_files, key=lambda x: int(x.split(".")[0].split("DSC")[-1]) + ) + # Get the full paths of the .png files + png_file_paths = [ + os.path.join(im_split_name_path, f).replace( + dataset_path, "data/scannetpp" + ) + for f in png_files + ] + co_file_paths = [ + os.path.join(co_split_name_path, f.replace(".JPG", ".npy")).replace( + dataset_path, "data/scannetpp" + ) + for f in png_files + ] + for i in range(0, len(png_file_paths), 4): + split_dict[f"{name}_{i//4}"] = {} + split_dict[f"{name}_{i//4}"]["pointclouds"] = os.path.join( + pc_split_path, name + ) + split_dict[f"{name}_{i//4}"]["images"] = png_file_paths[i : i + 4] + split_dict[f"{name}_{i//4}"]["correspondences"] = co_file_paths[ + i : i + 4 + ] + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.py b/pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.py new file mode 100644 index 0000000000000000000000000000000000000000..4a6177f11e46b5eaaa5acc70873bd21bd076476c --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.py @@ -0,0 +1,557 @@ +""" +Preprocessing Script for Structured3D + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import argparse +import io +import os +from PIL import Image +import cv2 +import zipfile +import numpy as np +import multiprocessing as mp + +VALID_CLASS_IDS_25 = ( + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 11, + 14, + 15, + 16, + 17, + 18, + 19, + 22, + 24, + 25, + 32, + 34, + 35, + 38, + 39, + 40, +) +CLASS_LABELS_25 = ( + "wall", + "floor", + "cabinet", + "bed", + "chair", + "sofa", + "table", + "door", + "window", + "picture", + "desk", + "shelves", + "curtain", + "dresser", + "pillow", + "mirror", + "ceiling", + "refrigerator", + "television", + "nightstand", + "sink", + "lamp", + "otherstructure", + "otherfurniture", + "otherprop", +) + + +def normal_from_cross_product(points_2d: np.ndarray) -> np.ndarray: + xyz_points_pad = np.pad(points_2d, ((0, 1), (0, 1), (0, 0)), mode="symmetric") + xyz_points_ver = (xyz_points_pad[:, :-1, :] - xyz_points_pad[:, 1:, :])[:-1, :, :] + xyz_points_hor = (xyz_points_pad[:-1, :, :] - xyz_points_pad[1:, :, :])[:, :-1, :] + xyz_normal = np.cross(xyz_points_hor, xyz_points_ver) + xyz_dist = np.linalg.norm(xyz_normal, axis=-1, keepdims=True) + xyz_normal = np.divide( + xyz_normal, xyz_dist, out=np.zeros_like(xyz_normal), where=xyz_dist != 0 + ) + return xyz_normal + + +class Structured3DReader: + def __init__(self, files): + super().__init__() + if isinstance(files, str): + files = [files] + self.readers = [zipfile.ZipFile(f, "r") for f in files] + self.names_mapper = dict() + for idx, reader in enumerate(self.readers): + for name in reader.namelist(): + self.names_mapper[name] = idx + + def filelist(self): + return list(self.names_mapper.keys()) + + def listdir(self, dir_name): + dir_name = dir_name.lstrip(os.path.sep).rstrip(os.path.sep) + file_list = list( + np.unique( + [ + f.replace(dir_name + os.path.sep, "", 1).split(os.path.sep)[0] + for f in self.filelist() + if f.startswith(dir_name + os.path.sep) + ] + ) + ) + if "" in file_list: + file_list.remove("") + return file_list + + def read(self, file_name): + split = self.names_mapper[file_name] + return self.readers[split].read(file_name) + + def read_camera(self, camera_path): + z2y_top_m = np.array([[0, 1, 0], [0, 0, 1], [1, 0, 0]], dtype=np.float32) + cam_extr = np.fromstring(self.read(camera_path), dtype=np.float32, sep=" ") + cam_t = np.matmul(z2y_top_m, cam_extr[:3] / 1000) + if cam_extr.shape[0] > 3: + cam_front, cam_up = cam_extr[3:6], cam_extr[6:9] + cam_n = np.cross(cam_front, cam_up) + cam_r = np.stack((cam_front, cam_up, cam_n), axis=1).astype(np.float32) + cam_r = np.matmul(z2y_top_m, cam_r) + cam_f = cam_extr[9:11] + else: + cam_r = np.eye(3, dtype=np.float32) + cam_f = None + return cam_r, cam_t, cam_f + + def read_depth(self, depth_path): + depth = cv2.imdecode( + np.frombuffer(self.read(depth_path), np.uint8), cv2.IMREAD_UNCHANGED + )[..., np.newaxis] + depth[depth == 0] = 65535 + return depth + + def read_color(self, color_path): + color = cv2.imdecode( + np.frombuffer(self.read(color_path), np.uint8), cv2.IMREAD_UNCHANGED + )[..., :3][..., ::-1] + return color + + def read_segment(self, segment_path): + segment = np.array(Image.open(io.BytesIO(self.read(segment_path))))[ + ..., np.newaxis + ] + return segment + + +def parse_scene( + scene, + dataset_root, + output_root, + ignore_index=-1, + grid_size=None, + fuse_prsp=True, + fuse_pano=True, + parse_pointclouds=True, + parse_depths=True, + vis=False, +): + assert fuse_prsp or fuse_pano + pc_output_root = output_root + im_output_root = os.path.join(output_root, "images") + reader = Structured3DReader( + [ + os.path.join(dataset_root, f) + for f in os.listdir(dataset_root) + if f.endswith(".zip") + ] + ) + scene_id = int(os.path.basename(scene).split("_")[-1]) + if scene_id < 3000: + split = "train" + elif 3000 <= scene_id < 3250: + split = "val" + else: + split = "test" + + print(f"Processing: {scene} in {split}") + + rooms = reader.listdir(os.path.join("Structured3D", scene, "2D_rendering")) + for room in rooms: + im_save_path = os.path.join( + im_output_root, split, os.path.basename(scene), f"room_{room}" + ) + pc_save_path = os.path.join( + pc_output_root, split, os.path.basename(scene), f"room_{room}" + ) + if os.path.exists(im_save_path) and os.path.exists(pc_save_path): + print(f"exist {split}/{os.path.basename(scene)}/room_{room}") + continue + + room_path = os.path.join("Structured3D", scene, "2D_rendering", room) + coord_list = list() + color_list = list() + normal_list = list() + segment_list = list() + prsp_list = list() + pano_list = list() + prsp_depth_list = list() + pano_depth_list = list() + prsp_correspondence_list = list() + pano_correspondence_list = list() + Ks_list = list() + Ts_list = list() + if fuse_prsp and scene: + prsp_path = os.path.join(room_path, "perspective", "full") + frames = reader.listdir(prsp_path) + + for frame_id, frame in enumerate(frames): + try: + cam_r, cam_t, cam_f = reader.read_camera( + os.path.join(prsp_path, frame, "camera_pose.txt") + ) + depth = reader.read_depth( + os.path.join(prsp_path, frame, "depth.png") + ) + color = reader.read_color( + os.path.join(prsp_path, frame, "rgb_rawlight.png") + ) + segment = reader.read_segment( + os.path.join(prsp_path, frame, "semantic.png") + ) + except: + print( + f"Skipping {scene}_room{room}_frame{frame} perspective view due to loading error" + ) + else: + fx, fy = cam_f + height, width = depth.shape[0], depth.shape[1] + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + k = np.diag([1.0, 1.0, 1.0]) + + k[0, 2] = width / 2 + k[1, 2] = height / 2 + + k[0, 0] = k[0, 2] / np.tan(fx) + k[1, 1] = k[1, 2] / np.tan(fy) + coord = ( + depth.reshape((-1, 1)) * (np.linalg.inv(k) @ pixel.T).T + ).reshape(height, width, 3) + coord = coord @ np.array([[0, 0, 1], [0, -1, 0], [1, 0, 0]]) + normal = normal_from_cross_product(coord) + + # Filtering invalid points + view_dist = np.maximum( + np.linalg.norm(coord, axis=-1, keepdims=True), float(10e-5) + ) + cosine_dist = np.sum( + (coord * normal / view_dist), axis=-1, keepdims=True + ) + cosine_dist = np.abs(cosine_dist) + mask = ((cosine_dist > 0.15) & (depth < 65535) & (segment > 0))[ + ..., 0 + ].reshape(-1) + + coord = np.matmul(coord / 1000, cam_r.T) + cam_t + normal = normal_from_cross_product(coord) + + T = np.eye(4) + T[:3, :3] = cam_r + T[:3, 3] = cam_t + + Ks_list.append(k) + Ts_list.append(T) + + pixel[:, 2] = pixel[:, 2] * frame_id + pixel_valid = pixel[mask] + + if sum(mask) > 0: + prsp_depth_list.append(depth) + coord_list.append(coord.reshape(-1, 3)[mask]) + color_list.append(color.reshape(-1, 3)[mask]) + normal_list.append(normal.reshape(-1, 3)[mask]) + segment_list.append(segment.reshape(-1, 1)[mask]) + prsp_list.append(color) + prsp_correspondence_list.append(pixel_valid) + else: + print( + f"Skipping {scene}_room{room}_frame{frame} perspective view due to all points are filtered out" + ) + + if fuse_pano: + pano_path = os.path.join(room_path, "panorama") + try: + _, cam_t, _ = reader.read_camera( + os.path.join(pano_path, "camera_xyz.txt") + ) + depth = reader.read_depth(os.path.join(pano_path, "full", "depth.png")) + color = reader.read_color( + os.path.join(pano_path, "full", "rgb_rawlight.png") + ) + segment = reader.read_segment( + os.path.join(pano_path, "full", "semantic.png") + ) + except: + print(f"Skipping {scene}_room{room} panorama view due to loading error") + else: + p_h, p_w = depth.shape[:2] + pixel = np.transpose(np.indices((p_w, p_h)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + p_a = np.arange(p_w, dtype=np.float32) / p_w * 2 * np.pi - np.pi + p_b = np.arange(p_h, dtype=np.float32) / p_h * np.pi * -1 + np.pi / 2 + p_a = np.tile(p_a[None], [p_h, 1])[..., np.newaxis] + p_b = np.tile(p_b[:, None], [1, p_w])[..., np.newaxis] + p_a_sin, p_a_cos, p_b_sin, p_b_cos = ( + np.sin(p_a), + np.cos(p_a), + np.sin(p_b), + np.cos(p_b), + ) + x = depth * p_a_cos * p_b_cos + y = depth * p_b_sin + z = depth * p_a_sin * p_b_cos + coord = np.concatenate([x, y, z], axis=-1) / 1000 + normal = normal_from_cross_product(coord) + + # Filtering invalid points + view_dist = np.maximum( + np.linalg.norm(coord, axis=-1, keepdims=True), float(10e-5) + ) + cosine_dist = np.sum( + (coord * normal / view_dist), axis=-1, keepdims=True + ) + cosine_dist = np.abs(cosine_dist) + mask = ((cosine_dist > 0.15) & (depth < 65535) & (segment > 0))[ + ..., 0 + ].reshape(-1) + coord = coord + cam_t + + pixel = np.hstack((pixel, -np.ones((pixel.shape[0], 1)))) + pixel_valid = pixel[mask] + + if sum(mask) > 0: + pano_depth_list.append(depth) + coord_list.append(coord.reshape(-1, 3)[mask]) + color_list.append(color.reshape(-1, 3)[mask]) + normal_list.append(normal.reshape(-1, 3)[mask]) + segment_list.append(segment.reshape(-1, 1)[mask]) + pano_list.append(color) + pano_correspondence_list.append(pixel_valid) + else: + print( + f"Skipping {scene}_room{room} panorama view due to all points are filtered out" + ) + if len(prsp_correspondence_list) > 0 and len(pano_correspondence_list) > 0: + coord = np.concatenate(coord_list, axis=0) + coord = coord @ np.array([[1, 0, 0], [0, 0, 1], [0, 1, 0]]) + color = np.concatenate(color_list, axis=0) + normal = np.concatenate(normal_list, axis=0) + normal = normal @ np.array([[1, 0, 0], [0, 0, 1], [0, 1, 0]]) + segment = np.concatenate(segment_list, axis=0) + segment25 = np.ones_like(segment, dtype=np.int64) * ignore_index + Ks_list = np.stack(Ks_list, axis=0) + Ts_list = np.stack(Ts_list, axis=0) + Ts_list = Ts_list @ np.array( + [[1, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0], [0, 0, 0, 1]] + ) + for idx, value in enumerate(VALID_CLASS_IDS_25): + mask = np.all(segment == value, axis=-1) + segment25[mask] = idx + correspondence = np.concatenate(prsp_correspondence_list, axis=0) + correspondence = np.concatenate( + [correspondence, pano_correspondence_list[0]], axis=0 + ) + data_dict = dict( + coord=coord.astype(np.float32), + color=color.astype(np.uint8), + normal=normal.astype(np.float32), + segment=segment25.astype(np.int16), + correspondence=correspondence.astype(np.int32), + ) + + # exclude ignore, wall, floor, ceiling + valid = np.sum(~np.isin(data_dict["segment"], [-1, 0, 1, 16])) + if valid == 0: + print( + f"Skip {scene}_room{room} due to no effective points (exclude wall, floor, ceiling)" + ) + continue + + # Grid sampling data + if grid_size is not None: + grid_coord = np.floor(coord / grid_size).astype(int) + _, idx = np.unique(grid_coord, axis=0, return_index=True) + # coord = coord[idx] + for key in data_dict.keys(): + data_dict[key] = data_dict[key][idx] + + correspondence = data_dict["correspondence"] + correspondence = np.concatenate( + [correspondence, np.arange(correspondence.shape[0])[:, None]], axis=1 + ) + frame_id_list = np.unique(correspondence[:, 2]) + frame_id_list = frame_id_list[frame_id_list != -1] + pano_correspondence_list = [ + correspondence[correspondence[:, 2] == -1][:, [0, 1, 3]] + ] + prsp_correspondence_list = [ + correspondence[correspondence[:, 2] == i][:, [0, 1, 3]] + for i in frame_id_list + ] + if parse_pointclouds: + os.makedirs(pc_save_path, exist_ok=True) + # Save data + for key in data_dict.keys(): + if key in ["correspondence"]: + continue + np.save(os.path.join(pc_save_path, f"{key}.npy"), data_dict[key]) + + os.makedirs(im_save_path, exist_ok=True) + if fuse_prsp: + prsp_save_path = os.path.join(im_save_path, "color", "prsp") + prsp_correspondence_save_path = os.path.join( + im_save_path, "correspondence", "prsp_correspondence" + ) + Ks_save_path = os.path.join(im_save_path, "intrinsic") + Ts_save_path = os.path.join(im_save_path, "pose") + os.makedirs(prsp_correspondence_save_path, exist_ok=True) + os.makedirs(prsp_save_path, exist_ok=True) + os.makedirs(Ks_save_path, exist_ok=True) + os.makedirs(Ts_save_path, exist_ok=True) + if parse_depths: + prsp_depth_save_path = os.path.join(im_save_path, "depth", "prsp") + os.makedirs(prsp_depth_save_path, exist_ok=True) + for idx in range(len(prsp_list)): + prsp_img_path = os.path.join(prsp_save_path, f"{idx}.png") + cv2.imwrite(prsp_img_path, prsp_list[idx][..., ::-1]) + if parse_depths: + prsp_depth_img_path = os.path.join( + prsp_depth_save_path, f"{idx}.png" + ) + cv2.imwrite(prsp_depth_img_path, prsp_depth_list[idx]) + np.save( + os.path.join(prsp_correspondence_save_path, f"{idx}.npy"), + prsp_correspondence_list[idx], + ) + np.save( + os.path.join(Ks_save_path, f"{idx}.npy"), + Ks_list[idx], + ) + np.save( + os.path.join(Ts_save_path, f"{idx}.npy"), + Ts_list[idx], + ) + + if fuse_pano: + pano_save_path = os.path.join(im_save_path, "color", "pano") + pano_correspondence_save_path = os.path.join( + im_save_path, "correspondence", "pano_correspondence" + ) + os.makedirs(pano_save_path, exist_ok=True) + os.makedirs(pano_correspondence_save_path, exist_ok=True) + if parse_depths: + pano_depth_save_path = os.path.join(im_save_path, "depth", "pano") + os.makedirs(pano_depth_save_path, exist_ok=True) + for idx in range(len(pano_list)): + pano_img_path = os.path.join(pano_save_path, f"{idx}.png") + cv2.imwrite(pano_img_path, pano_list[idx][..., ::-1]) + if parse_depths: + pano_depth_img_path = os.path.join( + pano_depth_save_path, f"{idx}.png" + ) + cv2.imwrite(pano_depth_img_path, pano_depth_list[idx]) + for idx, pano_correspondence in enumerate(pano_correspondence_list): + np.save( + os.path.join(pano_correspondence_save_path, f"{idx}.npy"), + pano_correspondence, + ) + + if vis: + from pointcept.utils.visualization import save_point_cloud + + os.makedirs("./vis", exist_ok=True) + save_point_cloud( + coord, color / 255, f"./vis/{scene}_room{room}_color.ply" + ) + save_point_cloud( + coord, (normal + 1) / 2, f"./vis/{scene}_room{room}_normal.ply" + ) + else: + print(f"Skipping {scene}_room{room} due to no valid points") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the Structured3D dataset containing scene folders.", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located.", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + parser.add_argument( + "--thread_id", + default=0, + type=int, + help="thread_id.", + ) + parser.add_argument( + "--grid_size", default=None, type=float, help="Grid size for grid sampling." + ) + parser.add_argument("--ignore_index", default=-1, type=float, help="Ignore index.") + parser.add_argument( + "--fuse_prsp", action="store_true", help="Whether fuse perspective view." + ) + parser.add_argument( + "--fuse_pano", action="store_true", help="Whether fuse panorama view." + ) + parser.add_argument( + "--parse_pointclouds", action="store_true", help="Whether parse point clouds" + ) + parser.add_argument( + "--parse_depths", action="store_true", help="Whether parse depths" + ) + config = parser.parse_args() + + reader = Structured3DReader( + [ + os.path.join(config.dataset_root, f) + for f in os.listdir(config.dataset_root) + if f.endswith(".zip") + ] + ) + + scenes_list = reader.listdir("Structured3D") + scenes_list = sorted(scenes_list) + split_scenes_list = np.array_split(scenes_list, config.num_workers) + split_scenes_list_ = split_scenes_list[config.thread_id] + + for scenes_list_i in split_scenes_list_: + parse_scene( + scenes_list_i, + config.dataset_root, + config.output_root, + config.ignore_index, + config.grid_size, + config.fuse_prsp, + config.fuse_pano, + config.parse_pointclouds, + config.parse_depths, + ) diff --git a/pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.sh b/pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.sh new file mode 100644 index 0000000000000000000000000000000000000000..a9a3e67eed37a9722a7ae364c69dc1f09b588375 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.sh @@ -0,0 +1,43 @@ +#!/bin/bash + +dataset_root="" +output_root="" +num_workers=16 +parse_depths=false +parse_pointclouds=false + +while getopts "d:o:n:pc" opt; do + case $opt in + d) dataset_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + p) parse_depths=true ;; + c) parse_pointclouds=true ;; + *) echo "Usage: $0 -d -o [-n ] [-p] [-c]"; exit 1 ;; + esac +done + +if [ -z "$dataset_root" ] || [ -z "$output_root" ]; then + echo "Usage: $0 -d -o [-n ] [-p] [-c]" + exit 1 +fi + +for i in $(seq 0 $((num_workers - 1))); do + cmd="python pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.py --thread_id $i \ + --num_workers $num_workers \ + --dataset_root $dataset_root \ + --output_root $output_root \ + --fuse_prsp --fuse_pano" + + if $parse_depths; then + cmd="$cmd --parse_depths" + fi + + if $parse_pointclouds; then + cmd="$cmd --parse_pointclouds" + fi + + eval "$cmd &" +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/structured3d/splits.py b/pointcept/datasets/preprocessing/concerto/structured3d/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..6743bf465b1f35096edf9c0d7cb4f8a5a567350e --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/structured3d/splits.py @@ -0,0 +1,69 @@ +import os +import json +import argparse + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["train", "val"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split).replace( + dataset_path, "data/structured3d" + ) + split_names = [f.name for f in os.scandir(im_split_path) if f.is_dir()] + split_dict = {} + for name in split_names: + cam_split_name_path = os.path.join(im_split_path, name) + cam_names = [f.name for f in os.scandir(cam_split_name_path) if f.is_dir()] + for cam_name in cam_names: + im_split_name_path = os.path.join( + cam_split_name_path, cam_name, "color", "prsp" + ) + co_split_name_path = os.path.join( + cam_split_name_path, + cam_name, + "correspondence", + "prsp_correspondence", + ) + png_files = [ + f for f in os.listdir(im_split_name_path) if f.endswith(".png") + ] + png_files = sorted(png_files, key=lambda x: int(x.split(".")[0])) + # Get the full paths of the .png files + png_file_paths = [ + os.path.join(im_split_name_path, f).replace( + dataset_path, "data/structured3d" + ) + for f in png_files + ] + co_file_paths = [ + os.path.join(co_split_name_path, f.replace(".png", ".npy")).replace( + dataset_path, "data/structured3d" + ) + for f in png_files + ] + split_dict[f"{name}_{cam_name}"] = {} + split_dict[f"{name}_{cam_name}"]["pointclouds"] = os.path.join( + pc_split_path, name, cam_name + ) + split_dict[f"{name}_{cam_name}"]["images"] = png_file_paths + split_dict[f"{name}_{cam_name}"]["correspondences"] = co_file_paths + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the Structured3D dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.py b/pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.py new file mode 100644 index 0000000000000000000000000000000000000000..83870092329d9227cd0662ea4eab9abfe00dd1a7 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.py @@ -0,0 +1,625 @@ +""" +Preprocessing Script for ScanNet 20/200 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import warnings + +warnings.filterwarnings("ignore", category=DeprecationWarning) + +import os + +os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3" +os.environ["CUDA_VISIBLE_DEVICES"] = "-1" +cam_order_list = ["SIDE_RIGHT", "SIDE_LEFT", "FRONT_RIGHT", "FRONT_LEFT", "FRONT"] + +import argparse +import numpy as np +import tensorflow.compat.v1 as tf +from pathlib import Path +from PIL import Image +import io +import open3d as o3d + +os.environ["OMP_NUM_THREADS"] = "1" +from waymo_open_dataset.utils import frame_utils +from waymo_open_dataset.utils import transform_utils +from waymo_open_dataset.utils import range_image_utils +from waymo_open_dataset import dataset_pb2 as open_dataset +from waymo_open_dataset.wdl_limited.camera.ops import py_camera_model_ops +import glob +import multiprocessing as mp + + +def project_vehicle_to_image(vehicle_pose, calibration, points): + """Projects from vehicle coordinate system to image with global shutter. + + Arguments: + vehicle_pose: Vehicle pose transform from vehicle into world coordinate + system. + calibration: Camera calibration details (including intrinsics/extrinsics). + points: Points to project of shape [N, 3] in vehicle coordinate system. + + Returns: + Array of shape [N, 3], with the latter dimension composed of (u, v, ok). + """ + # Transform points from vehicle to world coordinate system. + pose_matrix = np.array(vehicle_pose.transform).reshape(4, 4) + world_points = (points @ pose_matrix[:3, :3].T + pose_matrix[:3, 3]).astype( + points.dtype + ) + + # Populate camera image metadata. Velocity and latency stats are filled with + # zeroes. + extrinsic = tf.reshape( + tf.constant(list(calibration.extrinsic.transform), dtype=tf.float32), [4, 4] + ) + intrinsic = tf.constant(list(calibration.intrinsic), dtype=tf.float32) + metadata = tf.constant( + [ + calibration.width, + calibration.height, + open_dataset.CameraCalibration.GLOBAL_SHUTTER, + ], + dtype=tf.int32, + ) + camera_image_metadata = list(vehicle_pose.transform) + [0.0] * 10 + + # Perform projection and return projected image coordinates (u, v, ok). + return py_camera_model_ops.world_to_image( + extrinsic, intrinsic, metadata, camera_image_metadata, world_points + ).numpy() + + +def get_normals(cam_center, coords): + Cs = np.repeat(cam_center.reshape((1, -1)), coords.shape[0], axis=0) + view_dirs = coords - Cs + view_dirs = view_dirs / np.linalg.norm(view_dirs, axis=-1, keepdims=True) + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coords) + pcd.estimate_normals() + normals = np.asarray(pcd.normals) + dot_product = np.sum(normals * view_dirs, axis=-1) + flip_mask = dot_product > 0 + normals[flip_mask] = -normals[flip_mask] + + # Normalize normals a nd m + normals = normals / np.linalg.norm(normals, axis=-1, keepdims=True) + return normals + + +def create_lidar_and_normals(frame): + """Points plus per-point normals, in the SAME point order as create_lidar/create_label. + + Point order matters here. This file builds the same points two ways: + + per return (create_lidar, create_label): [s0r0, s1r0 ... snr0, s0r1 ... snr1] + per sensor (what this used to do): [s0r0, s0r1, s1r0, s1r1, ...] + + They are permutations of each other, so mixing them lines up two arrays against different + points without any length or shape error. `strength` below comes from create_lidar, which is + per return, and handle_process pairs this function's coord with create_label's labels, which + is also per return. Building per sensor here therefore misaligned both. + + Normals are still estimated per sensor, because they need that sensor's origin to be oriented + and estimate_normals should see both returns of a sensor together. They are then placed back + into their per return slots. + """ + ( + range_images, + camera_projections, + segmentation_labels, + range_image_top_pose, + ) = frame_utils.parse_range_image_and_camera_projection(frame) + + # These return every sensor at once, so calling them inside the per sensor loop recomputed the + # whole frame once per sensor. + points_0, _, valid_masks_0 = convert_range_image_to_point_cloud( + frame, + range_images, + camera_projections, + range_image_top_pose, + ri_index=0, + keep_polar_features=False, + ) + points_1, _, valid_masks_1 = convert_range_image_to_point_cloud( + frame, + range_images, + camera_projections, + range_image_top_pose, + ri_index=1, + keep_polar_features=False, + ) + + # Same sort convert_range_image_to_point_cloud uses to index the lists it returns. + calibrations = sorted(frame.context.laser_calibrations, key=lambda c: c.name) + + normals_0, normals_1 = [], [] + for sensor_index, c in enumerate(calibrations): + sensor_points_0 = points_0[sensor_index] + sensor_points_1 = points_1[sensor_index] + sensor_points_all = np.concatenate([sensor_points_0, sensor_points_1], axis=0) + + if sensor_points_all.shape[0] == 0: + # Append empty blocks rather than `continue`: the point lists still carry this + # sensor's empty slots, so skipping would desync normals from points. + normals_0.append(np.zeros((len(sensor_points_0), 3))) + normals_1.append(np.zeros((len(sensor_points_1), 3))) + continue + + extrinsic = np.array(c.extrinsic.transform).reshape(4, 4) + lidar_center_vehicle_frame = extrinsic[:3, 3] + + normals = get_normals(lidar_center_vehicle_frame, sensor_points_all) + normals_0.append(normals[: len(sensor_points_0)]) + normals_1.append(normals[len(sensor_points_0) :]) + + # List concatenation, not array: [all ri0 blocks] + [all ri1 blocks] is create_lidar's order. + final_points = np.concatenate(points_0 + points_1, axis=0) + final_normals = np.concatenate(normals_0 + normals_1, axis=0) + + # Raw intensity: handle_process applies the tanh, as the standard Waymo script does. + velodyne, _ = create_lidar(frame) + strength = velodyne.reshape(-1, 4)[:, -1].reshape([-1, 1]) + assert len(final_points) == len(strength) == len(final_normals), ( + f"point/strength/normal length mismatch ({len(final_points)}/{len(strength)}/" + f"{len(final_normals)}) -- the two orderings have diverged again" + ) + point_cloud_with_strength = np.c_[final_points, strength] + # Mirrors create_lidar's [ri0, ri1] shape. + return point_cloud_with_strength, final_normals, [valid_masks_0, valid_masks_1] + + +def create_lidar(frame): + """Parse and save the lidar data in psd format. + Args: + frame (:obj:`Frame`): Open dataset frame proto. + """ + ( + range_images, + camera_projections, + segmentation_labels, + range_image_top_pose, + ) = frame_utils.parse_range_image_and_camera_projection(frame) + + points, cp_points, valid_masks = convert_range_image_to_point_cloud( + frame, + range_images, + camera_projections, + range_image_top_pose, + keep_polar_features=True, + ) + points_ri2, cp_points_ri2, valid_masks_ri2 = convert_range_image_to_point_cloud( + frame, + range_images, + camera_projections, + range_image_top_pose, + ri_index=1, + keep_polar_features=True, + ) + + # 3d points in vehicle frame. + points_all = np.concatenate(points, axis=0) + points_all_ri2 = np.concatenate(points_ri2, axis=0) + # point labels. + + points_all = np.concatenate([points_all, points_all_ri2], axis=0) + + velodyne = np.c_[points_all[:, 3:6], points_all[:, 1]] + velodyne = velodyne.reshape((velodyne.shape[0] * velodyne.shape[1])) + + valid_masks = [valid_masks, valid_masks_ri2] + return velodyne, valid_masks + + +def create_label(frame): + ( + range_images, + camera_projections, + segmentation_labels, + range_image_top_pose, + ) = frame_utils.parse_range_image_and_camera_projection(frame) + + point_labels = convert_range_image_to_point_cloud_labels( + frame, range_images, segmentation_labels + ) + point_labels_ri2 = convert_range_image_to_point_cloud_labels( + frame, range_images, segmentation_labels, ri_index=1 + ) + + # point labels. + point_labels_all = np.concatenate(point_labels, axis=0) + point_labels_all_ri2 = np.concatenate(point_labels_ri2, axis=0) + point_labels_all = np.concatenate([point_labels_all, point_labels_all_ri2], axis=0) + + labels = point_labels_all + return labels + + +def convert_range_image_to_cartesian( + frame, range_images, range_image_top_pose, ri_index=0, keep_polar_features=False +): + """Convert range images from polar coordinates to Cartesian coordinates. + + Args: + frame: open dataset frame + range_images: A dict of {laser_name, [range_image_first_return, + range_image_second_return]}. + range_image_top_pose: range image pixel pose for top lidar. + ri_index: 0 for the first return, 1 for the second return. + keep_polar_features: If true, keep the features from the polar range image + (i.e. range, intensity, and elongation) as the first features in the + output range image. + + Returns: + dict of {laser_name, (H, W, D)} range images in Cartesian coordinates. D + will be 3 if keep_polar_features is False (x, y, z) and 6 if + keep_polar_features is True (range, intensity, elongation, x, y, z). + """ + cartesian_range_images = {} + frame_pose = tf.convert_to_tensor( + value=np.reshape(np.array(frame.pose.transform), [4, 4]) + ) + + # [H, W, 6] + range_image_top_pose_tensor = tf.reshape( + tf.convert_to_tensor(value=range_image_top_pose.data), + range_image_top_pose.shape.dims, + ) + # [H, W, 3, 3] + range_image_top_pose_tensor_rotation = transform_utils.get_rotation_matrix( + range_image_top_pose_tensor[..., 0], + range_image_top_pose_tensor[..., 1], + range_image_top_pose_tensor[..., 2], + ) + range_image_top_pose_tensor_translation = range_image_top_pose_tensor[..., 3:] + range_image_top_pose_tensor = transform_utils.get_transform( + range_image_top_pose_tensor_rotation, range_image_top_pose_tensor_translation + ) + + for c in frame.context.laser_calibrations: + range_image = range_images[c.name][ri_index] + if len(c.beam_inclinations) == 0: # pylint: disable=g-explicit-length-test + beam_inclinations = range_image_utils.compute_inclination( + tf.constant([c.beam_inclination_min, c.beam_inclination_max]), + height=range_image.shape.dims[0], + ) + else: + beam_inclinations = tf.constant(c.beam_inclinations) + + beam_inclinations = tf.reverse(beam_inclinations, axis=[-1]) + extrinsic = np.reshape(np.array(c.extrinsic.transform), [4, 4]) + + range_image_tensor = tf.reshape( + tf.convert_to_tensor(value=range_image.data), range_image.shape.dims + ) + pixel_pose_local = None + frame_pose_local = None + if c.name == open_dataset.LaserName.TOP: + pixel_pose_local = range_image_top_pose_tensor + pixel_pose_local = tf.expand_dims(pixel_pose_local, axis=0) + frame_pose_local = tf.expand_dims(frame_pose, axis=0) + range_image_cartesian = range_image_utils.extract_point_cloud_from_range_image( + tf.expand_dims(range_image_tensor[..., 0], axis=0), + tf.expand_dims(extrinsic, axis=0), + tf.expand_dims(tf.convert_to_tensor(value=beam_inclinations), axis=0), + pixel_pose=pixel_pose_local, + frame_pose=frame_pose_local, + ) + + range_image_cartesian = tf.squeeze(range_image_cartesian, axis=0) + + if keep_polar_features: + # If we want to keep the polar coordinate features of range, intensity, + # and elongation, concatenate them to be the initial dimensions of the + # returned Cartesian range image. + range_image_cartesian = tf.concat( + [range_image_tensor[..., 0:3], range_image_cartesian], axis=-1 + ) + + cartesian_range_images[c.name] = range_image_cartesian + + return cartesian_range_images + + +def convert_range_image_to_point_cloud( + frame, + range_images, + camera_projections, + range_image_top_pose, + ri_index=0, + keep_polar_features=False, +): + """Convert range images to point cloud. + + Args: + frame: open dataset frame + range_images: A dict of {laser_name, [range_image_first_return, + range_image_second_return]}. + camera_projections: A dict of {laser_name, + [camera_projection_from_first_return, + camera_projection_from_second_return]}. + range_image_top_pose: range image pixel pose for top lidar. + ri_index: 0 for the first return, 1 for the second return. + keep_polar_features: If true, keep the features from the polar range image + (i.e. range, intensity, and elongation) as the first features in the + output range image. + + Returns: + points: {[N, 3]} list of 3d lidar points of length 5 (number of lidars). + (NOTE: Will be {[N, 6]} if keep_polar_features is true. + cp_points: {[N, 6]} list of camera projections of length 5 + (number of lidars). + """ + calibrations = sorted(frame.context.laser_calibrations, key=lambda c: c.name) + points = [] + cp_points = [] + valid_masks = [] + + cartesian_range_images = convert_range_image_to_cartesian( + frame, range_images, range_image_top_pose, ri_index, keep_polar_features + ) + + for c in calibrations: + range_image = range_images[c.name][ri_index] + range_image_tensor = tf.reshape( + tf.convert_to_tensor(value=range_image.data), range_image.shape.dims + ) + range_image_mask = range_image_tensor[..., 0] > 0 + + range_image_cartesian = cartesian_range_images[c.name] + points_tensor = tf.gather_nd( + range_image_cartesian, tf.compat.v1.where(range_image_mask) + ) + + cp = camera_projections[c.name][ri_index] + cp_tensor = tf.reshape(tf.convert_to_tensor(value=cp.data), cp.shape.dims) + cp_points_tensor = tf.gather_nd(cp_tensor, tf.compat.v1.where(range_image_mask)) + points.append(points_tensor.numpy()) + cp_points.append(cp_points_tensor.numpy()) + valid_masks.append(range_image_mask.numpy()) + + return points, cp_points, valid_masks + + +def convert_range_image_to_point_cloud_labels( + frame, range_images, segmentation_labels, ri_index=0 +): + """Convert segmentation labels from range images to point clouds. + + Args: + frame: open dataset frame + range_images: A dict of {laser_name, [range_image_first_return, + range_image_second_return]}. + segmentation_labels: A dict of {laser_name, [range_image_first_return, + range_image_second_return]}. + ri_index: 0 for the first return, 1 for the second return. + + Returns: + point_labels: {[N, 2]} list of 3d lidar points's segmentation labels. 0 for + points that are not labeled. + """ + calibrations = sorted(frame.context.laser_calibrations, key=lambda c: c.name) + point_labels = [] + for c in calibrations: + range_image = range_images[c.name][ri_index] + range_image_tensor = tf.reshape( + tf.convert_to_tensor(range_image.data), range_image.shape.dims + ) + range_image_mask = range_image_tensor[..., 0] > 0 + + if c.name in segmentation_labels: + sl = segmentation_labels[c.name][ri_index] + sl_tensor = tf.reshape(tf.convert_to_tensor(sl.data), sl.shape.dims) + sl_points_tensor = tf.gather_nd(sl_tensor, tf.where(range_image_mask)) + else: + num_valid_point = tf.math.reduce_sum(tf.cast(range_image_mask, tf.int32)) + sl_points_tensor = tf.zeros([num_valid_point, 2], dtype=tf.int32) + + point_labels.append(sl_points_tensor.numpy()) + return point_labels + + +def project_lidar_to_image_with_color( + vehicle_pose, lidar_points, image, calibration, lidar_colors +): + """ + Projects LiDAR points to the image, fetches pixel color and pixel coordinates. + Returns: + filtered_points: (M, 3) - 3D points in camera frame that project onto the image. + colors: (M, 3) - RGB colors at projected 2D locations. + uv_coords: (M, 2) - Integer pixel coordinates (u, v) on the image. + mask: (N,) - (optional) Boolean mask indicating which lidar points are used. + """ + + lidar_uv_coords = project_vehicle_to_image(vehicle_pose, calibration, lidar_points) + u, v, ok = lidar_uv_coords.transpose() + ok = ok.astype(bool) + + # Skip object if any corner projection failed. Note that this is very + # strict and can lead to exclusion of some partially visible objects. + + # Clip box to image bounds. + # u = np.clip(u.round(), 0, calibration.width-1).astype(int) + # v = np.clip(v.round(), 0, calibration.height-1).astype(int) + u_rounded = u.round().astype(int) + v_rounded = v.round().astype(int) + mask_u_valid = (u_rounded >= 0) & (u_rounded < calibration.width) + mask_v_valid = (v_rounded >= 0) & (v_rounded < calibration.height) + valid_mask = mask_u_valid & mask_v_valid & ok + u_filtered = u_rounded[valid_mask] + v_filtered = v_rounded[valid_mask] + ok = ok & valid_mask + lidar_colors[ok] = image[v_filtered, u_filtered, :] + lidar_uv_coords = np.stack((u_rounded, v_rounded), axis=1).astype(np.int32) + lidar_uv_coords[~valid_mask] = -1 + return lidar_colors, lidar_uv_coords, valid_mask + + +def handle_process(file_path, output_root, test_frame_list): + file = os.path.basename(file_path) + split = os.path.basename(os.path.dirname(file_path)) + print(f"Parsing {split}/{file}") + save_path = Path(output_root) / split / file.split(".")[0] + + data_group = tf.data.TFRecordDataset(file_path, compression_type="") + for data in data_group: + frame = open_dataset.Frame() + frame.ParseFromString(bytes(data.numpy())) + context_name = frame.context.name + timestamp = str(frame.timestamp_micros) + img_save_path = ( + Path(output_root) / "images" / split / file.split(".")[0] / timestamp + ) + color_save_path = img_save_path / "color" + correspondence_save_path = img_save_path / "correspondence" + intrinsic_save_path = img_save_path / "intrinsic" + pose_save_path = img_save_path / "pose" + + if split != "testing": + # for training and validation frame, extract labelled frame + if not frame.lasers[0].ri_return1.segmentation_label_compressed: + continue + else: + # for testing frame, extract frame in test_frame_list + if f"{context_name},{timestamp}" not in test_frame_list: + continue + os.makedirs(color_save_path, exist_ok=True) + os.makedirs(correspondence_save_path, exist_ok=True) + os.makedirs(intrinsic_save_path, exist_ok=True) + os.makedirs(pose_save_path, exist_ok=True) + os.makedirs(save_path / timestamp, exist_ok=True) + + # extract frame pass above check + point_cloud, normal, valid_masks = create_lidar_and_normals(frame) + point_cloud = point_cloud.reshape(-1, 4) + coord = point_cloud[:, :3] + strength = np.tanh(point_cloud[:, -1].reshape([-1, 1])) + pose = np.array(frame.pose.transform, np.float32).reshape(4, 4) + mask = np.array(valid_masks, dtype=object) + + np.save(save_path / timestamp / "coord.npy", coord) + np.save(save_path / timestamp / "strength.npy", strength) + np.save(save_path / timestamp / "pose.npy", pose) + np.save(save_path / timestamp / "normal.npy", normal) + + # save mask for reverse prediction + if split != "training": + np.save(save_path / timestamp / "mask.npy", mask) + + # save label + if split != "testing": + # ignore TYPE_UNDEFINED, ignore_index 0 -> -1 + label = create_label(frame)[:, 1].reshape([-1]) - 1 + np.save(save_path / timestamp / "segment.npy", label) + + img_DLC = {} + for image in frame.images: + camera_name = open_dataset.CameraName.Name.Name(image.name) + img_DLC[camera_name] = np.array(Image.open(io.BytesIO(image.image))) + filename = color_save_path / f"{camera_name}.jpg" + with open(filename, "wb") as f: + f.write(image.image) + color = np.zeros((coord.shape[0], 3)) + # {'FRONT': 0, 'SIDE_RIGHT': 1, 'SIDE_LEFT': 2, ...} + order_map = {name: i for i, name in enumerate(cam_order_list)} + cam_sorted_calibrations = sorted( + frame.context.camera_calibrations, + key=lambda calib: order_map.get( + open_dataset.CameraName.Name.Name(calib.name), 999 + ), + ) + cam_mask = np.full((coord.shape[0],), False, dtype=bool) + for c in cam_sorted_calibrations: + camera_name = open_dataset.CameraName.Name.Name(c.name) + cam2ego = np.array(c.extrinsic.transform).reshape(4, 4) + intrinsic = np.array(c.intrinsic) + fx = intrinsic[0] + fy = intrinsic[1] + cx = intrinsic[2] + cy = intrinsic[3] + intrinsic = np.array( + [[fx, 0, cx], [0, fy, cy], [0, 0, 1]], dtype=np.float32 + ) + np.save(intrinsic_save_path / f"{camera_name}.npy", intrinsic) + np.save(pose_save_path / f"{camera_name}.npy", cam2ego) + color, uv_correspondence, mask = project_lidar_to_image_with_color( + frame.pose, coord, img_DLC[camera_name], c, color + ) + cam_mask = np.logical_or(cam_mask, mask) + correspondence_point_id = np.array( + range(uv_correspondence.shape[0]) + ).reshape((-1, 1)) + uv_correspondence = np.hstack([uv_correspondence, correspondence_point_id]) + np.save(correspondence_save_path / f"{camera_name}.npy", uv_correspondence) + np.save(save_path / timestamp / "color.npy", color) + np.save(save_path / timestamp / "cam_mask.npy", cam_mask) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the Waymo dataset", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--splits", + required=True, + nargs="+", + choices=["training", "validation", "testing"], + help="Splits need to process ([training, validation, testing]).", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + parser.add_argument( + "--thread_id", + default=0, + type=int, + help="Thread id.", + ) + config = parser.parse_args() + + # load file list + origin_splits = ["training", "validation", "testing"] + file_list = [] + for split in origin_splits: + file_list.extend( + glob.glob( + os.path.join(os.path.abspath(config.dataset_root), split, "*.tfrecord") + ) + ) + assert len(file_list) == 1150, f"have {len(file_list)} files" + + # Create output directories + for split in config.splits: + os.makedirs(os.path.join(config.output_root, split), exist_ok=True) + + file_list = [ + file + for file in file_list + if os.path.basename(os.path.dirname(file)) in config.splits + ] + + # Load test frame list + test_frame_file = os.path.join( + os.path.dirname(os.path.dirname(os.path.dirname(__file__))), + "waymo/3d_semseg_test_set_frames.txt", + ) + test_frame_list = [x.rstrip() for x in (open(test_frame_file, "r").readlines())] + + split_file_list = np.array_split(file_list, config.num_workers) + split_file_list_ = split_file_list[config.thread_id] + for file_list_i in split_file_list_: + handle_process(file_list_i, config.output_root, test_frame_list) diff --git a/pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.sh b/pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.sh new file mode 100644 index 0000000000000000000000000000000000000000..46b34c2b8eaa2ba29262fa6c18296cc24df86e95 --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.sh @@ -0,0 +1,34 @@ +#!/bin/bash + +dataset_root="" +output_root="" +num_workers=16 +splits="training validation" + +while getopts "d:o:n:s:" opt; do + case $opt in + d) dataset_root=$OPTARG ;; + o) output_root=$OPTARG ;; + n) num_workers=$OPTARG ;; + s) splits=$OPTARG ;; + *) echo "Usage: $0 -d -o -s '' [-n ]"; exit 1 ;; + esac +done + +if [ -z "$dataset_root" ] || [ -z "$output_root" ]; then + echo "Usage: $0 -d -o -s '' [-n ]" + exit 1 +fi + +for i in $(seq 0 $((num_workers - 1))); do + cmd="python pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.py \ + --dataset_root $dataset_root \ + --output_root $output_root \ + --splits $splits \ + --num_workers $num_workers \ + --thread_id $i" + + eval "$cmd" > "preprocess_waymo_thread_${i}.log" 2>&1 & +done + +wait diff --git a/pointcept/datasets/preprocessing/concerto/waymo/splits.py b/pointcept/datasets/preprocessing/concerto/waymo/splits.py new file mode 100644 index 0000000000000000000000000000000000000000..3b63047c7a4c995e25696c7cc33d5cae3ace382e --- /dev/null +++ b/pointcept/datasets/preprocessing/concerto/waymo/splits.py @@ -0,0 +1,63 @@ +import os +import json +import argparse + +CAM_TYPES = ["FRONT_LEFT", "FRONT_RIGHT", "FRONT", "SIDE_LEFT", "SIDE_RIGHT"] + + +def get_splits_paths(dataset_path): + # Get the names of all subfolders in the given folder + im_path = os.path.join(dataset_path, "images") + pc_path = dataset_path + splits = ["training", "validation"] + split_path = os.path.join(dataset_path, "splits") + os.makedirs(split_path, exist_ok=True) + for split in splits: + im_split_path = os.path.join(im_path, split) + pc_split_path = os.path.join(pc_path, split).replace(dataset_path, "data/waymo") + split_names = sorted([f.name for f in os.scandir(im_split_path) if f.is_dir()]) + split_dict = {} + for name in split_names: + im_split_subpaths = os.path.join(im_split_path, name) + subsplit_names = sorted( + [f.name for f in os.scandir(im_split_subpaths) if f.is_dir()] + ) + for subname in subsplit_names: + im_split_name_path = os.path.join(im_split_path, name, subname, "color") + co_split_name_path = os.path.join( + im_split_path, name, subname, "correspondence" + ) + jpg_files = [cam_type + ".jpg" for cam_type in CAM_TYPES] + # Get the full paths of the .jpg files + jpg_file_paths = [ + os.path.join(im_split_name_path, f).replace( + dataset_path, "data/waymo" + ) + for f in jpg_files + ] + co_file_paths = [ + os.path.join(co_split_name_path, f.replace(".jpg", ".npy")).replace( + dataset_path, "data/waymo" + ) + for f in jpg_files + ] + split_dict[f"{name}_{subname}"] = {} + split_dict[f"{name}_{subname}"]["pointclouds"] = os.path.join( + pc_split_path, name, subname + ) + split_dict[f"{name}_{subname}"]["images"] = jpg_file_paths + split_dict[f"{name}_{subname}"]["correspondences"] = co_file_paths + with open(os.path.join(split_path, f"{split}.json"), "w") as f: + json.dump(split_dict, f, indent=4) + + +# Example usage +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the waymo dataset containing scene folders", + ) + config = parser.parse_args() + get_splits_paths(config.dataset_root) diff --git a/pointcept/datasets/preprocessing/hm3d/filter_hm3d.py b/pointcept/datasets/preprocessing/hm3d/filter_hm3d.py new file mode 100644 index 0000000000000000000000000000000000000000..4ff216f1de466d3d8d1561feb18fecf506cda61a --- /dev/null +++ b/pointcept/datasets/preprocessing/hm3d/filter_hm3d.py @@ -0,0 +1,92 @@ +""" +Filtering Script for Habitat-Matterport 3D Dataset + +filter out and only keep top 10,000 size of processed HM3D + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import glob +import argparse +import numpy as np +import shutil +import tqdm +from pathlib import Path +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat + + +def count_scene(data_path, info_list, lock): + data_path = Path(data_path) + data_name = data_path.name + data_split = data_path.parent.name + print(f"Counting {data_name} in {data_split}..") + data_size = np.load(data_path / "color.npy").shape[0] + with lock: + info_list.append(dict(name=data_name, split=data_split, size=data_size)) + + +def main_process(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the Habitat-Matterport 3D dataset containing scene folders", + ) + + parser.add_argument( + "--num_keep", + default=10000, + type=int, + help="Number of scenes that kept for the dataset.", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + args = parser.parse_args() + + scene_list = glob.glob(os.path.join(args.dataset_root, "*", "*")) + if len(scene_list) <= args.num_keep: + return + manager = mp.Manager() + lock = manager.Lock() + info_list = manager.list() + + # Preprocess data. + print("Processing scenes...") + pool = ProcessPoolExecutor(max_workers=args.num_workers) + _ = list( + pool.map( + count_scene, + scene_list, + repeat(info_list), + repeat(lock), + ) + ) + name_list = np.array([scene["name"] for scene in info_list]) + split_list = np.array([scene["split"] for scene in info_list]) + size_list = np.array([scene["size"] for scene in info_list]) + remove_index = np.argsort(size_list)[: len(scene_list) - args.num_keep] + + for split in np.unique(split_list): + os.makedirs(os.path.join(args.dataset_root, f"{split}_rm"), exist_ok=True) + + source = [ + os.path.join(args.dataset_root, split_list[i], name_list[i]) + for i in remove_index + ] + target = [ + os.path.join(args.dataset_root, f"{split_list[i]}_rm") for i in remove_index + ] + for s, t in zip(source, target): + shutil.move(s, t) + + +if __name__ == "__main__": + main_process() diff --git a/pointcept/datasets/preprocessing/hm3d/hm3d_constants.py b/pointcept/datasets/preprocessing/hm3d/hm3d_constants.py new file mode 100644 index 0000000000000000000000000000000000000000..7ebc4e95eb37a9dec80395f13f1747b42c7a5aee --- /dev/null +++ b/pointcept/datasets/preprocessing/hm3d/hm3d_constants.py @@ -0,0 +1,44 @@ +from pathlib import Path + +CLASS_LABELS_40 = ( + "wall", # 0 + "floor", # 1 + "chair", # 2 + "door", # 3 + "table", # 4 + "picture", # 5 + "cabinet", # 6 + "cushion", # 7 + "window", # 8 + "sofa", # 9 + "bed", # 10 + "curtain", # 11 + "chest_of_drawers", # 12 + "plant", # 13 + "sink", # 14 + "stairs", # 15 + "ceiling", # 16 + "toilet", # 17 + "stool", # 18 + "towel", # 19 + "mirror", # 20 + "tv_monitor", # 21 + "shower", # 22 + "column", # 23 + "bathtub", # 24 + "counter", # 25 + "fireplace", # 26 + "lighting", # 27 + "beam", # 28 + "railing", # 29 + "shelving", # 30 + "blinds", # 31 + "gym_equipment", # 32 + "seating", # 33 + "board_panel", # 34 + "furniture", # 35 + "appliances", # 36 + "clothes", # 37 + "objects", # 38 + "misc", # 39 +) diff --git a/pointcept/datasets/preprocessing/hm3d/hm3dsem_category_mappings.tsv b/pointcept/datasets/preprocessing/hm3d/hm3dsem_category_mappings.tsv new file mode 100644 index 0000000000000000000000000000000000000000..888b1428fcdbaabd8037e58b93ecd16402a93d6b --- /dev/null +++ b/pointcept/datasets/preprocessing/hm3d/hm3dsem_category_mappings.tsv @@ -0,0 +1,2368 @@ +raw_category category mpcat40 +,1 unknown unlabeled +,4 unknown unlabeled +,5 unknown unlabeled +,6 unknown unlabeled +/unknown/ probably decoration unknown unlabeled +acoustic panel acoustic panel misc +advertisement advertisement misc +aemchair armchair chair +affice chair office chair chair +air conditioner air conditioner objects +air conditioning air conditioning objects +air duct air duct misc +air freshener air freshener objects +air heater air heater misc +air hockey air hockey objects +air purifier air purifier objects +air refresher air refresher objects +air vent air vent misc +air vent fan air vent fan objects +air vent installation air vent installation misc +aisle frame aisle frame misc +alarm alarm objects +alarm clock alarm clock objects +alarm control alarm control misc +alarm controller alarm controller misc +album album objects +albums albums objects +alcohol bottles alcohol bottles objects +american flag american flag misc +amp amplifier objects +amplifier amplifier objects +antique clock antique clock objects +antique telehone antique telephone objects +antique telephone antique telephone objects +antlers antlers misc +apple apple objects +appliance appliance misc +apron apron misc +aquarium aquarium misc +arcade arcade misc +arcade game arcade game misc +arch arch misc +archway archway door +arm chair armchair chair +armchair armchair chair +armchir armchair chair +art picture picture +art frame art frame misc +art piece art piece misc +art work artwork picture +art/clutter art/clutter misc +art/muscle shell art/muscle shell misc +art/statue art/statue misc +artframe art frame misc +artwork artwork picture +artwork frame artwork frame misc +ashtray ashtray objects +attic door attic door misc +attic entrance attic entrance misc +attic hatch attic hatch misc +audio cable audio cable misc +audio cables audio cables misc +audio player audio player objects +axe axe objects +baby chair baby chair misc +baby changing station baby changing station misc +baby changing table baby changing table misc +baby seat baby seat misc +backpack backpack objects +backrest backrest objects +backsplash backsplash misc +bad unknown unlabeled +badside cabinet bedside cabinet chest_of_drawers +bag bag objects +bag with sheets bag with sheets objects +bag with something bag with something objects +bags bags objects +balcony balcony misc +balcony railing balcony railing misc +balcony reiling balcony railing misc +balk ball misc +ball ball misc +ball chair ball chair chair +ball pool ball pool objects +ball pouffe ball pouffe seating +balustrade balustrade railing +banister banister railing +banner banner misc +bar bar objects +bar cabinet bar cabinet cabinet +bar chair bar chair chair +bar soap bar soap misc +barbecue barbecue misc +barbell barbell gym_equipment +barrel barrel misc +bascet basket objects +base base misc +baseball bat baseball bat objects +baseball cap baseball cap objects +baseboard baseboard wall +basin basin sink +basin faucet basin faucet objects +basket basket objects +basket /w books basket with books objects +basket of fruits basket of fruits objects +basket of something basket of something misc +basket of towels basket of towels misc +basket with clothes basket with clothes objects +basket wqith clothes basket with clothes objects +basketball hoop basketball hoop gym_equipment +basketballs basketballs objects +baskets baskets objects +bat tube bath tub bathtub +bath bath bathtub +bath bar bath bar misc +bath cabinet bath cabinet cabinet +bath carpet bath carpet floor +bath cosmetics bath cosmetics objects +bath curtain bath curtain curtain +bath curtain bar bath curtain bar misc +bath dial bath dial misc +bath door bath door door +bath door frame bath door frame door +bath faucet bath faucet objects +bath floor bath floor floor +bath grab bar bath grab bar misc +bath hanger bath hanger objects +bath mat bath mat floor +bath shelf bath shelf shelving +bath shower bath shower shower +bath side table bath side table table +bath sink bath sink misc +bath tap bath tap objects +bath towel bath towel towel +bath towels bath towels towel +bath tub bath tub bathtub +bath tube bath tub bathtub +bath utensil bath utensil misc +bath wall bath wall misc +bathmat bathmat floor +bathrobe bathrobe clothes +bathroom mat bathroom mat floor +bathroom accessory bathroom accessory clothes +bathroom art bathroom art objects +bathroom cabinet bathroom cabinet cabinet +bathroom cabinet door bathroom cabinet door misc +bathroom cabinet drawer bathroom cabinet drawer chest_of_drawers +bathroom cabint bathroom cabinet cabinet +bathroom counter bathroom counter counter +bathroom floor bathroom floor floor +bathroom glass bathroom glass objects +bathroom mat bathroom mat floor +bathroom mirror mirror mirror +bathroom rug bathroom rug floor +bathroom shelf bathroom shelf shelving +bathroom stuff bathroom stuff objects +bathroom towel bathroom towel towel +bathroom utencills bathroom utensils objects +bathroom utencils bathroom utensil objects +bathroom utensil bathroom utensil objects +bathroom wall bathroom wall wall +bathroom window bathroom window window +bathtub bathtub bathtub +bathtub knob bathtub knob misc +bathtub platform bathtub platform misc +bathtub tap bathtub tap objects +bathtub utensil bathtub utensil misc +bathtube bathtub bathtub +batroom art bathroom art objects +beam beam beam +bean bag chair bean bag chair chair +beanbag beanbag chair +beanbag chair beanbag chair chair +bed bed bed +bed base bed base bed +bed cabinet bed cabinet chest_of_drawers +bed cabinet lamp bed cabinet lamp lighting +bed comforter bed comforter misc +bed curtain bed curtain misc +bed for pet bed for pet misc +bed frame bedframe bed +bed ladder bed ladder stairs +bed lamp bedside lamp lighting +bed light bed light misc +bed sheet bed sheet bed +bed small bed small misc +bed stand bed stand misc +bed table bed table table +bedding bedding objects +bedframe bedframe bed +bedpost bedpost bed +bedroom ceiling bedroom ceiling misc +bedroom table bedroom table chest_of_drawers +bedside cabinet bedside cabinet chest_of_drawers +bedside cabinet door bedside cabinet door misc +bedside cabinet drawer bedside cabinet drawer chest_of_drawers +bedside lamp bedside lamp lighting +bedside table bedside table chest_of_drawers +beer crate beer crate misc +bell bell misc +belt belt misc +bench bench seating +beside table bedside table chest_of_drawers +bicycle bicycle objects +bicycle helmet bicycle helmet objects +bicycle helmets bicycle helmets objects +bidet bidet toilet +big bag big bag objects +bike bicycle objects +billiard balls billiard balls objects +billiard cue billiard cue objects +billiard cues billiard cues objects +bin bin objects +binder binder misc +binders binders misc +birdhouse birdhouse misc +brackboard blackboard objects +blackboard blackboard objects +blanket blanket objects +blanket basket blanket basket objects +blankets blankets objects +blind blinds blinds +blinder blinds blinds +blinds blinds blinds +blouse blouse clothes +board board board_panel +board game board game misc +board games board games misc +board with keys board with keys objects +boards boards board_panel +boat model boat model misc +boiler boiler misc +bojler boiler misc +bonsai tree bonsai tree misc +book book objects +book cabinet book cabinet cabinet +book display book display misc +book rack book rack misc +book shape bookshelf shelving +book shelf bookshelf shelving +bookcase bookshelf shelving +books book objects +bookshelf bookshelf shelving +bookstand bookstand shelving +bootles bottles objects +boots boots objects +bottle bottle objects +bottle dispenser bottle dispenser objects +bottle of soap bottle of soap misc +bottle of detergent bottle of detergent misc +bottle of soap bottle of soap misc +bottle of water bottle of water objects +bottle of wine bottle of wine objects +bottle opener bottle opener misc +bottlels bottles objects +bottles bottles objects +bottles of water bottles of water objects +bottles of wine bottles of wine misc +bottom of stairs bottom of stairs misc +bouquet bouquet objects +bowl bowl objects +bowl of fruit bowl of fruit objects +bowl of fruits bowl of fruits objects +bowl of sweets bowl of sweets objects +bowl with sweets bowl with sweets objects +bowls bowls objects +box box objects +box with books box with books misc +box /w books box with books misc +box /w shoes box with shoes misc +box /w toys box with toys misc +box of tissues box of tissues misc +box of food box of food misc +box of fruit box of fruit misc +box of paper box of paper misc +box of something box of something misc +box of tissue box of tissue misc +box of tissues box of tissues misc +box pen box pen misc +box tissues box of tissues misc +box with jewellery box with jewelry misc +box with photos box with photos misc +box with tea box with tea misc +boxes boxes objects +boxes /w books boxes with books misc +boxing ring boxing ring objects +bread bread misc +bread bin bread bin misc +bread box bread box misc +breadbox breadbox misc +breifcase briefcase objects +bricks bricks misc +bridge bridge misc +briefcase briefcase objects +brochure brochure objects +brochures brochures objects +broom broom objects +broom stick broomstick misc +broomstick broomstick misc +brtochure brochure objects +brush brush objects +brushes brushes objects +bucket bucket objects +buckets buckets objects +buffet buffet counter +bulletin board bulletin board tv_monitor +bunk bed bunk bed bed +bureau bureau chest_of_drawers +bust bust objects +button button misc +cabidet cabinet cabinet +cabient cabinet cabinet +cabiinet cabinet cabinet +cabinet cabinet cabinet +cabinet door cabinet door misc +cabinet /otherroom cabinet /otherroom cabinet +cabinet /w clutter cabinet /w clutter cabinet +cabinet /w cluttered art cabinet /w cluttered art misc +cabinet clutter cabinet clutter misc +cabinet counter cabinet counter misc +cabinet door cabinet door misc +cabinet drawer cabinet drawer cabinet +cabinet kitchen cabinet kitchen cabinet +cabinet mirror cabinet mirror cabinet +cabinet parts cabinet parts cabinet +cabinet table cabinet table table +cable cable misc +cables cables misc +cage cage objects +cailing lamp ceiling lamp lighting +cair chair chair +cake cake misc +calander calendar misc +calculator calculator appliances +calendar calendar misc +camera camera objects +camping chair camping chair chair +can can objects +can of paint can of paint misc +canal canal misc +candel candle objects +candelabra candelabra objects +candle candle objects +candle holder candle holder objects +candle stand candle stand objects +candles candle objects +candlestick candlestick objects +canister canister misc +canoe canoe objects +canopy canopy misc +cans cans objects +cans of paint cans of paint objects +canvas canvas misc +cap cap clothes +car car objects +car model car model objects +carboard box cardboard box misc +card card misc +cardboard cardboard misc +cardboard box cardboard box misc +cardboardbox cardboard box misc +carpet carpet floor +carpet roll carpet roll misc +cart cart objects +cartboard cardboard misc +case case misc +cases cases misc +cash register cash register objects +casket casket misc +cat cat misc +cat bed cat bed bed +cat food cat food misc +cat food bag cat food bag misc +cat litter box cat litter box toilet +cat tree cat tree objects +cats toilet cat toilet toilet +cd cd objects +cd player cd player appliances +cd's cds objects +cds cds objects +cebinet cabinet cabinet +ceiiling ceiling ceiling +ceiling balk unknown unlabeled +ceiilng ceiling ceiling +ceiliing ceiling ceiling +ceiliing lamp ceiling lamp lighting +ceiling ceiling ceiling +ceiling lamp ceiling lamp lighting +ceiling light ceiling light lighting +ceiling vent ceiling vent misc +ceiling /otheroom ceiling ceiling +ceiling /otherroom ceiling ceiling +ceiling air vent ceiling vent misc +ceiling arch ceiling arch ceiling +ceiling bedroom ceiling bedroom misc +ceiling boarder ceiling boarder misc +ceiling border ceiling border ceiling +ceiling chasis ceiling chassis ceiling +ceiling corridor ceiling corridor misc +ceiling decorative lamp ceiling decorative lamp lighting +ceiling dome ceiling dome misc +ceiling door ceiling door misc +ceiling duck ceiling duct ceiling +ceiling duct ceiling duct ceiling +ceiling fan ceiling fan objects +ceiling fan lamp ceiling fan lamp lighting +ceiling fan vent ceiling fan vent misc +ceiling fire detector ceiling fire detector objects +ceiling fixture ceiling fixture misc +ceiling floor ceiling floor floor +ceiling hanger ceiling hanger objects +ceiling ladder ceiling ladder objects +ceiling lamp ceiling lamp lighting +ceiling lamp hanger ceiling lamp hanger lighting +ceiling lamp rail ceiling lamp rail lighting +ceiling lan ceiling fan objects +ceiling light ceiling light lighting +ceiling light fixture connection ceiling light fixture connection misc +ceiling llower ceiling lower misc +ceiling lower ceiling lower misc +ceiling molding ceiling molding misc +ceiling panel ceiling panel ceiling +ceiling pipe ceiling pipe ceiling +ceiling pipes ceiling pipe ceiling +ceiling support ceiling support ceiling +ceiling under staircase ceiling under staircase misc +ceiling under stairs ceiling under stairs misc +ceiling vent ceiling vent misc +ceiling wall ceiling wall misc +ceiling window ceiling window misc +ceiling/ wall west ceiling/west wall misc +ceiling/ west wall ceiling/west wall misc +ceiling/west wall ceiling/west wall misc +ceilling ceiling ceiling +ceilling lamp ceiling lamp lighting +celing ceiling ceiling +celing lamp ceiling lamp lighting +celing lower ceiling lower misc +central heating furnace central heating furnace appliances +ceramics ceramics objects +cerpet carpet floor +certain curtain curtain +chain chain objects +chair chair chair +chair /otherroom chair chair +chair /w clutter chair /w clutter chair +chair stand chair stand misc +chaise chaise chair +chaise longue chaise longue sofa +chamber pot chamber pot toilet +chandelier chandelier lighting +changing table changing table table +charger charger objects +chasis chassis objects +chess chess misc +chest chest misc +chest bench chest bench misc +chest drawer chest drawer misc +chest drawers chest of drawers chest_of_drawers +chest of drawer chest of drawer chest_of_drawers +chest of drawers chest of drawers chest_of_drawers +chest of drawres chest of drawers chest_of_drawers +chest of dreawers chest of drawers chest_of_drawers +chiar chair chair +child car seat child car seat objects +chimney chimney misc +christmas tree christmas tree misc +cieling ceiling ceiling +circular sofa circular sofa sofa +clarinet clarinet objects +cleaner cleaner objects +cleaner bottle cleaner bottle objects +cleaner brush cleaner brush objects +cleaning clutter cleaning clutter misc +cleaning fluid cleaning fluid objects +cleaning liquid cleaning liquid objects +cleaning paper cleaning paper objects +cleaning paste cleaning paste objects +cleaning powders/ liquids cleaning powders/ liquids objects +cleaning sponge cleaning sponge objects +cleaning spray cleaning spray objects +clock clock objects +closet closet cabinet +closet area for hanging clothes closet area for hanging clothes misc +closet door closet door door +closet floor closet floor misc +closet mirror wall closet mirror wall misc +closet rod closet rod objects +closet shelf closet shelf shelving +closet shelving closet shelving misc +closet storage area closet storage area misc +clotes dryer clothes dryer appliances +cloth cloth objects +cloth dryer cloth dryer appliances +cloth hanger cloth hanger objects +cloth hangers cloth hangers objects +cloth holder cloth holder objects +clothes clothes clothes +clothes rack clothes rack misc +clothes bag clothes bag misc +clothes container clothes container misc +clothes dryer clothes dryer appliances +clothes hamper clothes hamper objects +clothes hanger clothes hanger objects +clothes hanger rod clothes hanger rod objects +clothes on shelf clothes on shelf clothes +clothes rack clothes rack misc +clothing stand clothing stand misc +clutter clutter misc +coach coach misc +coaster coaster objects +coat coat clothes +coat hanger coat hanger objects +coat rack coat rack misc +cofee table coffee machine appliances +coffe machine coffee machine appliances +coffe table coffee table table +coffee machine coffee machine appliances +coffee maker coffee maker appliances +coffee mug coffee mug objects +coffee table coffee table table +coffie machine coffee machine appliances +coffiee machine coffee machine appliances +coffiee table coffee table table +coffy table coffee table table +column column misc +compound wall compound wall misc +compressor compressor appliances +computer computer objects +computer chair computer chair chair +computer desk computer desk table +computer equipment computer equipment misc +computer mouse computer mouse objects +computer tower computer tower objects +condensation furnace condensation furnace objects +condiment condiment misc +cone cone misc +conference phone conference phone appliances +console console objects +console pad console pad objects +console pad charger console pad charger misc +construction stuff construction stuff misc +container container objects +containers containers objects +control control objects +control panel control panel objects +controller controller objects +coocker cooker objects +cooffe machine coffee machine appliances +cookbook cook book objects +cooker cooker objects +cooker hood cooker hood appliances +cookies cookies misc +copier copier objects +copier machine copier machine misc +cork board cork board objects +cornice cornice misc +cosmetic cosmetic objects +cosmetics cosmetics objects +couch couch sofa +counter counter counter +counter desk counter desk table +counter door counter door misc +countertop countertop counter +countertop /otherroom countertop /otherroom counter +countertop item countertop item misc +courtain curtain curtain +cover cover misc +cradle cradle bed +crate crate objects +crayon crayon objects +crib crib misc +cross cross objects +cross-trainer cross-trainer gym_equipment +crutches crutches objects +cuddly toy cuddly toy misc +cup cup objects +cupboard cabinet cabinet +cups cups objects +curatin curtain curtain +curb curb objects +curtain curtain curtain +curtain rod curtain rod curtain +curtain /otheroom curtain curtain +curtain bar curtain bar curtain +curtain box curtain box curtain +curtain hanger curtain hanger curtain +curtain rail curtain rail misc +curtain rod curtain rod curtain +curtain rod cover curtain rod cover curtain +curtain valence curtain valence curtain +curtains curtain curtain +cushiom cushion cushion +cushion cushion cushion +cutlery cutlery objects +cutting board cutting board objects +cutting boards cutting board objects +cyp cyp misc +dacorative plant decorative plant plant +dartboard dartboard objects +deccoration decoration objects +deck chair deck chair chair +decoartive plate decorative plate misc +decoder decoder misc +decooration decoration objects +decor decor objects +decor plate decorative plate misc +decoratinon decoration objects +decoratiom decoration objects +decoration decoration objects +decoration bowl decorative bowl objects +decoration plan decorative plant misc +decoration plant decorative plant misc +decoration window decorative window misc +decorations decoration objects +decorative bottle decorative bottle objects +decorative bowl decorative bowl objects +decorative ceramic decorative ceramic objects +decorative cloth decorative cloth objects +decorative dinnerware decorative dinnerware objects +decorative frame decorative frame objects +decorative lamp decorative lamp objects +decorative lantern decorative lantern objects +decorative mask decorative mask objects +decorative pl;ant decorative plant misc +decorative plant decorative plant misc +decorative plants decorative plant misc +decorative plate decorative plate misc +decorative quilt decorative quilt misc +decorative tray decorative tray objects +decorative vase decorative vase objects +decorative vessel decorative vessel objects +decoratrion decoration objects +decoratrive plant decorative plant misc +decorayion decoration objects +decotarion decoration objects +decration decoration objects +dehumidifier dehumidifier objects +den den misc +desk desk table +desk cabinet desk cabinet cabinet +desk chair desk chair chair +desk clutter desk clutter misc +desk door desk door cabinet +desk lamp desk lamp lighting +desk organizer desk organizer objects +detergent bottle detergent bottle objects +detergent bottles detergent bottle objects +detergents detergent misc +device device misc +device/ probably paper shredder device misc +dining chair dining chair chair +dining table dining table table +dinner chair dinner chair chair +dinner table dinner table table +dinnerware dinnerware objects +dinnerware decoration decorative dinnerware objects +diplom diploma misc +diploma diploma misc +dirt ground dirt ground misc +dish dish objects +dish cabinet dish cabinet cabinet +dish dryer dish dryer appliances +dish rack dish rack objects +dish washer dishwasher appliances +dish with food dish objects +dishawasher dishwasher appliances +dishes dishes objects +dishes dryer dish dryer appliances +dishrag dishrag objects +dishwasher dishwasher appliances +dishwasher machine dishwasher appliances +disk dish objects +dispenser dispenser objects +display display tv_monitor +display cabinet display cabinet cabinet +display case display case objects +display of pictures display of pictures picture +display table display table table +document document misc +document holder document holder objects +documents document misc +dog bed dog bed bed +dog leash dog leash misc +dog toy dog toy objects +doile doily misc +doily doily misc +doll doll objects +dool doll objects +door door door +door frame door frame door +door hinge door hinge misc +door /otherroom door /otherroom door +door cabinet door cabinet cabinet +door drame door frame door +door f door frame door +door fame door frame door +door fframe door frame door +door frame door frame door +door framr door frame door +door handle door handle misc +door hanger door handle misc +door hine door hinge misc +door hinge door hinge misc +door kinge door hinge misc +door knob door knob misc +door mat doormat floor +door rame door frame door +door slide sliding door door +door stoper door stopper misc +door stopper door stopper misc +door stoppper door stopper misc +door window door window misc +door/window door/window misc +door/window frame door/window frame misc +doorbell doorbell misc +doorf rame door frame door +doorframe door frame door +doormat doormat floor +doorpost doorpost objects +doors door door +doorstep doorstep objects +doorway doorway door +dorr door door +double armchair double armchair chair +drain drain objects +drainage drainage misc +drainpipe drainpipe objects +draw draw objects +drawer drawer chest_of_drawers +drawer cabinet drawer cabinet cabinet +drawer cart drawer cart objects +drawer desk drawer desk misc +drawer sink table drawer sink table misc +drawers drawers chest_of_drawers +drawers for clothes drawers for clothes misc +drawing drawing misc +dreser dresser chest_of_drawers +dress dress clothes +dresser dresser chest_of_drawers +dressing table dressing table table +dried flowers dried flowers objects +drill drill misc +drum drum objects +drums drum objects +dryer clothes dryer appliances +drywall board drywall board misc +drywall boards drywall board misc +duct duct misc +dumb bell dumbbell gym_equipment +dumb-bells dumbbell gym_equipment +dumbbel dumbbell gym_equipment +dumbbell dumbbell gym_equipment +dumbbell handles dumbbell gym_equipment +dumbbells dumbbell gym_equipment +dust bin dustbin objects +dustbin dustbin objects +dustpan dustpan objects +duvet duvet misc +dvd dvd objects +dvd movies dvd objects +dvd player dvd player objects +dvds dvd objects +easel easel shelving +easels easel shelving +easy chair easy chair chair +electric box electric box misc +electric cable electric cable misc +electric cord electric cord misc +electric device electric device misc +electric drums electric drum objects +electric freshener electric freshener appliances +electric guitar electric guitar objects +electric guitar pack electric guitar objects +electric heater electric heater appliances +electric hub electric hub objects +electric installation electric installation misc +electric kettle electric kettle appliances +electric outlet electric outlet objects +electric percussion electric percussion appliances +electric plug electric plug objects +electric toothbrush electric toothbrush appliances +electric wire electric wire misc +electric wire casing electric wire casing misc +electrical box electrical box objects +electrical controller electrical controller misc +electrical device electrical device misc +electrical installation electrical installation misc +electrical switchboard electrical switchboard misc +electricity box electricity box misc +electronics electronics misc +elephant sculpture elephant sculpture misc +elevator elevator misc +elevator door elevator door door +emergency sign emergency sign misc +end table end table table +entertainment set entertainment set misc +entrance arch entrance arch misc +entry phone entry phone objects +excercise mat exercise mat gym_equipment +excersie ball exercise ball gym_equipment +excersie equipment exercise equipment gym_equipment +exercise ball exercise ball gym_equipment +exercise bike exercise bike gym_equipment +exercise equipment exercise equipment gym_equipment +exercise ladder exercise ladder gym_equipment +exercise machine exercise machine misc +exercise mat exercise mat gym_equipment +exercise mat roll exercise mat roll misc +exercising blocks exercising blocks gym_equipment +exhaust pipe exhaust pipe misc +exhibition panel exhibition panel misc +exhibition picture exhibition picture picture +exhibition table exhibition table table +exhibition window exhibition window window +exhibition window frame exhibition window frame window +exit sign exit sign misc +extension cord extension cord misc +extension lead extension lead misc +extinguisher fire extinguisher objects +extractor extractor misc +extractor hood extractor hood misc +eyeglasses eyeglasses objects +fan fan objects +fan-coil fan coil objects +fan air vent fan air vent misc +fans fan objects +faucet faucet objects +fence fence misc +fframe frame misc +figure figure misc +figurine figurine misc +file file misc +file binder file binder misc +file cabinet file cabinet cabinet +files file misc +fire alarm fire alarm objects +fire detector fire detector objects +fire etenguisher fire extinguisher objects +fire extinguisher fire extinguisher objects +fire pit fire pit objects +fire screen fire screen objects +fire sprinkler fire sprinkler misc +firebox firebox objects +fireplace fireplace fireplace +fireplace brush fireplace brush misc +fireplace floor fireplace floor misc +fireplace mirror fireplace mirror misc +fireplace sconce fireplace sconce misc +fireplace shelf fireplace shelf misc +fireplace tool set fireplace tool set misc +fireplace utensil fireplace utensil misc +fireplace wall fireplace wall misc +firewood firewood misc +firewood bag firewood bag misc +firewood chest firewood chest misc +firewood holder firewood holder misc +fish tank fish tank misc +fishing pole fishing pole misc +fishing rod fishing rod misc +fitness ball fitness ball gym_equipment +flag flag misc +flashlight flashlight objects +floer floor floor +floor floor floor +floor lamp floor lamp lighting +floor mat floor mat floor +floor vent floor vent misc +floor /otherroom floor /otherroom floor +floor /outside floor /outside floor +floor lamp floor lamp lighting +floor mat floor mat floor +floor stand floor stand misc +floor vent floor vent misc +floormat floor mat floor +florr floor floor +flower flower plant +flower vase flower vase objects +flower pot flowerpot objects +flower stand flower stand misc +flower vase flower vase objects +flowerbed flowerbed objects +flowerpot flowerpot objects +flowers flowers plant +flowers in vase flower vase objects +fluorescent light fluorescent light misc +flush flush misc +flush button flush button misc +flush buttons flush button misc +flush push flush push misc +folded chair folded chair chair +folded chairs folded chair chair +folded table folded table table +folder folder objects +folders folder objects +folding chair folding chair chair +folding stand folding stand misc +food food misc +food processor food processor appliances +food stand food stand misc +food tray food tray objects +foor frame floor frame floor +foor lamp floor lamp lighting +foosball game table foosball game table table +foosball table foosball table table +foot rest footrest stool +foot spa foot spa misc +foot stand foot stand misc +football football objects +footrest footrest stool +footstool footstool stool +fork fork objects +frame frame misc +frame door frame misc +frame part frame misc +framed document document misc +framed mirror mirror mirror +framed photo photo picture +framed photos photos picture +framed picture picture picture +framed pictures pictures picture +framed text framed text misc +freezer freezer appliances +fridge refrigerator appliances +fruit fruit objects +fruit bowl fruit bowl objects +fruits fruit objects +frying pan frying pan objects +fume cupboard fume cupboard misc +fur fur misc +fur carpet fur carpet floor +furnace furnace misc +furniture furniture furniture +furniture parts furniture furniture +fuse box fuse box misc +fuse panel fuse panel misc +game board game board objects +game console game console objects +gap gap objects +garage door garage door door +garage door frame garage door frame misc +garage door motor garage door motor misc +garage door opener garage door opener misc +garage door opener bar garage door opener bar misc +garage door opener motor garage door opener motor misc +garage door opener railing garage door opener railing misc +garage door railing garage door railing misc +garage light garage light misc +garden bench garden bench seating +garden chair garden chair chair +garden deck garden deck misc +garden hose hose misc +garden swing swing objects +gas box gas box objects +gas container gas container objects +gas furnace gas furnace appliances +gas meter gas meter misc +gate gate misc +gauge gauge misc +gift gift misc +glass glass objects +glass bottle bottle objects +glass container glass container objects +glass door glass door door +glass pane glass pane misc +glasses glasses objects +globe globe misc +globe stand globe stand misc +glove gloves objects +gloves gloves objects +glue glue objects +goal goalpost objects +golf sticks golf sticks objects +grab bar grab bar misc +gramophone gramophone objects +grandfather clock grandfather clock objects +grass grass plant +grate grate misc +gravel gravel floor +grill grill misc +groceries groceries objects +guitar guitar objects +guitar amp amplifier objects +guitar amplifier amplifier objects +guitar box amplifier objects +guitar case guitar case objects +guitar case suit guitar case cover objects +guitar cases guitar cases objects +guitar frame guitar frame objects +guitar pedals guitar pedals objects +guitar stand guitar stand objects +guitar straps guitar straps objects +gun gun objects +gutter gutter misc +gym equipment gym equipment gym_equipment +gym mat gym mat floor +gym rope gym rope objects +gym stepper gym stepper objects +gym-equipment gym equipment gym_equipment +hair dryer hair dryer objects +hairbrush hair brush objects +hairdryer hair dryer objects +hammer hammer objects +hammock hammock bed +hand cart handcart objects +hand cloth hand cloth misc +hand drier hand dryer objects +hand soap hand soap objects +hand towel hand towel objects +handbag handbag objects +handkerchiefs handkerchiefs objects +handle handle objects +handrail handrail railing +handhold handrail railing +handshower showerhead shower +hanger hanger objects +hangers hanger objects +hanging clothes hanging clothes clothes +hat hat clothes +hatch hatch misc +hats hats objects +headboard headboard bed +headphones headphones objects +headset headset objects +hearth hearth misc +heat vent vent misc +heater heater objects +heater piping heater piping misc +heater piping heater piping misc +heating furnace part furnace misc +heating vent vent misc +herb basket basket objects +hi hat hi-hat objects +hi hat stand hi-hat stand objects +high chair highchair chair +high shelf high shelf misc +highchair highchair chair +highchairs highchair chair +holder holder objects +holy cross holy cross misc +hood range hood objects +hook hook misc +hose hose misc +hot drinks machine beverage dispenser objects +hot water/cold water knob hot water/cold water knob misc +hourglass hourglass objects +hoverboard hoverboard objects +hunting trophy hunting trophy misc +hutch hutch misc +ice maker ice maker objects +icebox icebox appliances +identifier identifier misc +image picture picture +indow curtain window curtain curtain +induction hob stovetop misc +information information misc +insect door door screen door +door screen door screen door +installation installation objects +instrument instrument objects +iron iron objects +iron board iron board misc +ironing board ironing board objects +island island counter +jacket jacket clothes +jacuzzi jacuzzi misc +jar jar objects +jars jars objects +jewellery jewelry objects +jewellery ehibition jewelry objects +jewellery exhibition jewelry objects +jewellery exposition jewelry objects +jewelry jewelry objects +jewelry box jewelry box misc +jewlery box jewelry box misc +joga mat yoga mat misc +jug jug objects +junk junk objects +keg keg objects +kegerator keg objects +kettle kettle objects +keyboard keyboard objects +keyboard box keyboard box objects +keyboard case suit keyboard cover objects +keyboard piano keyboard piano objects +keyboard stand keyboard stand objects +keys keys objects +kitchen appliance kitchen appliance appliances +kitchen board cutting board objects +kitchen cabinet kitchen cabinet cabinet +kitchen cabinet door kitchen cabinet door cabinet +kitchen cabinet drawer kitchen cabinet drawer cabinet +kitchen cabinet lower kitchen cabinet lower cabinet +kitchen ceiling kitchen ceiling ceiling +kitchen chair kitchen chair chair +kitchen coocking cabinet kitchen cabinet cabinet +kitchen counter kitchen counter counter +kitchen counter support kitchen counter support misc +kitchen countertop kitchen counter counter +kitchen countertop item kitchen countertop item misc +kitchen countertop items kitchen countertop items misc +kitchen coutertop item kitchen counter item objects +kitchen decoration kitchen decoration objects +kitchen extractor kitchen extractor misc +kitchen glowes kitchen gloves objects +kitchen handle kitchen handle objects +kitchen hood range hood objects +kitchen island kitchen island counter +kitchen knife set kitchen knife set objects +kitchen lower cabinet kitchen lower cabinet cabinet +kitchen lower shelf kitchen lower shelf shelving +kitchen shelf kitchen shelf shelving +kitchen sink kitchen sink sink +kitchen sink cabinet kitchen sink cabinet cabinet +kitchen sitting kitchen seating seating +kitchen stuff clutter misc +kitchen table kitchen table table +kitchen tools kitchen utensils objects +kitchen top kitchen top counter +kitchen towel kitchen towel objects +kitchen untensils kitchen utensils objects +kitchen upper cabinet kitchen cabinet cabinet +kitchen utensil kitchen utensil objects +kitchen utensils kitchen utensils objects +kitchen wall kitchen wall wall +kitchen walll kitchen wall wall +kitchenware kitchenware objects +kitchhen cabinet door kitchen cabinet door cabinet +kitcyhen cabinet kitchen cabinet cabinet +knife knife objects +knife holder knife holder misc +knife set knife set objects +knife stand knife stand objects +knob knob misc +knofe set knife set objects +l-shaped sofa l-shaped sofa sofa +lace doily lace doily objects +lacy doily lace doily objects +ladder ladder stairs +lamp lamp lighting +lamp ceiling ceiling lamp lighting +lamp desk desk lamp lighting +lamp shade lamp shade lighting +lamp stand lamp stand misc +lamp table lamp table table +lamps lamp lighting +lampshade lampshade lighting +landing landing floor +lantern lantern lighting +laptop laptop objects +laundry laundry clothes +laundry bag laundry bag misc +laundry basket laundry basket objects +laundry machine laundry machine misc +launger lounger seating +lawn lawn floor +lawn mower lawn mower objects +lawnmower lawn mower objects +leaflet leaflets objects +leaflets leaflets objects +led tv led tv misc +led tv led tv misc +ledge ledge objects +ledtv led tv misc +leg rest leg rest misc +letter document misc +level level objects +lid lid misc +lids lid misc +light light lighting +light fixture light fixture lighting +light switch light switch objects +lighter lighter objects +lighting fixture lighting fixture objects +lighting grid lighting fixture objects +lights light lighting +liquid liquid misc +liquid container liquid container objects +liquid cleaner liquid cleaner objects +liquid soap liquid soap misc +lmap lamp lighting +locker locker misc +loft hatch hatch misc +logs firewood misc +lounge chair lounge chair chair +lounger lounger seating +lower cabinet lower cabinet cabinet +luggage luggage objects +lundry basket laundry basket objects +machine machine objects +magazine magazine objects +magazine rack magazine rack misc +magazines magazines objects +magazines, albums magazines objects +magazines, albums, bookshelf magazines objects +magazines, books and albums magazines objects +magic marker magic marker objects +magic marker box magic marker box objects +mail mail objects +mailbox mailbox objects +make up accesories makeup accessories objects +mannequin mannequin objects +mantel mantel fireplace +mantle mantle fireplace +map map objects +mascot mascot objects +mascots mascots objects +mask decoration decoration objects +massage bed massage bed bed +mat mat floor +mat floor mat floor +material material misc +mattress bed bed +measuring tape measuring tape objects +medal medal objects +medal collection medal collection objects +media console media console misc +medical lamp medical lamp lighting +medical object medical object objects +menu menu objects +menu board menu board objects +meshwork meshwork misc +meter meter misc +microphone microphone objects +microphone accesory microphone accessory objects +microvawe microwave appliances +microwave microwave appliances +microweave microwave appliances +midi keyboard keyboard piano objects +mini fridge mini fridge appliances +miocrowave microwave appliances +mircowave microwave appliances +miror mirror mirror +mirror mirror mirror +mirror /otherroom mirror /otherroom mirror +mirror door mirror door misc +mirror frame mirror frame mirror +mirror stand mirror mirror +mixer mixer misc +mobile mobile objects +model model objects +modem modem objects +molding molding wall +monitor monitor tv_monitor +moose head/sculpture/hunting trophy moose head/sculpture/hunting trophy misc +mop mop objects +mortar mortar misc +motion detector motion detector misc +motion sensor motion detector misc +motorcycle motorcycle objects +mouse mouse objects +mousepad mousepad objects +mug mug objects +multi-gym gym equipment gym_equipment +music album shelf music album shelf shelving +music player music player objects +music stand music equipment stand objects +musical equipment music equipment objects +napkin napkin objects +napkins napkins objects +newspaper newspaper misc +newspaper basket newspaper basket objects +niche niche wall +nighstand nightstand chest_of_drawers +night lamp night lamp lighting +night stand nightstand chest_of_drawers +night table nightstand chest_of_drawers +nightsand nightstand chest_of_drawers +nightstand nightstand chest_of_drawers +nigtstand nightstand chest_of_drawers +note note objects +notebook notebook objects +notebooks notebooks objects +notes notes objects +noticeboard note board objects +oar oar objects +object object objects +objects objects objects +office chair office chair chair +office wall office wall wall +office drawer drawers chest_of_drawers +office phone phone objects +office stuff stationery objects +office table office table table +office utensils utensil objects +oil lamp oil lamp lighting +organizer drawers drawers chest_of_drawers +ornament ornament objects +ornament flower flower plant +ornament plant plant plant +ornamental plant plant plant +ornamental plate plate objects +ornaments ornament objects +ottoman ottoman stool +outflow unknown unlabeled +outlet outlet objects +outside unknown unlabeled +oven oven appliances +oven and stove oven and stove misc +oven hood range hood objects +oven vent oven vent misc +overhang overhang misc +overlay unknown unlabeled +package package objects +pad pad misc +painiting painting objects +paint unknown unlabeled +painting painting objects +painting frame painting frame misc +painting /otherroom painting /otherroom objects +painting frame painting frame misc +painting roll painting roll objects +painting rolls painting rolls objects +painting stuff painting stuff objects +painting tray painting tray objects +paintng painting objects +paintning painting objects +pair of shoes shoes objects +paiting painting objects +pan pan objects +panel panel board_panel +panel screen panel screen misc +panel wall wall panel wall +paneling paneling wall +pantry pantry misc +paper paper objects +paper holder paper holder objects +paper shelf shelf shelving +paper storage paper storage misc +paper towel paper towel towel +paper towel dispenser paper towel dispenser misc +paper towel holder paper towel holder objects +paper towels paper towels towel +papers papers objects +parapet parapet misc +partial partial misc +partition partition misc +patio patio misc +patio chair patio chair chair +patio floor patio floor misc +pavement pavement floor +paving pavement floor +payment terminal payment terminal objects +pc mouse mouse objects +pc tower pc tower misc +pedestal pedestal misc +peinting painting objects +pen pen objects +pen cup pen cup objects +pencil pencil objects +pencil case pencil case objects +pencil holder pencil holder objects +pendant pendant objects +pendrive pen drive objects +perfume perfume objects +pet bed pet bed objects +pet bowl pet bowl objects +phillar pillar column +phone phone objects +photo photo picture +photo mount photo mount misc +photo mounts photo mounts misc +photo stand photo stand objects +photography photo picture +photos photos picture +piano piano objects +piano bench piano bench seating +piano lower part piano objects +piano stool piano stool stool +pictrure picture picture +picture picture picture +picture frame picture frame picture +picture frame picture frame picture +pictured frame picture frame picture +pictures pictures picture +picure picture picture +piillow pillow cushion +pile of boxes boxes objects +pile of cups cups objects +pile of clothes clothes clothes +pile of magazines pile of magazines misc +pile of papers papers objects +pile of stuff unknown unlabeled +pillar pillar column +pilllow pillow cushion +pillow pillow cushion +pillow seat pillow seat seating +pillows pillow cushion +ping pong table ping pong table table +pipe pipe misc +pipe part pipe misc +pipes pipe misc +piping pipe misc +pitcher pitcher objects +pitchfork pitchfork objects +place mat place mat objects +plane plane misc +plank plank misc +planner planner misc +plant plant plant +plant ornament plant ornament misc +planter pot objects +plants plant plant +plasma tv tv tv_monitor +plastic bag plastic bag objects +plate plate objects +plate of food plate of food misc +plates plates objects +platform platform misc +platter platter misc +player music player objects +playpen playpen misc +plenum box plenum box misc +pliers pliers objects +plstes plates objects +plug plug misc +plunger plunger objects +plush toy plush toy objects +podest podium objects +podium podium objects +pole pole objects +poles poles objects +pomp pump objects +pool pool misc +pool stick pool stick objects +pool table pool table table +porcelain porcelain objects +portrait portrait picture +post post column +poster poster picture +poster figure poster picture +pot pot objects +pot lid pot lid objects +poto mount photo mount misc +pots pot objects +potty potty objects +pouches pouches objects +poufe pouffe seating +pouffe pouffe seating +powder soap powder soap objects +power breaker box power breaker box misc +power cord power cord objects +power outlet outlet objects +power strip power strip objects +prduct box product box objects +press press objects +pressure washer pressure washer objects +price tag price tag objects +printer printer objects +product product objects +product box product box objects +product boxes product boxes objects +products products objects +projector projector objects +projector screen projector screen misc +prop prop misc +psinting painting objects +public tap drinking fountain misc +pump pump objects +punchbag punchbag objects +puncher hole puncher objects +puppet puppet objects +puppet cat puppet objects +purse purse objects +purses purse objects +rack rack shelving +rack of weights rack of weights misc +rack with shoes rack with shoes shelving +radfiator radiator objects +radiator radiator objects +radio radio misc +rafter rafter beam +rag rag objects +ragdoll ragdoll objects +ragdoll cat ragdoll cat objects +rail rail railing +railing railing railing +rain shower showerhead shower +rake rake objects +range hood range hood objects +receipt printer receipt printer objects +receipt spike receipt spike objects +recessed cubby recessed cubby misc +recessed shelving recessed shelving shelving +recessed wall recessed wall misc +recliner recliner chair +record player record player objects +records records objects +recuperator recuperator objects +recycle bin recycle bin misc +refridgerator refrigerator appliances +refrigearator refrigerator appliances +refrigerator refrigerator appliances +refrigerator cabinet refrigerator cabinet cabinet +relief relief wall +remote remote control objects +remote control remote control objects +remote controller remote control objects +rice cooker rice cooker objects +riser riser misc +robe robe clothes +rock rock objects +rocking chair rocking chair chair +rocking horse rocking horse objects +rocks rock objects +rod rod objects +rods rods objects +roll roll objects +rolled carpet rolled carpet objects +roller blind blinds blinds +rolling pin rolling pin objects +rolls of toilet paper toilet paper objects +roof roof ceiling +roomba roomba misc +rope rope objects +round chair round chair chair +round cushion round cushion misc +router router objects +row of theater chairs row of theater chairs misc +rug rug floor +ruler ruler objects +safe safe objects +salt and pepper salt and pepper objects +salt and pepper grinder salt and pepper grinder objects +salt pepper salt and pepper objects +salver salver objects +sandals sandals objects +saturator saturator appliances +sauna sauna appliances +sauna bowl sauna bowl objects +sauna ceiling ceiling ceiling +sauna floor floor floor +sauna heat rocks sauna heat rocks misc +sauna heater sauna heater misc +sauna oven sauna oven appliances +sauna seat sauna seat seating +sauna support sauna support misc +sauna wall wall wall +saw saw objects +saxophone saxophone objects +scale scale objects +scales scale objects +scanner scanner objects +scarf scarf clothes +schedule schedule misc +schoe shoe objects +sconce sconce lighting +scoop scoop objects +screen screen curtain +screen frame screen frame misc +screw box screw box objects +screwdriver screwdriver objects +sculpture sculpture objects +seat seat seating +secretary secretary misc +security camera security camera objects +self-closing mechanism self-closing mechanism misc +sensor sensor objects +separator separator objects +set of armchairs set of armchairs misc +set of boxes set of boxes misc +set of cosmetics set of cosmetics objects +set of hangers set of hangers objects +set of knives set of knives misc +set of pictures set of pictures picture +set of towels set of towels towel +set of valves set of valves appliances +sewing box sewing box appliances +sewing machine sewing machine appliances +sewing set sewing set appliances +sewing tools sewing tools appliances +shade shade objects +shade rail shade rail misc +shades shades blinds +shalf shelf shelving +shampoo shampoo objects +shedule schedule misc +sheet sheet misc +sheet music sheet music objects +sheet music stand sheet music stand objects +sheets sheets misc +sheets/ clothes sheets / clothes clothes +shelf shelf shelving +shelf /w art shelf with art shelving +shelf /w clutter shelf with clutter shelving +shelf clutter shelf clutter shelving +shelf cubby shelf cubby misc +shelf with cosmetics shelf with cosmetics shelving +shelf with shoes shelf with shoes shelving +shelf/cabinet shelf / cabinet misc +shelve shelf shelving +shelves shelving shelving +shelving shelving shelving +ship model ship model objects +ship toy ship toy objects +shirt shirt clothes +shisha shisha appliances +shlef shelf shelving +shoe shoe objects +shoe cabinet shoe cabinet cabinet +shoe case shoe case cabinet +shoe rack shoe rack shelving +shoe shelf shoe shelf shelving +shoehorn shoehorn objects +shoes shoes objects +shoes on shelf shoes on shelf shelving +shoes rack shoes rack shelving +shop shelf shop shelf shelving +shoulder bag shoulder bag objects +shovel shovel misc +shower shower shower +shower floor shower floor shower +shower knob shower knob misc +shower wall shower wall shower +shower bar shower bar misc +shower base shower base shower +shower battery shower battery misc +shower bench shower bench seating +shower cabin shower cabin misc +shower cabinet shower cabinet cabinet +shower caddy shower caddy misc +shower case shower case misc +shower ceiling shower ceiling shower +shower ceiling lamp shower ceiling lamp lighting +shower cockpit shower cockpit shower +shower cosmetics shower cosmetics objects +shower curtain shower curtain curtain +shower curtain bar shower curtain bar misc +shower curtain rod shower curtain rod curtain +shower dial shower dial misc +shower door shower door shower +shower door frame shower door frame shower +shower door knob shower door knob misc +shower floor shower floor shower +shower frame shower frame shower +shower glass shower glass misc +shower grab bar shower grab bar misc +shower handle shower handle misc +shower handrail shower handrail railing +shower hanger shower hanger misc +shower head showerhead shower +shower hose shower hose misc +shower hose/head shower hose/head misc +shower knob shower knob misc +shower mat shower mat floor +shower mirror shower mirror mirror +shower pipe shower pipe misc +shower rail shower rail misc +shower rod shower rod curtain +shower seat shower seat misc +shower shelf shower shelf shelving +shower soap shelf shower soap shelf misc +shower stall shower stall misc +shower step shower step shower +shower tap shower tap misc +shower tray shower tray shower +shower tub shower tub bathtub +shower utensill shower utensil objects +shower valve shower valve misc +shower wall shower wall shower +shower wall cubby shower wall cubby shower +shower window frame shower window frame window +shower-bath cabinet shower-bath cabinet cabinet +showerhead showerhead shower +shredder shredder appliances +shutter shutter blinds +shutters shutters blinds +side table side table table +sideboard sideboard misc +sign sign misc +silicone gun silicone gun objects +silicone tube silicone tube objects +sink sink sink +sink cabinet sink cabinet cabinet +sink pipe sink pipe misc +sink table sink table table +sink tap sink tap misc +sink/basin sink/basin misc +sitting bench sitting bench seating +skateboard skateboard objects +skates skates objects +ski ski objects +skirting board skirting board wall +sky sky misc +skylight skylight window +slab slab objects +sled sled objects +sledge sledge objects +sleeping bag sleeping bag objects +sliding door sliding door door +sliding glass door sliding glass door misc +slippers slippers objects +small table table table +small table/stand small table/stand misc +smoke alarm smoke alarm objects +smoke detector smoke detector misc +snack snack objects +soap soap objects +soap bottle soap bottle objects +soap dish soap dish objects +soap dish cubby soap dish cubby misc +soap dispenser soap dispenser objects +soap dispenser shelf in shower soap dispenser shelf in shower misc +soap tray soap tray objects +soapbox soapbox misc +socket socket objects +socks socks clothes +sofa sofa sofa +sofa chair sofa chair chair +sofa seat sofa seat misc +sofa set sofa set sofa +soft chair soft chair chair +solarium solarium misc +solarium door solarium door door +sombrero sombrero objects +sopp bottle soap bottle objects +sound bar soundbar appliances +soundbar soundbar appliances +spa armchair spa armchair chair +spa bathtub spa bathtub bathtub +spa bench spa bench seating +spatula spatula objects +speaker speaker objects +speaker stand speaker stand cabinet +spice boxes spice boxes objects +spice rack spice rack shelving +spices spices objects +spirit level spirit level objects +sponge sponge objects +spoon spoon objects +spray spray objects +spray can spray can objects +sprinkler sprinkler objects +square square objects +stack stack misc +stack of albums stack of albums objects +stack of bags stack of bags objects +stack of binders stack of binders objects +stack of blankets stack of blankets objects +stack of book stack of books objects +stack of books stack of books objects +stack of books/ papers stack of books / papers objects +stack of boxes stack of boxes objects +stack of cd's stack of cds objects +stack of cds stack of cds objects +stack of chairs stack of chairs chair +stack of clothes stack of clothes clothes +stack of files stack of files objects +stack of jackets stack of jackets clothes +stack of magazines stack of magazines objects +stack of music stands stack of music stands misc +stack of paper stack of papers misc +stack of papers stack of papers misc +stack of pillows stack of pillows cushion +stack of plates stack of plates objects +stack of pots stack of pots objects +stack of product boxes stack of product boxes objects +stack of shoes stack of shoes objects +stack of stuff stack of stuff misc +stack of t shirts stack of t-shirts clothes +stack of towels stack of towels towel +stack of trays stack of trays objects +stack of yarns stack of yarns clothes +stacked chair stacked chair chair +stacked chairs stacked chair chair +stage stage misc +stained glass stained glass window +stair stair stairs +stair frame stair frame misc +stair handle stair handle railing +stair railing banister railing +stair step stair step misc +stair wall stair wall misc +staircaise handrail staircase handrail misc +staircase staircase stairs +staircase handrail staircase handrail misc +staircase handrair staircase handrail misc +staircase trim staircase trim misc +staircase wall staircase wall wall +stairs stairs stairs +stairs railing stairs railing misc +stairs skirt stairs skirt misc +stairs trim stairs trim stairs +stairs wall stairs wall wall +stairwell stairwell stairs +stampler stapler objects +stand stand table +stand/small table stand/small table table +stapler stapler objects +star star objects +stationary stationery objects +stationery stationery objects +statue statue objects +statue/art statue/art misc +steel plate steel plate objects +step step stairs +step stool step stool stool +stereo stereo appliances +stereo set stereo set appliances +stick stick misc +sticker book sticker book objects +sticky notes sticky notes objects +stoll stole clothes +stone stone objects +stone bench stone bench seating +stone support structure stone support structure misc +stones stones misc +stonework stonework misc +stoo stool stool +stool stool stool +stools stools stool +storage storage objects +storage bin storage bin misc +storage box storage box misc +storage cabinet storage cabinet cabinet +storage shelving storage shelving shelving +storage space storage space misc +storage unit storage unit furniture +stove stove appliances +stove and oven oven appliances +stove door door door +stove utensil utensil objects +stove utensils utensil objects +stovetop stovetop misc +strands strings misc +stricker book book objects +strings strings misc +stripes decoration objects +stroke stroke misc +strongbox strongbox objects +stuffed animal stuffed animal objects +stuffed duck stuffed animal objects +subwoofer subwoofer objects +suitcase luggage objects +sunbed sunbed furniture +support support misc +support beam support beam beam +support stand stand table +supporting beam support beam beam +supporting structure support misc +surface surface counter +surfboard surfboard objects +sweets bowl bowl objects +swing swing objects +switch switch objects +switches switch objects +swivel chair swivel chair chair +t shirt t-shirt clothes +table table table +table /w books table table +table chair chair chair +table cloth table cloth table +table clutter clutter misc +table lamp table lamp lighting +table on wheels table table +table pad table pad table +table plant plant plant +table shelf shelf shelving +table stand table stand misc +table tennis table table tennis table table +table top table table +table tray tray objects +table vase vase objects +table border unknown unlabeled +tablecloth table cloth table +tablet tablet table +tabletop table table +tabletop box box objects +tabletop games game objects +tabletop trinket trinket objects +tableware flatware objects +tailet toilet toilet +tambourine tambourine objects +tank tank misc +tap tap objects +tap dial knob misc +tap/ water source tap objects +tapestry tapestry objects +tea box box objects +tea boxes boxes objects +teapot teapot objects +teaset tea set objects +teddy bear stuffed animal objects +teepee toilet paper objects +telephone telephone objects +telephpne telephone objects +telescope telescope objects +temperature control thermostat objects +tennis racket tennis racket objects +tent tent objects +terrace terrace misc +terrace door door door +therapeutic chair chair chair +thermal mug mug objects +thermometer thermometer objects +thermostat thermostat objects +threadmill treadmill gym_equipment +three three misc +throw blanket throw blanket misc +tile tile objects +tiled floor floor floor +tiled wall wall wall +tiles tiles misc +tiling tiles misc +tin box objects +tire tire objects +tisse box tissue box objects +tissue tissue misc +tissue box tissue box objects +tissues tissue misc +title title misc +toaster toaster appliances +toaster oven toaster oven appliances +toester toaster appliances +toiled toilet toilet +toiled paper toilet paper objects +toilet toilet toilet +toilet bin trash can objects +toilet bowl brush holder toilet brush holder misc +toilet brush toilet brush misc +toilet brush holder toilet brush holder misc +toilet cabinet cabinet cabinet +toilet cleaner toilet cleaner misc +toilet counter counter counter +toilet flush handle objects +toilet handle handle objects +toilet holder toilet brush holder misc +toilet paper toilet paper objects +toilet paper dispenser toilet paper dispenser misc +toilet paper holder toilet paper dispenser misc +toilet paper stand toilet paper dispenser misc +toilet plunger plunger objects +toilet seat toilet seat toilet +toilet sink sink sink +toilet sliding door door door +toilet utensil utensil objects +toilete toilet toilet +toilete brush toilet brush misc +toilete paper holder toilet paper dispenser misc +toiletry toiletry objects +toiletry bag toiletry bag objects +toliet toilet toilet +tolilet toilet toilet +tool tool objects +tool board tool rack wall +tool box tool box objects +tool rack tool rack wall +toolbox tool box objects +tools tool objects +toothbrush toothbrush objects +toothbrush cup cup objects +toothbrush holder cup objects +toothpaste toothpaste objects +torch torch objects +toster toaster appliances +towel towel towel +towel bar towel bar objects +towel bar shelf shelf shelving +towel basket basket objects +towel box box objects +towel hang towel bar objects +towel hanger towel bar objects +towel holder towel bar objects +towel paper dispenser toilet paper dispenser misc +towel paper holder toilet paper dispenser misc +towel rack towel bar objects +towel rail towel bar objects +towel ring towel ring objects +towel rod towel bar objects +towel shelf shelf shelving +towell towel towel +towels towel towel +towels in a basket basket objects +toy toy objects +toy /otherroom toy objects +toy mickey mouse toy objects +toy airplane toy objects +toy car toy objects +toy cars toy objects +toy duck toy objects +toy trailer toy objects +toy train toy objects +toys toy objects +track toy objects +traffic cone traffic cone objects +training bench weight bench gym_equipment +training mat training mat gym_equipment +trampoline trampoline gym_equipment +trascan trash can objects +trash trash can objects +trash bin trash can objects +trash bag trash bag misc +trash bin trash can objects +trash can trash can objects +trashcan trash can objects +tray tray objects +treadmill treadmill gym_equipment +tree tree plant +tree branch tree branch objects +trimmer trimmer objects +trinket trinket objects +tripod tripod shelving +trofeum trophy misc +troley trolley objects +trolley trolley objects +trombone trombone objects +trophies trophy misc +trophy trophy misc +trough trough misc +trumpet trumpet objects +trumpet stand stand table +tub bathtub bathtub +tube tv tv_monitor +tv tv tv_monitor +tv cabinet tv stand furniture +tv decoder tv remote objects +tv led tv tv_monitor +tv remote tv remote objects +tv remote control tv remote objects +tv stand tv stand furniture +tv stand door door door +tv table tv stand furniture +twigs in vase plant plant +typewriter typewriter objects +ubnknown/ probably clothes clothes clothes +uknknown unknown unlabeled +uknown unknown unlabeled +uknown device unknown unlabeled +ukulele ukulele objects +umbrella umbrella objects +umbrella stand umbrella stand furniture +umbrellas umbrella objects +umknown unknown unlabeled +unkknown unknown unlabeled +unknaown unknown unlabeled +unknnown unknown unlabeled +unknon unknown unlabeled +unknow unknown unlabeled +unknowm unknown unlabeled +unknown unknown unlabeled +unknown /otherroom unknown unlabeled +unknown /outside unknown unlabeled +unknown /probably cup cup objects +unknown cluter clutter misc +unknown clutter clutter misc +unknown countertop item unknown unlabeled +unknown device unknown unlabeled +unknown facility unknown unlabeled +unknown kitchen appliance kitchen appliance appliances +unknown kitchen stuff unknown unlabeled +unknown machines appliance misc +unknown office stuff unknown unlabeled +unknown picture/window picture picture +unknown stuff unknown unlabeled +unknown wall wall wall +unknown/ unknown unlabeled +unknown/ a pie? food misc +unknown/ bin? bin objects +unknown/ pile of something clutter misc +unknown/ proably cup cup objects +unknown/ probably a book book objects +unknown/ probably advertisement magazine objects +unknown/ probably air refresher air freshener objects +unknown/ probably air vent vent misc +unknown/ probably air vent fan fan objects +unknown/ probably albums album objects +unknown/ probably attic entrance door door +unknown/ probably bag bag objects +unknown/ probably basket basket objects +unknown/ probably bin bin objects +unknown/ probably board board board_panel +unknown/ probably board games game objects +unknown/ probably boiler boiler misc +unknown/ probably book book objects +unknown/ probably books book objects +unknown/ probably books or albums book objects +unknown/ probably bottle bottle objects +unknown/ probably bowl bowl objects +unknown/ probably box box objects +unknown/probably cabinet cabinet cabinet +unknown/ probably cabinet cabinet cabinet +unknown/ probably calculator calculator appliances +unknown/ probably candle candle objects +unknown/ probably candle stand candle objects +unknown/ probably candles candle objects +unknown/ probably canopy canopy misc +unknown/ probably casket casket misc +unknown/ probably charger charger objects +unknown/ probably clock clock objects +unknown/ probably clothes clothes clothes +unknown/ probably coffee machine coffee machine appliances +unknown/ probably computer computer objects +unknown/ probably coocking books book objects +unknown/ probably cookies food misc +unknown/ probably cosmetic cosmetics objects +unknown/ probably cosmetics cosmetics objects +unknown/ probably cups cups objects +unknown/ probably decoration decoration objects +unknown/ probably decorative plant decorative plant plant +unknown/ probably desk or box box objects +unknown/ probably dinnerware flatware objects +unknown/ probably dishwasher dishwasher appliances +unknown/ probably drum drum objects +unknown/ probably drums drum objects +unknown/ probably fan vent fan objects +unknown/ probably fire alarm fire alarm objects +unknown/ probably fire detector smoke detector misc +unknown/ probably fire sprinkler sprinkler objects +unknown/ probably foam foam misc +unknown/ probably folded table table table +unknown/ probably frame frame misc +unknown/ probably framed picture picture picture +unknown/ probably fridge refrigerator appliances +unknown/ probably glasses glasses objects +unknown/ probably hand grab unknown unlabeled +unknown/ probably hat hat clothes +unknown/ probably heater heater objects +unknown/ probably home theater player tv tv_monitor +unknown/ probably jar jar objects +unknown/ probably lamp lamp lighting +unknown/ probably letters papers objects +unknown/ probably liquid soap soap objects +unknown/ probably magazine magazine objects +unknown/ probably monitor monitor tv_monitor +unknown/ probably napkins napkins objects +unknown/ probably notebook notebook objects +unknown/ probably paper towel dispenser paper towels towel +unknown/ probably paper towel holder paper towels towel +unknown/ probably pipe pipe misc +unknown/ probably plate plate objects +unknown/ probably pot pot objects +unknown/ probably printer printer objects +unknown/ probably radio radio misc +unknown/ probably rug rug floor +unknown/ probably scarf scarf clothes +unknown/ probably security detector security detector misc +unknown/ probably shampoo shampoo objects +unknown/ probably sheet sheet misc +unknown/ probably sheets sheet misc +unknown/ probably shelf shelf shelving +unknown/ probably shower cabine shower shower +unknown/ probably shower gel shampoo objects +unknown/ probably sitting pillow pillow cushion +unknown/ probably soap soap objects +unknown/ probably sofa sofa sofa +unknown/ probably solarium chair chair chair +unknown/ probably stand stand table +unknown/ probably table lamp table lamp lighting +unknown/ probably tank tank misc +unknown/ probably telephone telephone objects +unknown/ probably tissue dispenser tissue misc +unknown/ probably wall lamp wall lamp lighting +unknown/ probably washing powder washing powder misc +unknown/ probably washing soap soap objects +unknown/ probaby albums albums objects +unknown// probably fire sprinkler fire sprinkler misc +unknown/probably decoration decoration objects +unknown/remove unknown/remove misc +unknwn unknown unlabeled +unknwom unknown unlabeled +unknwon unknown unlabeled +unkown unknown unlabeled +unkown clutter clutter misc +unknown" unknown unlabeled + unknown unlabeled +ups unknown unlabeled +urinal urinal objects +utensil utensil objects +vaccum cleaner vacuum cleaner appliances +vacuum cleaner vacuum cleaner appliances +valve plumbing objects +vanity vanity table +vanity table vanity table +vase vase objects +vase with flower flower plant +vegetables vegetables objects +vegetation plant plant +vent vent misc +vent pipe pipe misc +ventialtion ventilation objects +ventilation ventilation objects +ventilation hood ventilation hood misc +ventilation pipe pipe misc +ventilator ventilation objects +vese vessel objects +vessel vessel objects +vessel sink vessel sink misc +vice unknown unlabeled +video cassette cassette objects +vinyl records vinyl records objects +violin violin objects +violin case violin case objects +wadrobe wardrobe furniture +waffle iron waffle iron appliances +wal wall wall +wal hanging decoration decoration objects +wall wall wall +wall outside wall outside wall +wall clock wall clock objects +wall lamp wall lamp lighting +wall /otheroom wall wall +wall /outside wall /outside wall +wall balk wall wall +wall balks wall wall +wall beam wall beam misc +wall board wall board misc +wall cabinet wall cabinet cabinet +wall clock wall clock objects +wall control wall control misc +wall controll wall control misc +wall controller wall control misc +wall coping wall wall +wall corridor wall wall +wall cubby wall cubby misc +wall decoration picture picture +wall detail wall detail misc +wall device wall electronics misc +wall electronic wall electronics misc +wall electronics wall electronics misc +wall frame picture picture +wall hanger wall hanger objects +wall hanging decoration wall hanging decoration misc +wall hanging organizer wall hanger objects +wall indent wall indent misc +wall lamp wall lamp lighting +wall light wall lamp lighting +wall lmap wall lamp lighting +wall niche wall hanging decoration misc +wall of green wall wall +wall painting painting objects +wall panel wall panel wall +wall panel frame wall panel frame wall +wall post wall post misc +wall shelf wall shelf shelving +wall sign wall sign misc +wall sing wall sign misc +wall soap shelf wall soap shelf misc +wall statue wall statue misc +wall sticker decoration objects +wall toilet paper wall toilet paper misc +wall top wall top misc +wall tv wall tv misc +wall vent vent misc +wallkitchen island wall wall +walll wall wall +wardeobe wardrobe furniture +wardeobe door wardrobe furniture +warderobe wardrobe furniture +wardobe wardrobe furniture +wardrobe wardrobe furniture +wardrobe accesories wardrobe furniture +wardrobe door wardrobe furniture +wardrobe door frame wardrobe furniture +wardrobe drawer drawer chest_of_drawers +wardrobe mirror mirror mirror +wardrobe shelf shelf shelving +wardrobe sliding door sliding door door +wash basin washbasin sink +wash basin cabinet wash cabinet cabinet +wash cabinet wash cabinet cabinet +washbasin washbasin sink +washbasin cabinet wash cabinet cabinet +washbasin counter washbasin counter counter +washbasin countertop washbasin counter counter +washbasin table washbasin counter counter +washcloth washcloth objects +washdisher machine dishwasher appliances +washer dryer washer-dryer misc +washer-dryer washer-dryer misc +washing container container objects +washing liquid detergent misc +washing machine washing machine appliances +washing machine and dryer washing machine and dryer appliances +washing powder washing powder misc +washing stuff washing stuff misc +watch watch objects +water basin water basin misc +water bottle bottle objects +water dispenser water dispenser objects +water fountain water fountain misc +water heater water heater objects +water meter water meter objects +water outlet water outlet objects +water pump water pump objects +water tank water tank objects +watering can watering can objects +weight weight gym_equipment +weight bench weight bench gym_equipment +weights weights gym_equipment +wheel wheel objects +wheelbarrow wheelbarrow objects +whine shelf whine shelf misc +whiteboard whiteboard tv_monitor +widnow frame window frame window +wifi repeater wifi router objects +wifi router wifi router objects +wifi transmitter wifi router objects +wind chime decoration objects +windo window window +windoe frame window frame window +windoow shade window shade blinds +windor window window +window window window +window curtain window curtain curtain +window frame window frame window +window glass window glass objects +window shade window shade blinds +window /otherroom window /otherroom window +window /outside window /outside window +window /wall window window +window bars window window +window behind shutters window window +window blinds blinds blinds +window curtain window curtain curtain +window curtain bar window curtain curtain +window fame window frame window +window fram window frame window +window frame window frame window +window frame /otherroom window frame /otherroom window +window glass window glass objects +window glass /outside window window +window pane window window +window panel window window +window panes window window +window place window window +window rame window window +window seat window seat seating +window shade window shade blinds +window shades window shade blinds +window shutter window shutter objects +window shutters window shutters objects +window sill window frame window +window valence window valence curtain +window/door window/door misc +windowframe window frame window +windown frame window frame window +windows window window +windows frame window frame window +windowsil window frame window +windowsill window frame window +windw frame window frame window +wine wine misc +wine bottle wine bottle objects +wine cabinet wine cabinet cabinet +wine rack wine rack misc +wine refrigerator refrigerator appliances +wine storage wine storage shelving +winndow window window +winndow frame window frame window +wire wire misc +wood wood misc +wood burner fireplace fireplace +wood frame frame misc +wooden balk wood misc +wooden panel panel board_panel +wool cloth objects +woor frame frame misc +workout bike workout bike misc +workout weight weights gym_equipment +workstation workstation objects +worktop worktop misc +wreath wreath objects +wrench wrench misc +yoga mat yoga mat misc +~aper unknown unlabeled +łamp lamp lighting +śign sign misc diff --git a/pointcept/datasets/preprocessing/hm3d/preprocess_hm3d.py b/pointcept/datasets/preprocessing/hm3d/preprocess_hm3d.py new file mode 100644 index 0000000000000000000000000000000000000000..ed5e6412fc544d0eceadb788c302f314689f71ce --- /dev/null +++ b/pointcept/datasets/preprocessing/hm3d/preprocess_hm3d.py @@ -0,0 +1,209 @@ +""" +Preprocessing Script for Habitat-Matterport 3D Dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import glob +import argparse +import numpy as np +import trimesh +from pathlib import Path +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat +from pathlib import Path + +from hm3d_constants import CLASS_LABELS_40 + +MPCAT2INDEX = dict([(CLASS_LABELS_40[i], i) for i in range(40)]) +MPCAT2INDEX["unlabeled"] = -1 + +CAT2INDEX = dict() +with open(Path(__file__).parent / "hm3dsem_category_mappings.tsv") as f: + f.readline() # raw_category category mpcat40 + lines = f.readlines() +for line in lines: + line = line.strip().split("\t") + if len(line) == 2: + # L2164: "\tunknown\tunlabeled" + line.insert(0, "") + CAT2INDEX[line[1]] = MPCAT2INDEX[line[2]] + + +def uv_to_texture_color(uv, texture): + width, height = texture.size + u = int(uv[0] * width) + v = int((1 - uv[1]) * height) # Flip y-axis for image coordinates + if 0 <= u < width and 0 <= v < height: + return texture.getpixel((u, v)) + else: + return 0, 0, 0 + + +def load_hex_mapping(mapping_path): + hex2label = {} + with open(mapping_path) as f: + f.readline() # remove 'HM3D Semantic Annotations\n' + lines = f.readlines() # get the left + for line in lines: + line = line.strip().split(",") + cat = line[2].strip('"') + if cat == "trashcan": + cat = "trash can" + elif cat == "fridge": + cat = "refrigerator" + hex2label[line[1]] = dict(instance=line[0], segment=CAT2INDEX[cat]) + return hex2label + + +def handle_process( + scene_path, + output_root, + density=0.02, +): + scene_path = Path(scene_path) + scene_label_path = scene_path.with_suffix(".semantic.glb") + scene_mapping_path = scene_path.with_suffix(".semantic.txt") + scene_name = scene_path.parent.name.replace("-", "_") + scene_id = scene_name.split("_")[0] + labeled = True if scene_label_path.is_file() else False + + # test split (900-1000) is reserved by official + if 0 <= int(scene_id) < 800: + split = "train" + else: + split = "val" + print(f"Parsing scene {scene_name} in {split} split...") + + scene = trimesh.load(scene_path) + if labeled: + labeled_scene_ = trimesh.load(scene_label_path) + labeled_scene = trimesh.Scene() + for name, mesh in labeled_scene_.geometry.items(): + # some case, name in labeled scene and scene is not matched, so only use chunk id + labeled_scene.add_geometry(mesh, geom_name=name.split("_")[0]) + del labeled_scene_ + hex2label = load_hex_mapping(scene_mapping_path) + + room_dict = {} + for name, mesh in scene.geometry.items(): + room_id = "_".join(name.split("_")[1:3]).replace("group", "").replace("sub", "") + if room_id not in room_dict.keys(): + room_dict[room_id] = trimesh.Scene() + room_dict[room_id].add_geometry(mesh, geom_name=name.split("_")[0]) + del scene + + for room_id, scene in room_dict.items(): + # seed by scene_id and room_id e.g. 00802-000-002 -> 802000002 + np.random.seed(int(scene_id + room_id.replace("_", ""))) + room_coord = [] + room_color = [] + room_normal = [] + if labeled: + room_label_color = [] + + for name in scene.geometry.keys(): + mesh = scene.geometry[name] + num_points = int(np.sum(mesh.area_faces) / density**2) + if num_points == 0: + continue + coords, face_indices = mesh.sample(num_points, return_index=True) + faces = mesh.faces[face_indices] + triangles = mesh.vertices[faces] + bary_coords = trimesh.triangles.points_to_barycentric(triangles, coords) + uv_coords = mesh.visual.uv[faces] + sampled_uvs = np.einsum("ijk,ij->ik", uv_coords, bary_coords) + + pbr_material = mesh.visual.material + texture_image = pbr_material.baseColorTexture + if texture_image is None: + continue + colors = np.array( + [uv_to_texture_color(uv, texture_image) for uv in sampled_uvs] + ) + normals = mesh.vertex_normals[faces] + normals = np.einsum("ijk,ij->ik", normals, bary_coords) + room_coord.append(coords) + room_color.append(colors) + room_normal.append(normals) + + if labeled: + labeled_mash = labeled_scene.geometry[name] + label_texture_image = labeled_mash.visual.material.baseColorTexture + label_color = np.array( + [uv_to_texture_color(uv, label_texture_image) for uv in sampled_uvs] + ) + room_label_color.append(label_color) + + if len(room_coord) == 0: + continue + + room_coord = np.concatenate(room_coord, axis=0).astype("float32") + room_color = np.concatenate(room_color, axis=0).astype("uint8") + room_normal = np.concatenate(room_normal, axis=0).astype("float32") + data_dict = dict(coord=room_coord, color=room_color, normal=room_normal) + + if labeled: + room_label_color = np.concatenate(room_label_color, axis=0) + instance_label_color = np.unique(room_label_color, axis=0) + room_instance = -np.ones(len(room_label_color), dtype="int16") + room_segment = -np.ones(len(room_label_color), dtype="int16") + for i in range(len(instance_label_color)): + label_color = instance_label_color[i] + label_hex = "{c[0]:02x}{c[1]:02x}{c[2]:02x}".format(c=label_color) + mask = np.all(room_label_color == label_color, axis=-1) + room_instance[mask] = i + if label_hex.upper() in hex2label.keys(): + room_segment[mask] = hex2label[label_hex.upper()]["segment"] + data_dict["instance"] = room_instance + data_dict["segment"] = room_segment + save_path = Path(output_root) / split / "_".join([scene_name, room_id]) + os.makedirs(save_path, exist_ok=True) + for key, value in data_dict.items(): + np.save(save_path / f"{key}.npy", value) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the Habitat-Matterport 3D dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--density", + default=0.02, + type=float, + help="Sampling density on mesh surface (m)", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + args = parser.parse_args() + + scene_list = glob.glob(os.path.join(args.dataset_root, "*", "*.glb")) + scene_list = [scene for scene in scene_list if not scene.endswith("semantic.glb")] + assert len(scene_list) == 900 + + # Preprocess data. + print("Processing scenes...") + pool = ProcessPoolExecutor(max_workers=args.num_workers) + _ = list( + pool.map( + handle_process, + scene_list, + repeat(args.output_root), + repeat(args.density), + ) + ) diff --git a/pointcept/datasets/preprocessing/matterport3d/meta_data/category_mapping.tsv b/pointcept/datasets/preprocessing/matterport3d/meta_data/category_mapping.tsv new file mode 100644 index 0000000000000000000000000000000000000000..177c75ae021287bfc2462ae487b042886313cab1 --- /dev/null +++ b/pointcept/datasets/preprocessing/matterport3d/meta_data/category_mapping.tsv @@ -0,0 +1,1660 @@ +index raw_category category count nyuId nyu40id eigen13id nyuClass nyu40class eigen13class ModelNet40 ModelNet10 ShapeNetCore55 synsetoffset wnsynsetid wnsynsetkey mpcat40index mpcat40 +1 wall wall 7667 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +2 door door 2544 28 8 12 door door Wall door n03221720 door.n.01 4 door +3 ceiling ceiling 2363 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +4 floor floor 2252 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +5 picture picture 2125 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +6 window window 2013 59 9 13 window window Window n04587648 window.n.01 9 window +7 chair chair 1947 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +8 doorframe door frame 1935 28 8 12 door door Wall door n03221720 door.n.01 4 door +9 remove remove 1622 0 0 0 void void void 0 void +10 pillow pillow 1335 119 18 7 pillow pillow Objects pillow 03938244 n03938244 pillow.n.01 8 cushion +11 object object 1145 40 7 otherprop Objects n00002684 object.n.01 39 objects +12 light light 967 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +13 cabinet cabinet 785 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +14 curtain curtain 692 89 16 13 curtain curtain Window curtain n03151077 curtain.n.01 12 curtain +15 table table 691 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +16 plant plant 632 82 40 7 plant otherprop Objects plant n00017222 plant.n.02 14 plant +17 decoration decoration 532 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +18 window frame window frame 526 59 9 13 window window Window n04589593 window_frame.n.01 9 window +19 lamp lamp 540 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +20 mirror mirror 488 122 19 7 mirror mirror Objects n03773035 mirror.n.01 21 mirror +21 towel towel 437 135 27 7 towel towel Objects n04459362 towel.n.01 20 towel +22 sink sink 417 24 34 7 sink sink Objects sink n04223580 sink.n.01 15 sink +23 shelf shelf 417 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +24 couch couch 290 83 6 9 sofa sofa Sofa sofa sofa sofa 04256520 n04256520 sofa.n.01 10 sofa +25 dining chair dining chair 277 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +26 bed bed 270 157 4 1 bed bed Bed bed bed bed 02818832 n02818832 bed.n.01 11 bed +27 nightstand nightstand 275 158 32 6 night stand night stand Furniture night_stand night_stand n03015254 chest_of_drawers.n.01 13 chest_of_drawers +28 toilet toilet 268 124 33 7 toilet toilet Objects toilet toilet n04446276 toilet.n.01 18 toilet +29 sofa chair sofa chair 262 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +30 column pillar 243 94 38 7 column otherstructure Objects n03073977 column.n.07 24 column +31 handrail handrail 241 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +32 stair stair 241 215 38 7 stairs otherstructure Objects stairs n04314914 step.n.04 16 stairs +33 stool stool 232 150 40 7 stool otherprop Objects stool n04326896 stool.n.01 19 stool +34 armchair armchair 224 5 5 4 chair chair Chair chair chair chair 03001627 n02738535 armchair.n.01 3 chair +35 kitchen cabinet kitchen cabinet 212 3 3 6 cabinet cabinet Furniture n02933112 cabinet.n.01 7 cabinet +36 vase vase 205 78 40 7 vase otherprop Objects vase jar 03593526 n04522168 vase.n.01 39 objects +37 cushion cushion 204 119 18 7 pillow pillow Objects n03151500 cushion.n.03 8 cushion +38 tv tv 204 172 25 11 television television TV tv or monitor 03211117 n03211117 display.n.06 22 tv_monitor +39 door frame door frame 190 28 8 12 door door Wall door n03221720 door.n.01 4 door +40 unknown unknown 186 20 40 7 unknown otherprop Objects n08632096 unknown.n.01 41 unlabeled +41 pot pot 185 16 40 7 pot otherprop Objects n03991062 pot.n.04 39 objects +42 desk desk 187 36 14 10 desk desk Table desk desk table 04379243 n03179701 desk.n.01 5 table +43 painting picture 194 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +44 roof roof 173 4 22 3 ceiling ceiling Ceiling n04105068 roof.n.01 17 ceiling +45 box box 172 26 29 7 box box Objects n02883344 box.n.01 39 objects +46 shower wall shower wall 168 21 1 12 wall wall Wall n04208936 shower.n.01 23 shower +47 coffee table coffee table 159 356 39 6 coffee table otherfurniture Furniture table table table 04379243 n03063968 coffee_table.n.01 5 table +48 countertop countertop 159 7 12 6 counter counter Furniture n03118245 countertop.n.01 26 counter +49 bench bench 154 204 39 6 bench otherfurniture Furniture bench bench 02828884 n02828884 bench.n.01 34 seating +50 wall frame picture 142 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +51 trash can trashcan 131 12 39 6 garbage bin otherfurniture Furniture trash_bin 02747177 n02747177 ashcan.n.01 39 objects +52 fireplace fireplace 128 372 38 7 fireplace otherstructure Objects n03346455 fireplace.n.01 27 fireplace +53 clothes clothes 125 141 21 7 clothes clothes Objects n02728440 apparel.n.01 38 clothes +54 pillar pillar 117 94 38 7 column otherstructure Objects n03073977 column.n.07 24 column +55 bathtub bathtub 121 136 36 7 bathtub bathtub Objects bathtub bathtub tub 02808440 n02808440 bathtub.n.01 25 bathtub +56 ceiling duct ceiling duct 110 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +57 bath cabinet bath cabinet 111 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +58 book book 104 1 23 2 book books Books n02870526 book.n.11 39 objects +59 beam beam 101 38 7 otherstructure Objects n02815950 beam.n.02 29 beam +60 vent vent 101 25 38 7 air vent otherstructure Objects n04526241 vent.n.01 40 misc +61 shower floor shower floor 97 11 2 5 floor floor Floor n04208936 shower.n.01 23 shower +62 faucet faucet 97 9 40 7 faucet otherprop Objects faucet 03325088 n03325088 faucet.n.01 39 objects +63 photo photo 100 508 40 7 photo otherprop Objects n03925226 photograph.n.01 6 picture +64 delete remove 90 0 0 0 void void void 0 void +65 toilet paper toilet paper 90 139 40 7 toilet paper otherprop Objects n15075141 toilet_tissue.n.01 39 objects +66 counter counter 91 7 12 6 counter counter Furniture table table table 04379243 n03116530 counter.n.01 26 counter +67 fan fan 85 74 40 7 fan otherprop Objects n03320046 fan.n.01 39 objects +68 step step 86 38 7 otherstructure Objects n04314914 step.n.04 16 stairs +69 table lamp table lamp 96 144 35 7 lamp lamp Objects lamp lamp 03636649 n04380533 table_lamp.n.01 28 lighting +70 wall / other room wall /otherroom 84 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +71 washbasin washbasin 91 24 34 7 sink sink Objects sink n04553920 washbasin.n.01 15 sink +72 rail railing 83 497 38 7 railing otherstructure Objects n04047401 railing.n.01 30 railing +73 side table table 85 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +74 decor decoration 79 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +75 shelves shelving 79 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +76 statue statue 78 294 40 7 sculpture otherprop Objects n04306847 statue.n.01 39 objects +77 dresser dresser 79 169 17 6 dresser dresser Furniture dresser dresser n03015254 chest_of_drawers.n.01 13 chest_of_drawers +78 stairs stair 76 215 38 7 stairs otherstructure Objects stairs n04314914 step.n.04 16 stairs +79 rug rug 77 130 40 7 rug floor mat Objects n04118021 rug.n.01 2 floor +80 ottoman ottoman 79 359 39 6 ottoman otherfurniture Furniture stool n03380724 footstool.n.01 19 stool +81 round table table 74 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +82 bottle bottle 101 2 40 7 bottle otherprop Objects bottle bottle 02876657 n02876657 bottle.n.01 39 objects +83 kitchen counter kitchen counter 73 7 12 6 counter counter Furniture table table table 04379243 n03116530 counter.n.01 26 counter +84 windowframe window frame 66 59 9 13 window window Window n04589593 window_frame.n.01 9 window +85 office chair office chair 70 5 5 4 chair chair Chair chair chair chair 03001627 n04373704 swivel_chair.n.01 3 chair +86 frame frame 66 38 7 otherstructure Objects 40 misc +87 refrigerator refrigerator 65 17 24 6 refridgerator refridgerator Furniture n04070727 refrigerator.n.01 37 appliances +88 bookshelf bookshelf 63 88 10 6 bookshelf bookshelf Furniture bookshelf bookshelf 02871439 n02871439 bookshelf.n.01 31 shelving +89 end table end table 67 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +90 wardrobe wardrobe 64 772 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 36 furniture +91 toiletry toiletry 63 40 7 otherprop Objects n04447443 toiletry.n.01 39 objects +92 windowsill window frame 62 59 9 13 window window Window n04589593 window_frame.n.01 9 window +93 pipe pipe 62 41 40 7 pipe otherprop Objects n03944672 pipe.n.02 40 misc +94 monitor monitor 60 49 40 7 monitor otherprop Objects monitor monitor tv or monitor 03211117 n03782190 monitor.n.04 22 tv_monitor +95 stand stand 62 50 39 6 stand otherfurniture Furniture table table table 04379243 n04301000 stand.n.04 5 table +96 drawers drawer 60 174 39 6 drawer otherfurniture Furniture n03233905 drawer.n.01 13 chest_of_drawers +97 dining table dining table 61 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +98 container container 59 140 40 7 container otherprop Objects n03094503 container.n.01 39 objects +99 light switch light switch 59 301 38 7 light switch otherstructure Objects n04372370 switch.n.01 39 objects +100 skylight skylight 57 59 9 13 window window Window n04232800 skylight.n.01 9 window +101 purse purse 57 181 40 7 purse otherprop Objects n02774152 bag.n.04 39 objects +102 wall /otherroom wall /otherroom 55 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +103 sofa couch 56 83 6 9 sofa sofa Sofa sofa sofa sofa 04256520 n04256520 sofa.n.01 10 sofa +104 books book 53 1 23 2 book books Books n02870526 book.n.11 39 objects +105 doorway doorway 54 609 38 7 door way otherstructure Objects door n03224032 doorway.n.01 4 door +106 railing railing 52 497 38 7 railing otherstructure Objects n04047401 railing.n.01 30 railing +107 wainscotting paneling 52 21 1 12 wall wall Wall n03882611 paneling.n.01 1 wall +108 basket basket 52 39 40 7 basket otherprop Objects basket 02801938 n02801938 basket.n.01 39 objects +109 closet closet 50 40 7 otherprop Objects n03148324 cupboard.n.01 7 cabinet +110 arch arch 48 40 7 otherprop Objects n02733524 arch.n.04 40 misc +111 chandelier chandelier 48 342 38 7 chandelier otherstructure Objects n03005285 chandelier.n.01 28 lighting +112 oven oven 47 238 38 7 oven otherstructure Objects n03862676 oven.n.01 37 appliances +113 clock clock 47 56 40 7 clock otherprop Objects clock 03046257 n03046257 clock.n.01 39 objects +114 footstool footstool 48 359 39 6 ottoman otherfurniture Furniture stool n03380724 footstool.n.01 19 stool +115 stove stove 44 242 38 7 stove otherstructure Objects stove 04330267 n04330267 stove.n.02 37 appliances +116 trashcan trashcan 44 12 39 6 garbage bin otherfurniture Furniture trash_bin 02747177 n02747177 ashcan.n.01 39 objects +117 drawer drawer 45 174 39 6 drawer otherfurniture Furniture n03233905 drawer.n.01 13 chest_of_drawers +118 bathroom countertop object object 43 40 7 otherprop Objects n00002684 object.n.01 39 objects +119 washing machine washing machine 43 278 39 6 washing machine otherfurniture Furniture washing_machine 04554684 n04554684 washer.n.03 37 appliances +120 shower curtain shower curtain 43 123 28 7 shower curtain shower curtain Objects curtain n04209239 shower_curtain.n.01 12 curtain +121 rack rack 41 50 39 6 stand otherfurniture Furniture n04038440 rack.n.05 31 shelving +122 art picture 40 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +123 firealarm fire alarm 37 338 40 7 fire alarm otherprop Objects n03343737 fire_alarm.n.02 39 objects +124 bin bin 37 307 40 7 bin otherprop Objects n02839910 bin.n.01 39 objects +125 chest chest 36 344 39 6 chest otherfurniture Furniture dresser dresser 40 misc +126 microwave microwave 36 13 40 7 microwave otherprop Objects microwave 03761084 n03761084 microwave.n.02 37 appliances +127 blinds blinds 36 80 13 13 blinds blinds Window n04589190 window_blind.n.01 32 blinds +128 bowl bowl 35 22 40 7 bowl otherprop Objects bowl bowl 02880940 n02880940 bowl.n.03 39 objects +129 ceiling pipe ceiling pipe 34 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +130 tree tree 34 82 40 7 plant otherprop Objects plant n13104059 tree.n.01 14 plant +131 vanity vanity 34 169 17 6 dresser dresser Furniture dresser dresser table 04379243 n03238586 dressing_table.n.01 5 table +132 ceiling fan ceiling fan 37 74 40 7 fan otherprop Objects n03320046 fan.n.01 39 objects +133 tissue box tissue box 34 138 40 7 tissue box otherprop Objects n02883344 box.n.01 39 objects +134 desk chair desk chair 34 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +135 plate plate 32 233 40 7 plate otherprop Objects n03959485 plate.n.04 39 objects +136 tv stand tv stand 32 291 39 6 tv stand otherfurniture Furniture tv_stand n03290653 entertainment_center.n.01 36 furniture +137 shoes shoe 32 149 40 7 shoe otherprop Objects n04199027 shoe.n.01 39 objects +138 heater heater 32 111 39 6 heater otherfurniture Furniture n03508101 heater.n.01 39 objects +139 bedframe bedframe 33 157 4 1 bed bed Bed n02822579 bedstead.n.01 11 bed +140 headboard headboard 33 161 39 6 headboard otherfurniture Furniture n03502200 headboard.n.01 11 bed +141 post post 32 94 38 7 column otherstructure Objects n03988170 post.n.04 24 column +142 swivel chair swivel chair 31 5 5 4 chair chair Chair chair chair chair 03001627 n04373704 swivel_chair.n.01 3 chair +143 pedestal pedestal 31 50 39 6 stand otherfurniture Furniture 40 misc +144 fence fence 31 38 7 otherstructure Objects n03327234 fence.n.01 40 misc +145 ceiling pipes ceiling pipe 30 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +146 pew pew 28 204 39 6 bench otherfurniture Furniture bench bench 02828884 n03920867 pew.n.01 34 seating +147 decorative object decoration 28 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +148 bucket bucket 28 427 40 7 bucket otherprop Objects n02909870 bucket.n.01 39 objects +149 mask decorative mask 27 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +150 candle candle 27 137 40 7 candle otherprop Objects lamp n02948072 candle.n.01 39 objects +151 flowerpot flowerpot 28 146 40 7 flower pot otherprop Objects flower_pot flower pot 03991062 n03991062 pot.n.04 39 objects +152 speaker speaker 27 54 40 7 speaker otherprop Objects speaker 03691459 n03691459 loudspeaker.n.01 39 objects +153 seat seat 26 524 39 6 furniture otherfurniture Furniture n04161981 seat.n.03 34 seating +154 sign sign 25 208 40 7 sign otherprop Objects n04217882 signboard.n.01 40 misc +155 air conditioner air conditioner 26 79 38 7 air conditioner otherstructure Objects n02686379 air_conditioner.n.01 39 objects +156 shower curtain rod shower curtain rod 25 40 7 otherprop Objects n04100174 rod.n.01 12 curtain +157 unknown / other room unknown /otherroom 24 20 40 7 unknown otherprop Objects n08632096 unknown.n.01 41 unlabeled +158 flowers plant 25 82 40 7 plant otherprop Objects plant n00017222 plant.n.02 14 plant +159 clutter clutter 24 40 7 otherprop Objects 40 misc +160 pillows pillow 24 119 18 7 pillow pillow Objects pillow 03938244 n03938244 pillow.n.01 8 cushion +161 plants plant 24 82 40 7 plant otherprop Objects plant n00017222 plant.n.02 14 plant +162 wall \other room wall /otherroom 24 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +163 fire extinguisher fire extinguisher 24 10 40 7 fire extinguisher otherprop Objects n03345837 fire_extinguisher.n.01 39 objects +164 towels towel 24 135 27 7 towel towel Objects n04459362 towel.n.01 20 towel +165 curtains curtain 23 89 16 13 curtain curtain Window curtain n03151077 curtain.n.01 12 curtain +166 curtain rod curtain rod 23 582 38 7 curtain rod otherstructure Objects n04100174 rod.n.01 12 curtain +167 kitchen countertop object object 23 40 7 otherprop Objects n00002684 object.n.01 39 objects +168 mat mat 23 143 20 5 floor mat floor mat Floor n03727837 mat.n.01 2 floor +169 flower plant 23 82 40 7 plant otherprop Objects plant n00017222 plant.n.02 14 plant +170 sculpture sculpture 23 294 40 7 sculpture otherprop Objects n04157320 sculpture.n.01 39 objects +171 shelving shelving 22 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +172 wall/other room wall /otherroom 22 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +173 knickknack object 21 40 7 otherprop Objects n00002684 object.n.01 39 objects +174 printer printer 21 66 40 7 printer otherprop Objects printer 04004475 n04004475 printer.n.03 39 objects +175 wall behind wall /otherroom 21 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +176 telephone telephone 21 32 40 7 telephone otherprop Objects telephone 04401088 n04401088 telephone.n.01 39 objects +177 bedside table nightstand 21 158 32 6 night stand night stand Furniture night_stand night_stand n03015254 chest_of_drawers.n.01 13 chest_of_drawers +178 moulding molding 21 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +179 handbag handbag 21 40 7 otherprop Objects n02774152 bag.n.04 39 objects +180 wall /other room wall /otherroom 21 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +181 blanket blanket 21 312 40 7 blanket otherprop Objects n02849154 blanket.n.01 39 objects +182 shower shower 21 38 7 otherstructure Objects n04208936 shower.n.01 23 shower +183 steps step 20 38 7 otherstructure Objects n04314914 step.n.04 16 stairs +184 switch switch 21 40 7 otherprop Objects n04372370 switch.n.01 39 objects +185 toilet paper dispenser toilet paper dispenser 20 40 7 otherprop Objects 40 misc +186 objects object 21 40 7 otherprop Objects n00002684 object.n.01 39 objects +187 handle handle 20 758 40 7 handle otherprop Objects n03485997 handle.n.01 39 objects +188 frame /outside frame /outside 20 38 7 otherstructure Objects 40 misc +189 screen screen 20 89 16 13 curtain curtain Window curtain n03151077 curtain.n.01 12 curtain +190 shower head showerhead 19 650 40 7 shower head otherprop Objects n04209383 showerhead.n.01 23 shower +191 baracade barricade 19 40 7 otherprop Objects n04096848 roadblock.n.02 40 misc +192 picture frame picture frame 25 64 11 8 picture picture Picture n03931765 picture_frame.n.01 6 picture +193 soap soap 19 133 40 7 soap otherprop Objects n04253437 soap.n.01 39 objects +194 staircase railing banister 18 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +195 keyboard keyboard 18 47 40 7 keyboard otherprop Objects keyboard computer keyboard 03085013 n03085013 computer_keyboard.n.01 39 objects +196 thermostat thermostat 18 110 40 7 thermostat otherprop Objects n04422875 thermostat.n.01 39 objects +197 radiator radiator 18 236 39 6 radiator otherfurniture Furniture n04041069 radiator.n.02 39 objects +198 kitchen island kitchen island 18 456 38 7 kitchen island otherstructure Objects n03620600 kitchen_island.n.01 26 counter +199 paper towel paper towel 18 113 40 7 paper towel otherprop Objects n03887697 paper_towel.n.01 20 towel +200 wall decoration picture 17 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +201 phone telephone 17 32 40 7 telephone otherprop Objects telephone 04401088 n04401088 telephone.n.01 39 objects +202 mirror frame mirror 17 122 19 7 mirror mirror Objects n03773035 mirror.n.01 21 mirror +203 clothes dryer clothes dryer 18 39 6 otherfurniture Furniture n03251766 dryer.n.01 37 appliances +204 panel panel 17 559 40 7 sheet otherprop Objects n03882058 panel.n.01 35 board_panel +205 glass glass 16 612 38 7 glass otherstructure Objects n03438257 glass.n.02 39 objects +206 soap dispenser soap dispenser 16 40 7 otherprop Objects n04254120 soap_dispenser.n.01 39 objects +207 dishwasher dishwasher 16 8 38 7 dishwasher otherstructure Objects dishwasher 03207941 n03207941 dishwasher.n.01 37 appliances +208 cup cup 16 35 40 7 cup otherprop Objects cup cup or mug 03797390 n03797390 mug.n.04 39 objects +209 bathroom cabinet bathroom cabinet 17 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +210 ladder ladder 17 48 39 6 ladder otherfurniture Furniture stairs n03632277 ladder.n.01 16 stairs +211 garage door garage door 16 850 38 7 garage door otherstructure Objects door 4 door +212 hat hat 15 193 40 7 hat otherprop Objects n03497657 hat.n.01 38 clothes +213 chest of drawers chest of drawers 15 524 39 6 furniture otherfurniture Furniture dresser dresser n03015254 chest_of_drawers.n.01 13 chest_of_drawers +214 exit sign exit sign 15 86 40 7 exit sign otherprop Objects 40 misc +215 sidetable side table 15 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +216 office table office table 15 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +217 piano piano 15 298 39 6 piano otherfurniture Furniture piano piano 03928116 n03928116 piano.n.01 39 objects +218 painter picture 17 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +219 board board 15 408 38 7 board otherstructure Objects 35 board_panel +220 windows window 14 59 9 13 window window Window n04587648 window.n.01 9 window +221 archway archway 14 21 1 12 wall wall Wall n02734217 arch.n.03 4 door +222 rope rope 14 560 40 7 rope otherprop Objects n04108268 rope.n.01 39 objects +223 ball ball 15 60 40 7 ball otherprop Objects 40 misc +224 gym equipment gym equipment 14 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +225 clothes hangers clothes hanger 13 211 40 7 hanger otherprop Objects n03057920 coat_hanger.n.01 39 objects +226 bathroom object object 13 40 7 otherprop Objects n00002684 object.n.01 39 objects +227 easy chair easy chair 13 5 5 4 chair chair Chair chair chair chair 03001627 n03262932 easy_chair.n.01 3 chair +228 lounge chair lounge chair 15 5 5 4 chair chair Chair chair chair chair 03001627 n03262932 easy_chair.n.01 3 chair +229 furniture furniture 13 524 39 6 furniture otherfurniture Furniture n03405725 furniture.n.01 36 furniture +230 cabinets cabinet 16 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +231 carpet carpet 14 130 40 7 rug floor mat Objects n04118021 rug.n.01 2 floor +232 food food 13 40 7 otherprop Objects n00021265 food.n.01 40 misc +233 plant pot pot 13 16 40 7 pot otherprop Objects n03991062 pot.n.04 39 objects +234 duct duct 13 40 7 otherprop Objects n03253398 duct.n.03 40 misc +235 ridge ridge 13 40 7 otherprop Objects 40 misc +236 candlestick candlestick 12 148 40 7 candlestick otherprop Objects n02948557 candlestick.n.01 39 objects +237 computer desk computer desk 12 36 14 10 desk desk Table desk desk table 04379243 n03179701 desk.n.01 5 table +238 shower door shower door 12 28 8 12 door door Wall door n04208936 shower.n.01 23 shower +239 trash trashcan 12 12 39 6 garbage bin otherfurniture Furniture trash_bin 02747177 n02747177 ashcan.n.01 39 objects +240 crown molding molding 12 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +241 wall sconce sconce 12 62 38 7 light otherstructure Objects n04148703 sconce.n.03 28 lighting +242 door handle door handle 12 758 40 7 handle otherprop Objects 40 misc +243 scale scale 12 639 40 7 scale otherprop Objects n04141975 scale.n.07 39 objects +244 trash bin trashcan 12 12 39 6 garbage bin otherfurniture Furniture trash_bin 02747177 n02747177 ashcan.n.01 39 objects +245 baseboard baseboard 13 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +246 window /otherroom window /otherroom 12 59 9 13 window window Window n04587648 window.n.01 9 window +247 bag bag 11 55 37 7 bag bag Objects suitcase 02773838 n02773838 bag.n.06 39 objects +248 laptop laptop 11 37 40 7 laptop otherprop Objects laptop laptop 03642806 n03642806 laptop.n.01 39 objects +249 treadmill treadmill 12 458 39 6 treadmill otherfurniture Furniture n04477387 treadmill.n.01 33 gym_equipment +250 staircase staircase 11 215 38 7 stairs otherstructure Objects n04298308 stairway.n.01 16 stairs +251 guitar guitar 11 300 40 7 guitar otherprop Objects guitar guitar 03467517 n03467517 guitar.n.01 39 objects +252 light fixture light fixture 11 62 38 7 light otherstructure Objects n03665366 light.n.02 28 lighting +253 pipes pipe 11 41 40 7 pipe otherprop Objects n03944672 pipe.n.02 40 misc +254 display case display case 11 540 39 6 display case otherfurniture Furniture n02975212 case.n.20 39 objects +255 weight machine exercise equipment 10 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +256 toilet paper holder toilet paper holder 10 647 40 7 toilet paper holder otherprop Objects 40 misc +257 basin basin 10 24 34 7 sink sink Objects sink n04223580 sink.n.01 15 sink +258 towel bar towel bar 10 51 38 7 bar otherstructure Objects n04459909 towel_rail.n.01 39 objects +259 floor behind floor /otherroom 10 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +260 wooden chair chair 10 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +261 potted plant plant 10 82 40 7 plant otherprop Objects plant n00017222 plant.n.02 14 plant +262 ceiling / other room ceiling /otherroom 10 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +263 window sill window frame 10 59 9 13 window window Window n04589593 window_frame.n.01 9 window +264 floor / other room floor /otherroom 10 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +265 cupboard cabinet 10 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +266 tray tray 10 179 40 7 tray otherprop Objects n04476259 tray.n.01 39 objects +267 urn urn 10 151 40 7 urn otherprop Objects vase jar 03593526 n04516116 urn.n.01 39 objects +268 church seating pew 10 204 39 6 bench otherfurniture Furniture bench bench 02828884 n03920867 pew.n.01 34 seating +269 decorative plate decorative plate 9 383 40 7 decorative plate otherprop Objects 40 misc +270 doors door 9 28 8 12 door door Wall door n03221720 door.n.01 4 door +271 bar bar 9 51 38 7 bar otherstructure Objects n02788689 bar.n.03 39 objects +272 stair rail banister 9 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +273 window shade window shade 9 40 7 otherprop Objects n04590129 window_shade.n.01 32 blinds +274 grass grass 11 82 40 7 plant otherprop Objects plant n12102133 grass.n.01 14 plant +275 pool table pool table 9 515 39 6 pool table otherfurniture Furniture table table table 04379243 n03982430 pool_table.n.01 5 table +276 coat coat 9 324 40 7 jacket otherprop Objects n03057021 coat.n.01 38 clothes +277 trees tree 9 82 40 7 plant otherprop Objects plant n13104059 tree.n.01 14 plant +278 cloth cloth 11 40 7 otherprop Objects n03309808 fabric.n.01 39 objects +279 bottle of soap bottle of soap 9 502 40 7 bottle of soap otherprop Objects 40 misc +280 floor lamp floor lamp 9 144 35 7 lamp lamp Objects lamp lamp 03636649 n03367059 floor_lamp.n.01 28 lighting +281 water cooler water cooler 9 509 39 6 water cooler otherfurniture Furniture n04559166 water_cooler.n.01 39 objects +282 pews pew 9 204 39 6 bench otherfurniture Furniture bench bench 02828884 n03920867 pew.n.01 34 seating +283 ledge ledge 10 38 7 otherstructure Objects n09337253 ledge.n.01 39 objects +284 kitchen shelf kitchen shelf 9 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +285 bathroom utencils bathroom utensil 8 267 40 7 utensil otherprop Objects n04516672 utensil.n.01 39 objects +286 hanger hanger 8 211 40 7 hanger otherprop Objects n03490884 hanger.n.02 39 objects +287 shrubbery shrubbery 8 40 7 otherprop Objects n08649067 shrubbery.n.01 39 objects +288 teapot teapot 8 678 40 7 tea pot otherprop Objects n04398044 teapot.n.01 39 objects +289 bottles bottle 8 2 40 7 bottle otherprop Objects bottle bottle 02876657 n02876657 bottle.n.01 39 objects +290 exercise equipment exercise equipment 8 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +291 boxes box 8 26 29 7 box box Objects n02883344 box.n.01 39 objects +292 locker locker 8 3 3 6 cabinet cabinet Furniture n02933462 cabinet.n.03 40 misc +293 wall cabinet wall cabinet 8 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +294 wainscotting \other room paneling /otherroom 8 21 1 12 wall wall Wall n03882611 paneling.n.01 1 wall +295 ceiling light ceiling lamp 9 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +296 ornament ornament 8 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +297 bidet bidet 8 124 33 7 toilet toilet Objects toilet toilet n02836174 bidet.n.01 18 toilet +298 shower soap shelf shower soap shelf 8 40 7 otherprop Objects 40 misc +299 window / door window/door 8 40 7 otherprop Objects 40 misc +300 stuffed animal stuffed animal 8 177 40 7 stuffed animal otherprop Objects n04399382 teddy.n.01 39 objects +301 paper towel dispenser paper towel dispenser 8 14 40 7 paper towel dispenser otherprop Objects 40 misc +302 chair bottom chair 8 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +303 fencing fencing 8 40 7 otherprop Objects n03327234 fence.n.01 40 misc +304 lampshade lampshade 8 859 40 7 lamp shade otherprop Objects n03637318 lampshade.n.01 28 lighting +305 door side door frame 12 28 8 12 door door Wall door n03221720 door.n.01 4 door +306 bust bust 7 294 40 7 sculpture otherprop Objects n02926188 bust.n.03 39 objects +307 car car 7 530 40 7 car otherprop Objects car car 02958343 n02958343 car.n.01 39 objects +308 figure figure 7 40 7 otherprop Objects 40 misc +309 sofa set sofa set 7 83 6 9 sofa sofa Sofa sofa sofa sofa 04256520 n04256520 sofa.n.01 10 sofa +310 commode toilet 7 124 33 7 toilet toilet Objects toilet toilet n04446276 toilet.n.01 18 toilet +311 toilet brush toilet brush 7 630 40 7 toilet brush otherprop Objects 40 misc +312 doll doll 7 99 40 7 doll otherprop Objects n03219135 doll.n.01 39 objects +313 drums drum 7 145 40 7 drum otherprop Objects n03249569 drum.n.01 39 objects +314 bathroom counter bathroom counter 7 7 12 6 counter counter Furniture table table table 04379243 n03116530 counter.n.01 26 counter +315 dress dress 7 40 7 otherprop Objects n03236735 dress.n.01 38 clothes +316 shower handle shower handle 7 758 40 7 handle otherprop Objects 40 misc +317 closet door closet door 7 28 8 12 door door Wall door n03221720 door.n.01 4 door +318 whiteboard whiteboard 7 45 30 7 whiteboard whiteboard Objects n03211616 display_panel.n.01 22 tv_monitor +319 garage door opener garage door opener 7 40 7 otherprop Objects 40 misc +320 range hood range hood 7 380 38 7 range hood otherstructure Objects range_hood n04053677 range_hood.n.01 39 objects +321 window curtain window curtain 7 89 16 13 curtain curtain Window curtain n03151077 curtain.n.01 12 curtain +322 easel easel 7 50 39 6 stand otherfurniture Furniture n03262809 easel.n.01 31 shelving +323 bowl of fruit bowl of fruit 7 22 40 7 bowl otherprop Objects bowl bowl 02880940 n02880940 bowl.n.03 39 objects +324 molding molding 10 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +325 pool pool 7 38 7 otherstructure Objects n03982060 pool.n.01 40 misc +326 kitchen appliance kitchen appliance 6 40 7 otherprop Objects n03620052 kitchen_appliance.n.01 37 appliances +327 candelabra candelabra 6 605 40 7 candelabra otherprop Objects n02947818 candelabrum.n.01 39 objects +328 ceiling lamp ceiling lamp 6 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +329 toy toy 6 389 40 7 toy otherprop Objects n03964744 plaything.n.01 39 objects +330 top top 6 40 7 otherprop Objects 40 misc +331 wall art picture 6 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +332 highchair highchair 6 5 5 4 chair chair Chair chair chair chair 03001627 n03518445 highchair.n.01 3 chair +333 footrest footrest 6 163 39 6 foot rest otherfurniture Furniture stool n03380724 footstool.n.01 19 stool +334 bathroom sink sink 6 24 34 7 sink sink Objects sink n04223580 sink.n.01 15 sink +335 soap dish soap dish 6 638 40 7 soap dish otherprop Objects n04254009 soap_dish.n.01 39 objects +336 miscellaneous object object 6 40 7 otherprop Objects n00002684 object.n.01 39 objects +337 trim molding 7 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +338 tabletop object object 6 40 7 otherprop Objects n00002684 object.n.01 39 objects +339 clothes hanger rod clothes hanger rod 6 40 7 otherprop Objects n04100174 rod.n.01 39 objects +340 altar altar 6 19 7 10 table table Table table table table 04379243 n02699629 altar.n.01 5 table +341 candles candle 6 137 40 7 candle otherprop Objects lamp n02948072 candle.n.01 39 objects +342 placemat place mat 6 154 40 7 placemat otherprop Objects n03952886 place_mat.n.01 39 objects +343 kitchen center island kitchen island 6 456 38 7 kitchen island otherstructure Objects n03620600 kitchen_island.n.01 26 counter +344 plate of food plate of food 6 40 7 otherprop Objects 40 misc +345 sheet sheet 7 559 40 7 sheet otherprop Objects 40 misc +346 wood wood 6 40 7 otherprop Objects 40 misc +347 robe robe 6 40 7 otherprop Objects n04097866 robe.n.01 38 clothes +348 bathroom stall bathroom stall 6 38 7 otherstructure Objects n02873839 booth.n.02 40 misc +349 tabletop decoration decoration 6 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +350 plush toy plush toy 6 389 40 7 toy otherprop Objects n04399382 teddy.n.01 39 objects +351 wash basin washbasin 6 24 34 7 sink sink Objects sink n04553920 washbasin.n.01 15 sink +352 celing ceiling 6 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +353 hangers hanger 6 211 40 7 hanger otherprop Objects n03490884 hanger.n.02 39 objects +354 bushes bush 6 82 40 7 plant otherprop Objects plant n13112664 shrub.n.01 14 plant +355 floor/other room floor /otherroom 6 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +356 dinner placesetting place mat 6 154 40 7 placemat otherprop Objects n03952886 place_mat.n.01 39 objects +357 curtain valence curtain valence 6 89 16 13 curtain curtain Window curtain n03151077 curtain.n.01 12 curtain +358 control control 6 40 7 otherprop Objects n03096960 control.n.09 39 objects +359 tap tap 6 9 40 7 faucet otherprop Objects faucet 03325088 n04559451 water_faucet.n.01 39 objects +360 shampoo shampoo 6 548 40 7 cleaner otherprop Objects n04183516 shampoo.n.01 39 objects +361 computer computer 7 46 40 7 computer otherprop Objects n03082979 computer.n.01 39 objects +362 massage bed massage bed 6 157 4 1 bed bed Bed bed bed bed 02818832 n02818832 bed.n.01 11 bed +363 knob knob 6 652 40 7 knob otherprop Objects 40 misc +364 door stopper door stopper 6 40 7 otherprop Objects 40 misc +365 bulletin board bulletin board 6 408 38 7 board otherstructure Objects n03211616 display_panel.n.01 22 tv_monitor +366 brick archway archway 6 21 1 12 wall wall Wall n02734217 arch.n.03 4 door +367 fruit bowl fruit bowl 6 22 40 7 bowl otherprop Objects bowl bowl 02880940 n02880940 bowl.n.03 39 objects +368 electric wire casing electric wire casing 5 40 7 otherprop Objects 40 misc +369 bookcase bookshelf 5 88 10 6 bookshelf bookshelf Furniture bookshelf bookshelf 02871439 n02871439 bookshelf.n.01 31 shelving +370 other room unknown /otherroom 6 20 40 7 unknown otherprop Objects n08632096 unknown.n.01 41 unlabeled +371 exercise machine exercise equipment 5 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +372 storage shelving storage shelving 5 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +373 coffee maker coffee maker 5 40 7 otherprop Objects n03063338 coffee_maker.n.01 37 appliances +374 shower ceiling shower ceiling 5 4 22 3 ceiling ceiling Ceiling n04208936 shower.n.01 23 shower +375 fire alarm fire alarm 5 338 40 7 fire alarm otherprop Objects n03343737 fire_alarm.n.02 39 objects +376 tissue paper tissue paper 5 15 26 7 paper paper Objects 40 misc +377 projector projector 5 90 40 7 projector otherprop Objects n04009552 projector.n.02 39 objects +378 coat hanger coat hanger 5 400 40 7 coat hanger otherprop Objects n03057920 coat_hanger.n.01 39 objects +379 wall cubby wall cubby 5 40 7 otherprop Objects 40 misc +380 balcony railing balcony railing 5 497 38 7 railing otherstructure Objects 40 misc +381 shelf /w clutter shelf /w clutter 5 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +382 case case 5 851 40 7 case otherprop Objects 40 misc +383 coat rack coat rack 5 40 7 otherprop Objects n03059103 coatrack.n.01 40 misc +384 pan pan 5 589 40 7 pan otherprop Objects n03880531 pan.n.01 39 objects +385 fridge refrigerator 5 17 24 6 refridgerator refridgerator Furniture n04070727 refrigerator.n.01 37 appliances +386 suitcase luggage 5 783 40 7 luggage otherprop Objects n02774630 baggage.n.01 39 objects +387 wardrobe rod closet rod 5 40 7 otherprop Objects n04100174 rod.n.01 39 objects +388 hamper clothes hamper 5 39 40 7 basket otherprop Objects basket 02801938 n03050864 clothes_hamper.n.01 39 objects +389 trinket trinket 5 844 40 7 trinket otherprop Objects n02787435 bangle.n.02 39 objects +390 c;lothes hangers clothes hanger 5 211 40 7 hanger otherprop Objects n03057920 coat_hanger.n.01 39 objects +391 paper paper 5 15 26 7 paper paper Objects n14974264 paper.n.01 39 objects +392 back splash backsplash 5 40 7 otherprop Objects 40 misc +393 chimney chimney 5 702 38 7 chimney otherstructure Objects n03017428 chimney.n.01 40 misc +394 arc arch 5 40 7 otherprop Objects n02733524 arch.n.04 40 misc +395 shower bench shower bench 5 204 39 6 bench otherfurniture Furniture bench bench 02828884 n02828884 bench.n.01 34 seating +396 person person 5 331 31 7 person person Objects person n05217688 person.n.02 39 objects +397 tablet tablet 5 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +398 exercise mat exercise mat 5 143 20 5 floor mat floor mat Floor n03727946 mat.n.03 33 gym_equipment +399 smoke alarm smoke alarm 5 525 40 7 alarm otherprop Objects n03343737 fire_alarm.n.02 39 objects +400 kitchen utensils kitchen utensil 5 267 40 7 utensil otherprop Objects n03621049 kitchen_utensil.n.01 39 objects +401 weights weight 5 40 7 otherprop Objects n04571292 weight.n.02 33 gym_equipment +402 display cabinet display cabinet 5 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +403 showcase display cabinet 5 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +404 bedpost bedpost 5 40 7 otherprop Objects n02821415 bedpost.n.01 11 bed +405 file cabinet file cabinet 5 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +406 umbrella umbrella 5 203 40 7 umbrella otherprop Objects n04507155 umbrella.n.01 39 objects +407 massage table massage table 5 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +408 laundry basket laundry basket 5 164 40 7 laundry basket otherprop Objects basket 02801938 n03050864 clothes_hamper.n.01 39 objects +409 jar jar 5 70 40 7 jar otherprop Objects jar 03593526 n03593526 jar.n.01 39 objects +410 exercise bike exercise bike 5 457 39 6 excercise equipment otherfurniture Furniture n03302671 exercise_bike.n.01 33 gym_equipment +411 hose hose 5 40 7 otherprop Objects n03539875 hose.n.03 40 misc +412 window dormer window dormer 5 40 7 otherprop Objects 40 misc +413 closet shelf closet shelf 5 40 7 otherprop Objects n04190052 shelf.n.01 31 shelving +414 power breaker box power breaker box 5 26 29 7 box box Objects 40 misc +415 smoke detector smoke detector 5 40 7 otherprop Objects 40 misc +416 door knob door knob 4 27 40 7 door knob otherprop Objects 40 misc +417 shelf side shelf 4 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +418 jacuzzi jacuzzi 4 40 7 otherprop Objects 40 misc +419 backpack backpack 4 206 40 7 backpack otherprop Objects n02769748 backpack.n.01 39 objects +420 wooden desk desk 4 36 14 10 desk desk Table desk desk table 04379243 n03179701 desk.n.01 5 table +421 bath mat bath mat 4 40 7 otherprop Objects n02807401 bath_mat.n.01 2 floor +422 unknown clutter unknown clutter 4 40 7 otherprop Objects 40 misc +423 hook hook 4 40 7 otherprop Objects 40 misc +424 sauna wall wall 4 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +425 elevator elevator 4 40 7 otherprop Objects 40 misc +426 tool tool 4 40 7 otherprop Objects n04451818 tool.n.01 39 objects +427 recliner recliner 4 5 5 4 chair chair Chair chair chair chair 03001627 n04062428 recliner.n.01 3 chair +428 recessed wall recessed wall 4 21 1 12 wall wall Wall 40 misc +429 closet bar closet rod 4 40 7 otherprop Objects n04100174 rod.n.01 39 objects +430 tank tank 4 40 7 otherprop Objects 40 misc +431 toaster toaster 4 251 40 7 toaster otherprop Objects n04442312 toaster.n.02 37 appliances +432 toilet brush holder toilet brush holder 4 40 7 otherprop Objects 40 misc +433 handrail /otherroom handrail /otherroom 4 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +434 landing landing 6 40 7 otherprop Objects n03638511 landing.n.01 2 floor +435 book rack book rack 4 224 39 6 bookrack otherfurniture Furniture 40 misc +436 wall mirror mirror 4 122 19 7 mirror mirror Objects n03773035 mirror.n.01 21 mirror +437 hunting trophy hunting trophy 4 547 40 7 trophy otherprop Objects 40 misc +438 rod rod 4 40 7 otherprop Objects n04100174 rod.n.01 39 objects +439 floor mat floor mat 4 143 20 5 floor mat floor mat Floor n03727837 mat.n.01 2 floor +440 motion detector motion detector 4 40 7 otherprop Objects 40 misc +441 clothing clothes 4 141 21 7 clothes clothes Objects n02728440 apparel.n.01 38 clothes +442 can of paint can of paint 4 40 7 otherprop Objects 40 misc +443 medicine cabinet medicine cabinet 4 3 3 6 cabinet cabinet Furniture cabinet 02933112 n03742115 medicine_chest.n.01 7 cabinet +444 sensor sensor 4 40 7 otherprop Objects n03180969 detector.n.01 39 objects +445 cart cart 4 305 40 7 cart otherprop Objects n03484083 handcart.n.01 39 objects +446 slab slab 4 38 7 otherstructure Objects n04233405 slab.n.01 39 objects +447 bean bag chair bean bag chair 4 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +448 window valence window valence 4 89 16 13 curtain curtain Window curtain n03151077 curtain.n.01 12 curtain +449 window /outside window /outside 4 59 9 13 window window Window n04587648 window.n.01 9 window +450 pole pole 4 40 7 otherprop Objects n03976657 pole.n.01 39 objects +451 picture / other room picture /otherroom 4 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +452 canister canister 4 794 40 7 canister otherprop Objects 40 misc +453 washbasin counter washbasin counter 4 7 12 6 counter counter Furniture table table table 04379243 n03116530 counter.n.01 26 counter +454 kitchen top kitchen top 4 7 12 6 counter counter Furniture n03118245 countertop.n.01 26 counter +455 unknown object object 4 40 7 otherprop Objects n00002684 object.n.01 39 objects +456 pitcher pitcher 4 273 40 7 pitcher otherprop Objects n03950228 pitcher.n.02 39 objects +457 showerhead showerhead 4 650 40 7 shower head otherprop Objects n04209383 showerhead.n.01 23 shower +458 podium podium 4 40 7 otherprop Objects n03159640 dais.n.01 39 objects +459 ceiling vent ceiling vent 4 25 38 7 air vent otherstructure Objects n04526241 vent.n.01 40 misc +460 throw pillow pillow 4 119 18 7 pillow pillow Objects pillow 03938244 n03938244 pillow.n.01 8 cushion +461 grill grill 4 700 38 7 grill otherstructure Objects n03459591 grill.n.02 40 misc +462 sink cabinet sink cabinet 4 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +463 tapestry tapestry 4 40 7 otherprop Objects n04393549 tapestry.n.02 39 objects +464 bed sheet bed sheet 5 559 40 7 sheet otherprop Objects n04188179 sheet.n.03 11 bed +465 shade shade 4 40 7 otherprop Objects n04181718 shade.n.03 39 objects +466 doorknob doorknob 4 27 40 7 door knob otherprop Objects n03222959 doorknob.n.01 4 door +467 stair railing banister 4 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +468 vacuum cleaner vacuum cleaner 4 306 40 7 vacuum cleaner otherprop Objects n04517823 vacuum.n.04 37 appliances +469 bed comforter bed comforter 4 484 40 7 comforter otherprop Objects 40 misc +470 door / other room door /otherroom 4 28 8 12 door door Wall door n03221720 door.n.01 4 door +471 pictures picture 4 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +472 bed table bed table 4 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +473 shirt shirt 4 40 7 otherprop Objects n04197391 shirt.n.01 38 clothes +474 dressing table dressing table 4 19 7 10 table table Table table table table 04379243 n03238586 dressing_table.n.01 5 table +475 shower wall cubby shower wall cubby 4 40 7 otherprop Objects n04208936 shower.n.01 23 shower +476 side wall wall 4 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +477 beside table beside table 4 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +478 curb curb 4 40 7 otherprop Objects n03149135 curb.n.02 39 objects +479 arches arch 4 40 7 otherprop Objects n02733524 arch.n.04 40 misc +480 storage bin storage bin 4 812 40 7 storage bin otherprop Objects 40 misc +481 support beam support beam 4 40 7 otherprop Objects n02815950 beam.n.02 29 beam +482 globe globe 5 347 40 7 globe otherprop Objects 40 misc +483 pantry pantry 4 38 7 otherstructure Objects n03885535 pantry.n.01 40 misc +484 skateboard skateboard 4 408 38 7 board otherstructure Objects skateboard 04225987 n04225987 skateboard.n.01 39 objects +485 stove hood range hood 4 380 38 7 range hood otherstructure Objects range_hood n04053677 range_hood.n.01 39 objects +486 cabin cabin 4 40 7 otherprop Objects 40 misc +487 shower bar shower bar 4 51 38 7 bar otherstructure Objects 40 misc +488 chaise chaise 4 5 5 4 chair chair Chair chair chair chair 03001627 n03002711 chaise_longue.n.01 3 chair +489 flower wash flower vase 4 78 40 7 vase otherprop Objects vase jar 03593526 n04522168 vase.n.01 39 objects +490 desk and chairs desk and chairs 4 40 7 otherprop Objects 40 misc +491 plant vase flower vase 4 78 40 7 vase otherprop Objects vase jar 03593526 n04522168 vase.n.01 39 objects +492 towel rack towel rack 4 40 7 otherprop Objects n04459773 towel_rack.n.01 40 misc +493 cross cross 4 49 40 7 monitor otherprop Objects n03857828 oscilloscope.n.01 39 objects +494 sliding door sliding door 4 28 8 12 door door Wall door n04239074 sliding_door.n.01 4 door +495 cosmetics cosmetics 4 40 7 otherprop Objects n03113152 cosmetic.n.01 39 objects +496 wall other room wall /otherroom 4 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +497 kettle kettle 4 16 40 7 pot otherprop Objects n03612814 kettle.n.01 39 objects +498 junk junk 4 40 7 otherprop Objects n14857897 debris.n.01 39 objects +499 stationery stationery 3 15 26 7 paper paper Objects n06258852 stationery.n.01 39 objects +500 lights light 3 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +501 window / other room window /otherroom 3 59 9 13 window window Window n04587648 window.n.01 9 window +502 dish cabinet dish cabinet 3 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +503 wall panel wall panel 3 21 1 12 wall wall Wall n04548503 wall_panel.n.01 1 wall +504 gate gate 3 223 38 7 gate otherstructure Objects n03427296 gate.n.01 40 misc +505 safe safe 3 26 29 7 box box Objects n04125021 safe.n.01 39 objects +506 ventilation ventilation 3 40 7 otherprop Objects n04526520 ventilation.n.02 39 objects +507 logs firewood 3 40 7 otherprop Objects n15100644 firewood.n.01 40 misc +508 sliding doors sliding door 3 28 8 12 door door Wall door n04239074 sliding_door.n.01 4 door +509 shower rod shower rod 3 40 7 otherprop Objects n04100174 rod.n.01 12 curtain +510 towel shelf towel shelf 3 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +511 row of theater chairs row of theater chairs 3 40 7 otherprop Objects 40 misc +512 closet wall wall 3 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +513 bath utencils bath utensil 3 267 40 7 utensil otherprop Objects 40 misc +514 sconce sconce 3 62 38 7 light otherstructure Objects n04148703 sconce.n.03 28 lighting +515 garage light garage light 3 62 38 7 light otherstructure Objects 40 misc +516 ceiling beam beam 3 38 7 otherstructure Objects n02815950 beam.n.02 29 beam +517 toolbox toolbox 3 344 39 6 chest otherfurniture Furniture n04452615 toolbox.n.01 39 objects +518 sponge stool stool 3 150 40 7 stool otherprop Objects stool n04326896 stool.n.01 19 stool +519 security camera security camera 3 212 40 7 security camera otherprop Objects camera 02942699 n02942699 camera.n.01 39 objects +520 gym machine exercise equipment 3 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +521 wall /outside wall /outside 3 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +522 mantle mantle 3 874 38 7 mantle otherstructure Objects mantel n03719343 mantel.n.01 27 fireplace +523 floor 2 floor 3 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +524 skirting skirting 3 40 7 otherprop Objects 40 misc +525 banister banister 3 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +526 lockers locker 3 3 3 6 cabinet cabinet Furniture n02933462 cabinet.n.03 40 misc +527 trophy trophy 3 547 40 7 trophy otherprop Objects 40 misc +528 tile tile 3 40 7 otherprop Objects n04435180 tile.n.01 39 objects +529 unknown / room below unknown /otherroom 3 20 40 7 unknown otherprop Objects n08632096 unknown.n.01 41 unlabeled +530 picture car picture car 3 530 40 7 car otherprop Objects car car 02958343 n02958343 car.n.01 39 objects +531 outlet outlet 3 40 7 otherprop Objects n04548771 wall_socket.n.01 39 objects +532 plant ridge plant ridge 3 40 7 otherprop Objects 40 misc +533 backsplash backsplash 3 40 7 otherprop Objects 40 misc +534 computer monitor monitor 3 49 40 7 monitor otherprop Objects monitor monitor tv or monitor 03211117 n03782190 monitor.n.04 22 tv_monitor +535 doorframe / other room doorframe /otherroom 3 615 38 7 door frame otherstructure Objects n03222722 doorframe.n.01 4 door +536 shower wall /otherroom shower wall /otherroom 3 21 1 12 wall wall Wall n04208936 shower.n.01 23 shower +537 shelves with wine shelving 3 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +538 shoe shoe 3 149 40 7 shoe otherprop Objects n04199027 shoe.n.01 39 objects +539 hedge hedge 3 40 7 otherprop Objects n03511175 hedge.n.01 40 misc +540 glass window window 3 59 9 13 window window Window n04587648 window.n.01 9 window +541 sofachair sofa chair 3 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +542 hand soap hand soap 3 133 40 7 soap otherprop Objects n04253437 soap.n.01 39 objects +543 window blinds blinds 3 80 13 13 blinds blinds Window n04589190 window_blind.n.01 32 blinds +544 flower vase flower vase 3 78 40 7 vase otherprop Objects vase jar 03593526 n04522168 vase.n.01 39 objects +545 appliance appliance 3 40 7 otherprop Objects 40 misc +546 christmas tree christmas tree 3 40 7 otherprop Objects 40 misc +547 support column column 3 94 38 7 column otherstructure Objects n03074380 column.n.06 40 misc +548 dish plate 3 233 40 7 plate otherprop Objects n03959485 plate.n.04 39 objects +549 closet floor closet floor 3 11 2 5 floor floor Floor 40 misc +550 celling ceiling 3 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +551 stuffed animals stuffed animal 3 177 40 7 stuffed animal otherprop Objects n04399382 teddy.n.01 39 objects +552 casket casket 3 40 7 otherprop Objects 40 misc +553 rowing machine exercise machine 3 220 40 7 machine otherprop Objects 40 misc +554 dining table centerpiece dining table centerpiece 3 878 40 7 centerpiece otherprop Objects 40 misc +555 bedside lamp bedside lamp 5 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +556 kitchen countertop item kitchen countertop item 3 40 7 otherprop Objects 40 misc +557 fountain fountain 3 40 7 otherprop Objects n03388043 fountain.n.01 40 misc +558 window glass window glass 3 612 38 7 glass otherstructure Objects n03881893 pane.n.01 39 objects +559 wall soffet soffit 3 40 7 otherprop Objects n04256758 soffit.n.01 39 objects +560 urinal urinal 3 40 7 otherprop Objects n04515991 urinal.n.01 39 objects +561 kitchen hood range hood 3 380 38 7 range hood otherstructure Objects range_hood n04053677 range_hood.n.01 39 objects +562 showpiece decoration 3 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +563 wall trim molding 3 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +564 barrel barrel 3 343 39 6 barrel otherfurniture Furniture 40 misc +565 firewood firewood 3 40 7 otherprop Objects n15100644 firewood.n.01 40 misc +566 columns pillar 3 94 38 7 column otherstructure Objects n03073977 column.n.07 24 column +567 beams beam 3 38 7 otherstructure Objects n02815950 beam.n.02 29 beam +568 carpet roll carpet roll 3 40 7 otherprop Objects 40 misc +569 portrait portrait 3 64 11 8 picture picture Picture n03987079 portrait.n.02 6 picture +570 table light table light 3 62 38 7 light otherstructure Objects 40 misc +571 water heater water heater 3 588 40 7 water heater otherprop Objects n04560113 water_heater.n.01 39 objects +572 stairs 2 stair 3 215 38 7 stairs otherstructure Objects stairs n04314914 step.n.04 16 stairs +573 pouffe pouffe 3 359 39 6 ottoman otherfurniture Furniture n03858418 ottoman.n.03 34 seating +574 ceiling behind ceiling /otherroom 3 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +575 pillars pillar 3 94 38 7 column otherstructure Objects n03073977 column.n.07 24 column +576 concrete block concrete block 3 40 7 otherprop Objects 40 misc +577 range stove 3 242 38 7 stove otherstructure Objects stove 04330267 n04330267 stove.n.02 37 appliances +578 shelf with objects shelf 3 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +579 toilet seat liner dispenser toilet seat liner dispenser 3 40 7 otherprop Objects 40 misc +580 patio chair patio chair 3 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +581 folding chair folding chair 3 5 5 4 chair chair Chair chair chair chair 03001627 n03376595 folding_chair.n.01 3 chair +582 island island 4 456 38 7 kitchen island otherstructure Objects n03620600 kitchen_island.n.01 26 counter +583 toaster oven toaster oven 3 275 40 7 toaster oven otherprop Objects n04442441 toaster_oven.n.01 37 appliances +584 overlook railing railing 3 497 38 7 railing otherstructure Objects n04047401 railing.n.01 30 railing +585 bathroom mirror mirror 3 122 19 7 mirror mirror Objects n03773035 mirror.n.01 21 mirror +586 recycle bin recycle bin 3 307 40 7 bin otherprop Objects 40 misc +587 counter top countertop 3 7 12 6 counter counter Furniture n03118245 countertop.n.01 26 counter +588 rafter rafter 3 40 7 otherprop Objects n04045644 rafter.n.01 29 beam +589 dryer clothes dryer 3 39 6 otherfurniture Furniture n03251766 dryer.n.01 37 appliances +590 bed lamp bedside lamp 3 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +591 wall paneling paneling 3 21 1 12 wall wall Wall n03882611 paneling.n.01 1 wall +592 stage stage 3 40 7 otherprop Objects 40 misc +593 fire sprinkler fire sprinkler 3 40 7 otherprop Objects 40 misc +594 brush brush 3 40 7 otherprop Objects n02908217 brush.n.02 39 objects +595 wall 2 wall 3 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +596 balcony balcony 3 40 7 otherprop Objects 40 misc +597 water tank water tank 2 263 40 7 vessel otherprop Objects n03035715 cistern.n.02 39 objects +598 pile of objects clutter 2 40 7 otherprop Objects 40 misc +599 garage door frame garage door frame 2 40 7 otherprop Objects 40 misc +600 back wall wall 2 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +601 plant soil plant soil 2 40 7 otherprop Objects 40 misc +602 globe stand globe stand 2 466 40 7 globe stand otherprop Objects 40 misc +603 bicycle bicycle 2 189 40 7 bicycle otherprop Objects bicycle 02834778 n02834778 bicycle.n.01 39 objects +604 stairframe stair frame 2 40 7 otherprop Objects 40 misc +605 separating screen partition 2 21 1 12 wall wall Wall n03894379 partition.n.01 40 misc +606 unknown wall object object 2 40 7 otherprop Objects n00002684 object.n.01 39 objects +607 air duct air duct 2 38 38 7 air duct otherstructure Objects n02690941 air_passage.n.01 40 misc +608 kitchen upper cabinet kitchen cabinet 2 3 3 6 cabinet cabinet Furniture n02933112 cabinet.n.01 7 cabinet +609 chair pillow pillow 2 119 18 7 pillow pillow Objects pillow 03938244 n03938244 pillow.n.01 8 cushion +610 led tv led tv 2 40 7 otherprop Objects 40 misc +611 deco decoration 2 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +612 ceiling under staircase ceiling under staircase 2 40 7 otherprop Objects 40 misc +613 chair /w books chair /w books 2 85 23 2 books books Books 40 misc +614 bed back rest headboard 2 161 39 6 headboard otherfurniture Furniture n03502200 headboard.n.01 11 bed +615 giraffe giraffe 2 40 7 otherprop Objects n02439033 giraffe.n.01 39 objects +616 lightswitch light switch 2 301 38 7 light switch otherstructure Objects n04372370 switch.n.01 39 objects +617 doorframe \other room doorframe /otherroom 2 615 38 7 door frame otherstructure Objects n03222722 doorframe.n.01 4 door +618 object / camera? object 2 40 7 otherprop Objects n00002684 object.n.01 39 objects +619 grandfather clock grandfather clock 2 462 39 6 grandfather clock otherfurniture Furniture clock 03046257 n03452594 grandfather_clock.n.01 39 objects +620 jewelry box jewelry box 2 26 29 7 box box Objects 40 misc +621 bottles of wine bottles of wine 2 766 40 7 wine otherprop Objects 40 misc +622 massage table base massage table 2 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +623 hood range hood 2 380 38 7 range hood otherstructure Objects range_hood n04053677 range_hood.n.01 39 objects +624 mirrorframe mirror frame 2 122 19 7 mirror mirror Objects n03773035 mirror.n.01 21 mirror +625 wall beam wall beam 2 40 7 otherprop Objects 40 misc +626 wooden trim molding 2 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +627 stalls stall 2 38 7 otherstructure Objects n02873839 booth.n.02 40 misc +628 partition partition 2 21 1 12 wall wall Wall n03894379 partition.n.01 40 misc +629 dog dog 2 701 40 7 dog otherprop Objects 40 misc +630 valance valance 2 40 7 otherprop Objects n03111296 cornice.n.01 40 misc +631 radio radio 2 188 40 7 radio otherprop Objects radio 40 misc +632 bush bush 2 82 40 7 plant otherprop Objects plant n13112664 shrub.n.01 14 plant +633 row of theater seats row of theater seats 2 40 7 otherprop Objects 40 misc +634 bath utencil bath utensil 2 267 40 7 utensil otherprop Objects 40 misc +635 basket of towels basket of towels 2 40 7 otherprop Objects 40 misc +636 laundry machine washing machine 2 278 39 6 washing machine otherfurniture Furniture washing_machine 04554684 n04554684 washer.n.03 37 appliances +637 mirror /otherroom mirror /otherroom 2 122 19 7 mirror mirror Objects n03773035 mirror.n.01 21 mirror +638 toilet sink toilet sink 2 24 34 7 sink sink Objects sink 40 misc +639 sauna heater sauna heater 2 111 39 6 heater otherfurniture Furniture 40 misc +640 dining bench dining bench 2 204 39 6 bench otherfurniture Furniture bench bench 02828884 n02828884 bench.n.01 34 seating +641 fume cupboard fume cupboard 2 40 7 otherprop Objects 40 misc +642 mouse mouse 2 103 40 7 mouse otherprop Objects n03793489 mouse.n.04 39 objects +643 boiler boiler 2 40 7 otherprop Objects 40 misc +644 hearth hearth 2 372 38 7 fireplace otherstructure Objects 40 misc +645 curtain darker curtain 2 89 16 13 curtain curtain Window curtain n03151077 curtain.n.01 12 curtain +646 round chair round chair 2 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +647 toilet bowl toilet 2 124 33 7 toilet toilet Objects toilet toilet n04446276 toilet.n.01 18 toilet +648 whine rack wine rack 2 299 40 7 wine rack otherprop Objects 40 misc +649 doorstep doorstep 3 28 8 12 door door Wall n03223686 doorsill.n.01 39 objects +650 binder binder 2 399 40 7 binder otherprop Objects 40 misc +651 shower door frame shower door frame 2 40 7 otherprop Objects n04208936 shower.n.01 23 shower +652 bed runner bed runner 2 157 4 1 bed bed Bed n02822579 bedstead.n.01 11 bed +653 cubicle cubicle 2 40 7 otherprop Objects 40 misc +654 fitness device exercise equipment 2 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +655 support support 2 40 7 otherprop Objects 40 misc +656 overhang overhang 2 40 7 otherprop Objects n03864356 overhang.n.01 40 misc +657 electric box electric box 2 550 38 7 electric box otherstructure Objects 40 misc +658 bathrobe bathrobe 3 40 7 otherprop Objects n02807616 bathrobe.n.01 38 clothes +659 door mat doormat 2 143 20 5 floor mat floor mat Floor n03223299 doormat.n.02 2 floor +660 jacket jacket 2 324 40 7 jacket otherprop Objects n03589791 jacket.n.01 38 clothes +661 cabinet table cabinet table 2 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +662 side frame frame 2 38 7 otherstructure Objects 40 misc +663 wainscoting paneling 2 21 1 12 wall wall Wall n03882611 paneling.n.01 1 wall +664 staircase trim staircase trim 2 40 7 otherprop Objects 40 misc +665 box /w books box /w books 2 85 23 2 books books Books 40 misc +666 nighstand nightstand 2 158 32 6 night stand night stand Furniture night_stand night_stand n03015254 chest_of_drawers.n.01 13 chest_of_drawers +667 window reflection window reflection 2 40 7 otherprop Objects 40 misc +668 pulpit pulpit 2 40 7 otherprop Objects n03159640 dais.n.01 39 objects +669 set of armchairs set of armchairs 2 40 7 otherprop Objects 40 misc +670 fish tank fish tank 2 782 38 7 fish tank otherstructure Objects n02732072 aquarium.n.01 40 misc +671 bathroom countertop objects objects 2 40 7 otherprop Objects n00002684 object.n.01 39 objects +672 concrete shelf shelf 2 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +673 ceiling wall ceiling wall 2 21 1 12 wall wall Wall 40 misc +674 picture \other room picture /otherroom 2 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +675 wall entry wall 2 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +676 lintel lintel 2 40 7 otherprop Objects n03503233 header.n.02 29 beam +677 wall table wall table 2 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +678 small table table 3 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +679 lighting fixture lighting fixture 2 40 7 otherprop Objects n03667380 lighting_fixture.n.01 39 objects +680 bed frame bedframe 2 157 4 1 bed bed Bed n02822579 bedstead.n.01 11 bed +681 freezer freezer 2 17 24 6 refridgerator refridgerator Furniture n03170635 deep-freeze.n.01 37 appliances +682 glass doors door 2 28 8 12 door door Wall door n03221720 door.n.01 4 door +683 extractor extractor 2 40 7 otherprop Objects 40 misc +684 flower pot flowerpot 2 146 40 7 flower pot otherprop Objects flower_pot flower pot 03991062 n03991062 pot.n.04 39 objects +685 soffet soffit 2 40 7 otherprop Objects n04256758 soffit.n.01 39 objects +686 platform platform 2 38 7 otherstructure Objects 40 misc +687 hot tub hot tub 2 136 36 7 bathtub bathtub Objects bathtub bathtub tub 02808440 n03543603 hot_tub.n.01 25 bathtub +688 paper towels paper towel 2 113 40 7 paper towel otherprop Objects n03887697 paper_towel.n.01 20 towel +689 kitchen utencils kitchen utensil 2 267 40 7 utensil otherprop Objects n03621049 kitchen_utensil.n.01 39 objects +690 shower grab bar shower grab bar 2 51 38 7 bar otherstructure Objects 40 misc +691 wall detail wall detail 2 40 7 otherprop Objects 40 misc +692 whineshelf whine shelf 2 40 7 otherprop Objects 40 misc +693 painting/other room painting /otherroom 2 64 11 8 picture picture Picture n03876519 painting.n.01 39 objects +694 television wall tv 2 40 7 otherprop Objects 40 misc +695 sauna bench bench 2 204 39 6 bench otherfurniture Furniture bench bench 02828884 n02828884 bench.n.01 34 seating +696 fruits fruit 2 286 40 7 fruit otherprop Objects n13134947 fruit.n.01 39 objects +697 picture frames picture frame 2 64 11 8 picture picture Picture n03931765 picture_frame.n.01 6 picture +698 buffet buffet 2 7 12 6 counter counter Furniture table table table 04379243 n04247736 snack_bar.n.01 26 counter +699 billow billow 2 40 7 otherprop Objects 40 misc +700 computer chair computer chair 2 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +701 sofa seat sofa seat 2 40 7 otherprop Objects 40 misc +702 wall tv wall tv 2 40 7 otherprop Objects 40 misc +703 ground floor 2 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +704 wall2 wall 2 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +705 ceiling under stairs ceiling under stairs 2 215 38 7 stairs otherstructure Objects stairs 40 misc +706 calendar calendar 2 583 40 7 calendar otherprop Objects 40 misc +707 dome dome 2 40 7 otherprop Objects 40 misc +708 object /outside object /outside 2 40 7 otherprop Objects n00002684 object.n.01 39 objects +709 poll poll 2 40 7 otherprop Objects n01817346 poll.n.04 39 objects +710 wet bar wet bar 2 51 38 7 bar otherstructure Objects table table table 04379243 n04573513 wet_bar.n.01 26 counter +711 folding table folding table 2 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +712 stovetop stovetop 2 40 7 otherprop Objects 40 misc +713 gym equip gym equipment 2 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +714 fruitbowl fruit bowl 2 22 40 7 bowl otherprop Objects bowl bowl 02880940 n02880940 bowl.n.03 39 objects +715 vending machine vending machine 2 220 40 7 machine otherprop Objects n04525305 vending_machine.n.01 39 objects +716 handwash hand wash 2 40 7 otherprop Objects 40 misc +717 wall clock wall clock 2 56 40 7 clock otherprop Objects clock 03046257 n04548280 wall_clock.n.01 39 objects +718 liquid soap liquid soap 2 133 40 7 soap otherprop Objects 40 misc +719 trinkets trinket 2 844 40 7 trinket otherprop Objects n02787435 bangle.n.02 39 objects +720 small table / stand small table/stand 2 40 7 otherprop Objects 40 misc +721 window shuts window shutters 2 40 7 otherprop Objects n04211356 shutter.n.02 39 objects +722 door frame 3 door frame 2 28 8 12 door door Wall door n03221720 door.n.01 4 door +723 stone stone 2 578 40 7 stones otherprop Objects n09416076 rock.n.01 39 objects +724 tripod tripod 2 50 39 6 stand otherfurniture Furniture n04485082 tripod.n.01 31 shelving +725 projector screen projector screen 2 53 38 7 projector screen otherstructure Objects 40 misc +726 window frame / other room window frame /otherroom 2 477 38 7 window frame otherstructure Objects n04589593 window_frame.n.01 9 window +727 wreath wreath 2 881 40 7 wreathe otherprop Objects n04606014 wreath.n.01 39 objects +728 door hinge door hinge 2 40 7 otherprop Objects 40 misc +729 sound speaker 2 54 40 7 speaker otherprop Objects speaker 03691459 n03691459 loudspeaker.n.01 39 objects +730 french door french door 2 28 8 12 door door Wall door n03394649 french_door.n.01 4 door +731 staircase handrail staircase handrail 2 40 7 otherprop Objects 40 misc +732 nighttable night table 2 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +733 photo in frame picure 2 64 11 8 picture picture Picture 40 misc +734 photoframe picture 2 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +735 stair wall stair wall 2 21 1 12 wall wall Wall 40 misc +736 closet ceiling closet ceiling 2 4 22 3 ceiling ceiling Ceiling 40 misc +737 stick stick 2 529 40 7 stick otherprop Objects 40 misc +738 fluorescent light fluorescent light 2 62 38 7 light otherstructure Objects 40 misc +739 wash cabinet wash cabinet 2 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +740 shower seat shower seat 2 40 7 otherprop Objects 40 misc +741 trellis trellis 2 40 7 otherprop Objects n04478512 trellis.n.01 40 misc +742 patio patio 2 40 7 otherprop Objects n03899768 patio.n.01 40 misc +743 dart board dartboard 2 408 38 7 board otherstructure Objects n03162940 dartboard.n.01 39 objects +744 comforter comforter 2 484 40 7 comforter otherprop Objects n04033995 quilt.n.01 39 objects +745 table /w books table /w books 2 85 23 2 books books Books 40 misc +746 picture/other room picture /otherroom 2 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +747 dirt dirt 2 40 7 otherprop Objects 40 misc +748 base base 2 40 7 otherprop Objects 40 misc +749 chemical tank chemical tank 2 40 7 otherprop Objects 40 misc +750 step stool step stool 2 276 40 7 step stool otherprop Objects stool n04315713 step_stool.n.01 19 stool +751 misc misc 2 40 7 otherprop Objects 40 misc +752 sink table sink table 2 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +753 curved wall wall 2 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +754 roof beam beam 2 38 7 otherstructure Objects n02815950 beam.n.02 29 beam +755 cover cover 2 312 40 7 blanket otherprop Objects 40 misc +756 reading table reading table 2 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +757 side steps wall steps wall 2 21 1 12 wall wall Wall 40 misc +758 sideboard sideboard 2 7 12 6 counter counter Furniture 40 misc +759 wall from another room wall /otherroom 2 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +760 electric outlet electric outlet 2 98 40 7 electrical outlet otherprop Objects n04548771 wall_socket.n.01 39 objects +761 stall door stall door 2 28 8 12 door door Wall door n03221720 door.n.01 4 door +762 separator separator 2 40 7 otherprop Objects n02995998 centrifuge.n.01 39 objects +763 toilet bowl brush holder toilet bowl brush holder 2 40 7 otherprop Objects 40 misc +764 vessel vessel 2 263 40 7 vessel otherprop Objects watercraft 04530566 n04530566 vessel.n.02 39 objects +765 table clutter table clutter 2 40 7 otherprop Objects 40 misc +766 partition wall wall 2 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +767 bed curtain bed curtain 2 89 16 13 curtain curtain Window curtain 40 misc +768 stairs skirt stairs skirt 2 40 7 otherprop Objects 40 misc +769 small chair chair 2 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +770 ceiling window ceiling window 2 59 9 13 window window Window 40 misc +771 rocking chair rocking chair 2 5 5 4 chair chair Chair chair chair chair 03001627 n04099969 rocking_chair.n.01 3 chair +772 window \other room window /otherroom 2 59 9 13 window window Window n04587648 window.n.01 9 window +773 under stair under stair 2 40 7 otherprop Objects 40 misc +774 unknown outside building unknown /outside 2 20 40 7 unknown otherprop Objects n08632096 unknown.n.01 41 unlabeled +775 bath wall bath wall 2 21 1 12 wall wall Wall 40 misc +776 panel screen panel screen 2 40 7 otherprop Objects 40 misc +777 doorfra,e doorframe 2 615 38 7 door frame otherstructure Objects n03222722 doorframe.n.01 4 door +778 shower mat shower mat 2 143 20 5 floor mat floor mat Floor n03727837 mat.n.01 2 floor +779 blackboard blackboard 2 225 38 7 blackboard otherstructure Objects n02846511 blackboard.n.01 39 objects +780 drawer desk drawer desk 2 36 14 10 desk desk Table desk desk 40 misc +781 poster picture 2 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +782 fireplace sconce fireplace sconce 2 40 7 otherprop Objects 40 misc +783 table support table support 2 40 7 otherprop Objects 40 misc +784 wall lamp wall lamp 2 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +785 closet shelving closet shelving 5 40 7 otherprop Objects 40 misc +786 closest area closest area 2 40 7 otherprop Objects 40 misc +787 scroll scroll 2 40 7 otherprop Objects 40 misc +788 foot stand foot stand 2 50 39 6 stand otherfurniture Furniture 40 misc +789 button button 2 774 40 7 button otherprop Objects 40 misc +790 art / clutter art/clutter 2 40 7 otherprop Objects 40 misc +791 door arc arch 2 40 7 otherprop Objects n02733524 arch.n.04 40 misc +792 stairs railing stairs railing 2 497 38 7 railing otherstructure Objects 40 misc +793 floor /otherroom floor /otherroom 2 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +794 shovel shovel 2 607 40 7 shovel otherprop Objects 40 misc +795 alarm controls alarm control 2 40 7 otherprop Objects 40 misc +796 lamb lamp 2 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +797 cluttered objects clutter 2 40 7 otherprop Objects 40 misc +798 door/other room door /otherroom 2 28 8 12 door door Wall door n03221720 door.n.01 4 door +799 closet shelves closet shelf 2 40 7 otherprop Objects n04190052 shelf.n.01 31 shelving +800 scultpure sculpture 2 294 40 7 sculpture otherprop Objects n04157320 sculpture.n.01 39 objects +801 arm chair armchair 2 5 5 4 chair chair Chair chair chair chair 03001627 n02738535 armchair.n.01 3 chair +802 door behind door /otherroom 2 28 8 12 door door Wall door n03221720 door.n.01 4 door +803 exercise ball exercise ball 2 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +804 yard yard 2 40 7 otherprop Objects 40 misc +805 semi chair semi chair 2 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +806 bouquet bouquet 2 40 7 otherprop Objects n02879087 bouquet.n.01 39 objects +807 garage door opener bar garage door opener bar 2 51 38 7 bar otherstructure Objects 40 misc +808 pots pot 2 16 40 7 pot otherprop Objects n03991062 pot.n.04 39 objects +809 decorations decoration 2 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +810 unkown unknown 2 20 40 7 unknown otherprop Objects n08632096 unknown.n.01 41 unlabeled +811 kitchen decoration kitchen decoration 2 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +812 archway corner archway corner 2 40 7 otherprop Objects 40 misc +813 kitchen wall kitchen wall 2 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +814 cabinet door cabinet door 2 28 8 12 door door Wall door 40 misc +815 sauna bowl sauna bowl 2 22 40 7 bowl otherprop Objects bowl bowl 02880940 n02880940 bowl.n.03 39 objects +816 bar chair bar chair 2 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +817 shelf cubby shelf cubby 2 40 7 otherprop Objects 40 misc +818 toilet plunger toilet plunger 2 563 40 7 toilet plunger otherprop Objects n03970156 plunger.n.03 39 objects +819 belts belt 2 610 40 7 belt otherprop Objects 40 misc +820 sewing machine sewing machine 2 890 40 7 sewing machine otherprop Objects n04179913 sewing_machine.n.01 37 appliances +821 hot water/cold water knob hot water/cold water knob 2 652 40 7 knob otherprop Objects 40 misc +822 barbeque barbecue 2 40 7 otherprop Objects n02790669 barbecue.n.03 40 misc +823 cutting board cutting board 2 247 40 7 cutting board otherprop Objects n03025513 chopping_board.n.01 39 objects +824 soapbox soapbox 2 671 40 7 soap box otherprop Objects 40 misc +825 washing stuff washing stuff 2 40 7 otherprop Objects 40 misc +826 dining table decoration decoration 2 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +827 copier copier 2 40 7 otherprop Objects n03257586 duplicator.n.01 39 objects +828 unknown (picture or window) unknown picture/window 2 40 7 otherprop Objects 40 misc +829 stair handle stair handle 2 758 40 7 handle otherprop Objects n04047401 railing.n.01 30 railing +830 reflection reflection 2 64 11 8 picture picture Picture n04068976 reflection.n.05 6 picture +831 horizonal bar for exercise? exercise equipment 2 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +832 shelving / other room shelving /otherroom 2 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +833 seats seat 2 524 39 6 furniture otherfurniture Furniture n04161981 seat.n.03 34 seating +834 shower stall shower stall 2 40 7 otherprop Objects n04209613 shower_stall.n.01 40 misc +835 chairs chair 2 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +836 stair /otherroom stair /otherroom 2 215 38 7 stairs otherstructure Objects stairs n04314914 step.n.04 16 stairs +837 kitchen stuff clutter 2 40 7 otherprop Objects 40 misc +838 unkown object object 2 40 7 otherprop Objects n00002684 object.n.01 39 objects +839 throne throne 2 5 5 4 chair chair Chair chair chair chair 03001627 n04429376 throne.n.01 3 chair +840 socket socket 2 40 7 otherprop Objects n04255163 socket.n.02 39 objects +841 bathroom art bathroom art 2 40 7 otherprop Objects n02743547 art.n.01 39 objects +842 night table night table 2 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +843 display display 2 40 7 otherprop Objects n03211117 display.n.06 22 tv_monitor +844 tabletop tabletop 2 19 7 10 table table Table n04381860 tabletop.n.01 39 objects +845 thrash bin trash bin 2 307 40 7 bin otherprop Objects trash_bin 02747177 n02747177 ashcan.n.01 39 objects +846 l shape sofa l-shaped sofa 2 83 6 9 sofa sofa Sofa sofa sofa sofa 04256520 n04256520 sofa.n.01 10 sofa +847 cardboard box cardboard box 2 26 29 7 box box Objects 40 misc +848 wall borader wall board 1 408 38 7 board otherstructure Objects 40 misc +849 kitchen appliances kitchen appliance 1 40 7 otherprop Objects n03620052 kitchen_appliance.n.01 37 appliances +850 ceiling object object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +851 wall hanging decoration wall hanging decoration 1 40 7 otherprop Objects 40 misc +852 stand / small table stand/small table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +853 kitchen ceiling kitchen ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +854 floor matt floor mat 1 143 20 5 floor mat floor mat Floor n03727837 mat.n.01 2 floor +855 wall indent wall indent 1 40 7 otherprop Objects 40 misc +856 chairir chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +857 window seat window seat 1 777 38 7 window seat otherstructure Objects bench bench 02828884 n04590021 window_seat.n.01 34 seating +858 bike bicycle 1 189 40 7 bicycle otherprop Objects bicycle 02834778 n02834778 bicycle.n.01 39 objects +859 towels (rolled) towel 1 135 27 7 towel towel Objects n04459362 towel.n.01 20 towel +860 wall \other oom wall /otherroom 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +861 shower pipe shower pipe 1 664 40 7 shower pipe otherprop Objects 40 misc +862 towel or curtain bar towel/curtain bar 1 51 38 7 bar otherstructure Objects 40 misc +863 shower glass shower glass 1 612 38 7 glass otherstructure Objects 40 misc +864 stone bench stone bench 1 204 39 6 bench otherfurniture Furniture bench bench 02828884 n02828884 bench.n.01 34 seating +865 window/other room window /otherroom 1 59 9 13 window window Window n04587648 window.n.01 9 window +866 back splash sink sink 1 24 34 7 sink sink Objects sink n04223580 sink.n.01 15 sink +867 iron board iron board 1 408 38 7 board otherstructure Objects 40 misc +868 computer equipment computer equipment 1 40 7 otherprop Objects 40 misc +869 shelf / cabinet shelf/cabinet 1 40 7 otherprop Objects 40 misc +870 stove door stove door 1 28 8 12 door door Wall door 40 misc +871 door inside door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +872 unknown stove accessory unknown stove accessory 1 40 7 otherprop Objects 40 misc +873 circular sofa circular sofa 1 83 6 9 sofa sofa Sofa sofa sofa sofa 04256520 n04256520 sofa.n.01 10 sofa +874 dustpan dustpan 1 40 7 otherprop Objects n03259009 dustpan.n.02 39 objects +875 bathroom door door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +876 paining painting 1 64 11 8 picture picture Picture n03876519 painting.n.01 39 objects +877 luggage luggage 1 783 40 7 luggage otherprop Objects n02774630 baggage.n.01 39 objects +878 dooframe doorframe 1 615 38 7 door frame otherstructure Objects n03222722 doorframe.n.01 4 door +879 outside wall wall 2 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +880 alarm control alarm control 1 40 7 otherprop Objects 40 misc +881 oil lamp oil lamp 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +882 scaffolding scaffolding 1 40 7 otherprop Objects n04141712 scaffolding.n.01 39 objects +883 bed light bed light 1 62 38 7 light otherstructure Objects 40 misc +884 baluster baluster 1 453 38 7 banister otherstructure Objects n02783994 baluster.n.01 39 objects +885 leg rest leg rest 1 40 7 otherprop Objects 40 misc +886 ceiling / upstairs room ceiling /otheroom 1 40 7 otherprop Objects 40 misc +887 tv cabinet tv stand 1 291 39 6 tv stand otherfurniture Furniture tv_stand n03290653 entertainment_center.n.01 36 furniture +888 hole hole 1 40 7 otherprop Objects n09304750 hole.n.05 39 objects +889 ping pong table ping pong table 1 625 39 6 ping pong table otherfurniture Furniture table table table 04379243 n04379243 table.n.02 5 table +890 hutch hutch 1 40 7 otherprop Objects 40 misc +891 low shelf shelf 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +892 foliage foliage 1 40 7 otherprop Objects 40 misc +893 windows sill window frame 1 59 9 13 window window Window n04589593 window_frame.n.01 9 window +894 toilet cistern toilet 1 124 33 7 toilet toilet Objects toilet toilet n04446276 toilet.n.01 18 toilet +895 stone circle stone circle 1 40 7 otherprop Objects 40 misc +896 bathroom sink sink 1 24 34 7 sink sink Objects sink n04223580 sink.n.01 15 sink +897 record player record player 1 220 40 7 machine otherprop Objects n04064401 record_player.n.01 39 objects +898 table cushion table cushion 1 40 7 otherprop Objects 40 misc +899 power outlet outlet 1 40 7 otherprop Objects n04548771 wall_socket.n.01 39 objects +900 machine machine 1 220 40 7 machine otherprop Objects n03699975 machine.n.01 39 objects +901 door post doorpost 1 40 7 otherprop Objects n03222857 doorjamb.n.01 39 objects +902 briefcase briefcase 1 617 40 7 briefcase otherprop Objects n02900705 briefcase.n.01 39 objects +903 wall of doors door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +904 showe wall shower wall 1 21 1 12 wall wall Wall n04208936 shower.n.01 23 shower +905 door stand door stand 1 50 39 6 stand otherfurniture Furniture 40 misc +906 energy box energy box 1 26 29 7 box box Objects 40 misc +907 balcony floor floor 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +908 bean bag beanbag 1 797 39 6 bean bag otherfurniture Furniture n02816656 beanbag.n.01 3 chair +909 kitchen extractor kitchen extractor 1 40 7 otherprop Objects 40 misc +910 toilet bin toilet bin 1 307 40 7 bin otherprop Objects 40 misc +911 wall in other room wall /otherroom 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +912 plumbing plumbing 1 40 7 otherprop Objects n03969041 plumbing.n.01 39 objects +913 moose head / sculpture / hunting trophy moose head/sculpture/hunting trophy 1 547 40 7 trophy otherprop Objects 40 misc +914 ceiling dome ceiling dome 1 40 7 otherprop Objects 40 misc +915 cabinet counter cabinet counter 1 7 12 6 counter counter Furniture 40 misc +916 cabinet \other room cabinet /otherroom 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +917 flowerbed flowerbed 1 157 4 1 bed bed Bed bed bed n03368352 flowerbed.n.01 39 objects +918 paingitn painting 1 64 11 8 picture picture Picture n03876519 painting.n.01 39 objects +919 bed headboard headboard 1 161 39 6 headboard otherfurniture Furniture n03502200 headboard.n.01 11 bed +920 antique clock antique clock 1 56 40 7 clock otherprop Objects clock 03046257 n03046257 clock.n.01 39 objects +921 rocks rock 1 40 7 otherprop Objects n09416076 rock.n.01 39 objects +922 shower caddy for soap etc. shower caddy 1 40 7 otherprop Objects 40 misc +923 window / frame window frame 1 59 9 13 window window Window n04589593 window_frame.n.01 9 window +924 media console media console 1 40 7 otherprop Objects 40 misc +925 cloths cloth 1 40 7 otherprop Objects n03309808 fabric.n.01 39 objects +926 vaccum cleaner vacuum cleaner 1 306 40 7 vacuum cleaner otherprop Objects n04517823 vacuum.n.04 37 appliances +927 door frame fireplace wall door frame 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +928 fruit dish fruit dish 1 40 7 otherprop Objects 40 misc +929 risers for theater seating risers for theater seating 1 40 7 otherprop Objects 40 misc +930 interior bathroom wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +931 hall wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +932 drawer sink table drawer sink table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +933 keyboard piano keyboard piano 1 298 39 6 piano otherfurniture Furniture piano piano 03928116 n03928116 piano.n.01 39 objects +934 tree branch tree branch 1 40 7 otherprop Objects n13163803 limb.n.02 39 objects +935 tiled floor tiled floor 1 11 2 5 floor floor Floor 40 misc +936 ceiling bedroom ceiling bedroom 1 40 7 otherprop Objects 40 misc +937 chandellier chandelier 1 342 38 7 chandelier otherstructure Objects n03005285 chandelier.n.01 28 lighting +938 ceilin ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +939 hearst hearst 1 40 7 otherprop Objects n11037278 hearst.n.01 39 objects +940 random wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +941 condiments condiment 1 40 7 otherprop Objects 40 misc +942 shelf cubbies shelf cubby 1 40 7 otherprop Objects 40 misc +943 book display book display 1 40 7 otherprop Objects 40 misc +944 endtable end table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +945 antique telephone antique telephone 1 32 40 7 telephone otherprop Objects telephone 04401088 n04401088 telephone.n.01 39 objects +946 clothes rack clothes rack 1 40 7 otherprop Objects 40 misc +947 doore door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +948 heater piping heater piping 1 40 7 otherprop Objects 40 misc +949 tissue box stand tissue box stand 1 50 39 6 stand otherfurniture Furniture 40 misc +950 utencils utensil 1 267 40 7 utensil otherprop Objects n04516672 utensil.n.01 39 objects +951 foodtray food tray 1 179 40 7 tray otherprop Objects n04476259 tray.n.01 39 objects +952 shelves /otherroom shelves /otherroom 1 42 15 6 shelves shelves Furniture 40 misc +953 cailing ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +954 cluttered chair chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +955 countertop object object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +956 art frames art frame 1 40 7 otherprop Objects 40 misc +957 soap bottle soap bottle 1 2 40 7 bottle otherprop Objects bottle bottle 02876657 n02876657 bottle.n.01 39 objects +958 small wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +959 watch watch 1 384 40 7 watch otherprop Objects n04555897 watch.n.01 39 objects +960 ceil rail 1 497 38 7 railing otherstructure Objects n04047401 railing.n.01 30 railing +961 fuse box fuse box 1 26 29 7 box box Objects 40 misc +962 painting board painting 1 64 11 8 picture picture Picture n03876519 painting.n.01 39 objects +963 door locker handle door locker handle 1 758 40 7 handle otherprop Objects 40 misc +964 box with clutter box 1 26 29 7 box box Objects n02883344 box.n.01 39 objects +965 knive holder knife holder 1 40 7 otherprop Objects 40 misc +966 computer mouse computer mouse 1 103 40 7 mouse otherprop Objects n03793489 mouse.n.04 39 objects +967 mirror / other room mirror /otherroom 1 122 19 7 mirror mirror Objects n03773035 mirror.n.01 21 mirror +968 door back side door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +969 ceiling bath ceiling bath 1 40 7 otherprop Objects 40 misc +970 chandelier? chandelier 1 342 38 7 chandelier otherstructure Objects n03005285 chandelier.n.01 28 lighting +971 table tray table tray 1 179 40 7 tray otherprop Objects 40 misc +972 aquarium aquarium 1 263 40 7 vessel otherprop Objects n02732072 aquarium.n.01 40 misc +973 wheelbarrow wheelbarrow 1 305 40 7 cart otherprop Objects n02797295 barrow.n.03 39 objects +974 curtain rail curtain rail 1 40 7 otherprop Objects 40 misc +975 rods / table rods/table 1 40 7 otherprop Objects 40 misc +976 counter /otherroom counter /otherroom 1 7 12 6 counter counter Furniture table table table 04379243 n03116530 counter.n.01 26 counter +977 gable gable 1 21 1 12 wall wall Wall n03409393 gable.n.01 1 wall +978 bothroom wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +979 balustrade balustrade 1 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +980 bathroom bathroom 1 40 7 otherprop Objects toilet toilet 40 misc +981 three three 1 40 7 otherprop Objects 40 misc +982 handcloth hand cloth 1 40 7 otherprop Objects 40 misc +983 wall curved wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +984 teapoy coffee table 1 356 39 6 coffee table otherfurniture Furniture table table table 04379243 n03063968 coffee_table.n.01 5 table +985 table with clutter table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +986 back door door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +987 rocky ground rocky ground 1 40 7 otherprop Objects 40 misc +988 doormat doormat 2 143 20 5 floor mat floor mat Floor n03223299 doormat.n.02 2 floor +989 backrest backrest 1 5 5 4 chair chair Chair n02767433 back.n.08 39 objects +990 cabinet with desk cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +991 faucet handles faucet 1 9 40 7 faucet otherprop Objects faucet 03325088 n03325088 faucet.n.01 39 objects +992 floor trim floor trim 1 868 38 7 floor trim otherstructure Objects 40 misc +993 toilet sliding door toilet sliding door 1 28 8 12 door door Wall door 40 misc +994 gym stuff gym equipment 1 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +995 clothes container clothes container 1 140 40 7 container otherprop Objects 40 misc +996 basketball hoop basketball hoop 1 162 40 7 basketball hoop otherprop Objects n02802215 basket.n.03 33 gym_equipment +997 cooktop stovetop 1 40 7 otherprop Objects 40 misc +998 both tub bathtub 1 136 36 7 bathtub bathtub Objects bathtub bathtub tub 02808440 n02808440 bathtub.n.01 25 bathtub +999 fireplace heart fireplace 1 372 38 7 fireplace otherstructure Objects n03346455 fireplace.n.01 27 fireplace +1000 spice racks spice rack 1 241 38 7 spice rack otherstructure Objects n04275175 spice_rack.n.01 31 shelving +1001 wall fireplace wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1002 cabinet /w cluttered art cabinet /w cluttered art 1 40 7 otherprop Objects 40 misc +1003 inside wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1004 transformer transformer 1 40 7 otherprop Objects n04471315 transformer.n.01 39 objects +1005 wall light wall lamp 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +1006 bathroom shelf bathroom shelf 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +1007 glass french door door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1008 lamp / other room lamp /otherroom 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +1009 picutre picture 1 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +1010 gift gift 1 40 7 otherprop Objects 40 misc +1011 mirror door mirror door 1 28 8 12 door door Wall door 40 misc +1012 deorative object decoration 1 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +1013 ceiling door ceiling door 1 28 8 12 door door Wall door 40 misc +1014 stack of papers stack of papers 1 483 40 7 papers otherprop Objects 40 misc +1015 holy cross holy cross 1 40 7 otherprop Objects 40 misc +1016 door window door window 1 59 9 13 window window Window 40 misc +1017 computer screen monitor 1 49 40 7 monitor otherprop Objects monitor monitor tv or monitor 03211117 n03782190 monitor.n.04 22 tv_monitor +1018 while bottles bottle 1 2 40 7 bottle otherprop Objects bottle bottle 02876657 n02876657 bottle.n.01 39 objects +1019 arcade game arcade game 1 40 7 otherprop Objects 40 misc +1020 unknown - probably part of trellis -- maybe blinds unknown - probably part of trellis -- maybe blinds 1 80 13 13 blinds blinds Window 40 misc +1021 compound wall compound wall 1 21 1 12 wall wall Wall 40 misc +1022 lamps lamp 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +1023 plug plug 1 40 7 otherprop Objects 40 misc +1024 round cushion round cushion 1 40 7 otherprop Objects 40 misc +1025 ceiling frame molding 1 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +1026 magazine magazine 1 71 40 7 magazine otherprop Objects n06595351 magazine.n.01 39 objects +1027 stair case staircase 1 215 38 7 stairs otherstructure Objects n04298308 stairway.n.01 16 stairs +1028 rolling pin rolling pin 1 267 40 7 utensil otherprop Objects n04103206 rolling_pin.n.01 39 objects +1029 shower knob shower knob 1 651 40 7 shower knob otherprop Objects 40 misc +1030 wall statue wall statue 1 40 7 otherprop Objects 40 misc +1031 wal wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1032 sink / basin sink/basin 1 40 7 otherprop Objects 40 misc +1033 celling borader ceiling boarder 1 40 7 otherprop Objects 40 misc +1034 clothes bag clothes bag 1 55 37 7 bag bag Objects 40 misc +1035 perfumes perfume 1 655 40 7 perfume otherprop Objects n03916031 perfume.n.02 39 objects +1036 heat heat 1 40 7 otherprop Objects n03509025 heating_system.n.01 39 objects +1037 kitchen counter support leg kitchen counter support 1 40 7 otherprop Objects 40 misc +1038 window frame window frame 1 59 9 13 window window Window n04589593 window_frame.n.01 9 window +1039 glassdoor door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1040 window closet door window closet door 1 28 8 12 door door Wall door 40 misc +1041 sticks stick 1 529 40 7 stick otherprop Objects 40 misc +1042 photo frame picture frame 1 64 11 8 picture picture Picture n03931765 picture_frame.n.01 6 picture +1043 water pump water pump 1 40 7 otherprop Objects n04561965 water_pump.n.01 39 objects +1044 shower doorframe shower door frame 1 40 7 otherprop Objects n04208936 shower.n.01 23 shower +1045 stairs behind stairs 1 215 38 7 stairs otherstructure Objects stairs n04314914 step.n.04 16 stairs +1046 flood floor 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1047 columned perimeter columned perimeter 1 40 7 otherprop Objects 40 misc +1048 shrine shrine 1 40 7 otherprop Objects n04210390 shrine.n.01 40 misc +1049 papers paper 1 15 26 7 paper paper Objects n14974264 paper.n.01 39 objects +1050 fireplace /w art fireplace 1 372 38 7 fireplace otherstructure Objects n03346455 fireplace.n.01 27 fireplace +1051 canvas stand canvas stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1052 art / man statue art/man statue 1 40 7 otherprop Objects 40 misc +1053 bath towel bath towel 1 135 27 7 towel towel Objects n02808304 bath_towel.n.01 20 towel +1054 cradenza credenza 1 7 12 6 counter counter Furniture n03129753 credenza.n.01 36 furniture +1055 doorr frame door frame 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1056 column statue column statue 1 40 7 otherprop Objects 40 misc +1057 water basin water basin 1 40 7 otherprop Objects 40 misc +1058 moulidng molding 1 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +1059 tablet computer tablet computer 1 46 40 7 computer otherprop Objects 40 misc +1060 artwork artwork 1 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +1061 towel bar shelf towel bar shelf 1 40 7 otherprop Objects 40 misc +1062 floor carpet carpet 1 130 40 7 rug floor mat Objects n04118021 rug.n.01 2 floor +1063 wine cabinet wine cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1064 shower rail shower rail 1 40 7 otherprop Objects 40 misc +1065 skirting board skirting board 1 408 38 7 board otherstructure Objects n02800354 baseboard.n.01 1 wall +1066 playpen playpen 1 815 39 6 playpen otherfurniture Furniture n03964495 playpen.n.01 40 misc +1067 room door door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1068 doorgrame doorframe 1 615 38 7 door frame otherstructure Objects n03222722 doorframe.n.01 4 door +1069 makeup makeup 1 40 7 otherprop Objects n03714235 makeup.n.01 39 objects +1070 plant / art plant/art 1 40 7 otherprop Objects 40 misc +1071 sauna seating seat 1 524 39 6 furniture otherfurniture Furniture n04161981 seat.n.03 34 seating +1072 bot bot 1 40 7 otherprop Objects n02311879 bot.n.01 39 objects +1073 handrail / other room handrail /otherroom 1 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +1074 rocking horse rocking horse 1 389 40 7 toy otherprop Objects n03523633 hobby.n.02 39 objects +1075 kitchen lower cabinet kitchen lower cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1076 floor /other room floor /otherroom 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1077 chair stand chair stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1078 wall pack wall pack 1 40 7 otherprop Objects 40 misc +1079 towel box towel box 1 26 29 7 box box Objects 40 misc +1080 vessel sink vessel sink 1 24 34 7 sink sink Objects sink 40 misc +1081 can can 1 329 40 7 can otherprop Objects can 02946921 n02946921 can.n.01 39 objects +1082 ceiling lattice ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1083 backslash backsplash 1 40 7 otherprop Objects 40 misc +1084 ire place fireplace 1 372 38 7 fireplace otherstructure Objects n03346455 fireplace.n.01 27 fireplace +1085 divider partition 1 21 1 12 wall wall Wall n03894379 partition.n.01 40 misc +1086 pathway pathway 1 40 7 otherprop Objects n03899533 pathway.n.02 40 misc +1087 laundry laundry 1 40 7 otherprop Objects n03648219 laundry.n.01 38 clothes +1088 tablecloth tablecloth 1 292 40 7 tablecloth otherprop Objects n04380143 tablecloth.n.01 39 objects +1089 water dispenser water dispenser 1 507 40 7 water dispenser otherprop Objects n03210683 dispenser.n.01 39 objects +1090 ;photo photo 1 508 40 7 photo otherprop Objects n03925226 photograph.n.01 6 picture +1091 door frame wall entry door frame 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1092 plastic tarp tarp 1 40 7 otherprop Objects n04395024 tarpaulin.n.01 39 objects +1093 clothing stand clothing stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1094 bathtab bathtub 1 136 36 7 bathtub bathtub Objects bathtub bathtub tub 02808440 n02808440 bathtub.n.01 25 bathtub +1095 apron apron 1 40 7 otherprop Objects 40 misc +1096 ceiling bedroom entry ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1097 ashtray ashtray 1 377 40 7 ashtray otherprop Objects n02747802 ashtray.n.01 39 objects +1098 control panel control panel 1 408 38 7 board otherstructure Objects n03098140 control_panel.n.01 39 objects +1099 ironing board ironing board 1 313 39 6 ironing board otherfurniture Furniture n03586090 ironing_board.n.01 39 objects +1100 exhaust pipe exhaust pipe 1 41 40 7 pipe otherprop Objects n03303510 exhaust_pipe.n.01 40 misc +1101 rack of theater electronics rack 1 50 39 6 stand otherfurniture Furniture n04038440 rack.n.05 31 shelving +1102 canvas canvas 1 559 40 7 sheet otherprop Objects 40 misc +1103 alarm clock alarm clock 1 156 40 7 alarm clock otherprop Objects clock 03046257 n02694662 alarm_clock.n.01 39 objects +1104 ceiling lower ceiling lower 1 40 7 otherprop Objects 40 misc +1105 yarn machine yarn machine 1 220 40 7 machine otherprop Objects 40 misc +1106 tarrace door terrace door 1 28 8 12 door door Wall door 40 misc +1107 stand/table stand/table 1 40 7 otherprop Objects 40 misc +1108 statue base statue 1 294 40 7 sculpture otherprop Objects n04306847 statue.n.01 39 objects +1109 swing swing 1 389 40 7 toy otherprop Objects n04371774 swing.n.02 39 objects +1110 extractor fan extractor fan 1 74 40 7 fan otherprop Objects 40 misc +1111 crib crib 1 485 39 6 crib otherfurniture Furniture 40 misc +1112 hallway wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1113 side wall 2 wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1114 side wall 5 wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1115 mens urinal urinal 1 40 7 otherprop Objects n04515991 urinal.n.01 39 objects +1116 stacked chairs stacked chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +1117 patio floor patio floor 1 11 2 5 floor floor Floor 40 misc +1118 stair steps stair step 1 40 7 otherprop Objects 40 misc +1119 electrical controller electrical controller 1 40 7 otherprop Objects 40 misc +1120 kitchencounter kitchen counter 1 7 12 6 counter counter Furniture table table table 04379243 n03116530 counter.n.01 26 counter +1121 rotunda railing rotunda railing 1 497 38 7 railing otherstructure Objects 40 misc +1122 lamp table lamp table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +1123 cabinet top cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1124 tray with bottles of soap and lotion tray 1 179 40 7 tray otherprop Objects n04476259 tray.n.01 39 objects +1125 speakers speaker 1 54 40 7 speaker otherprop Objects speaker 03691459 n03691459 loudspeaker.n.01 39 objects +1126 fancy cabinet cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1127 air vent air vent 1 25 38 7 air vent otherstructure Objects n04526241 vent.n.01 40 misc +1128 sofa couch couch 1 83 6 9 sofa sofa Sofa sofa sofa sofa 04256520 n04256520 sofa.n.01 10 sofa +1129 channel channel 1 40 7 otherprop Objects 40 misc +1130 wall sign wall sign 1 208 40 7 sign otherprop Objects 40 misc +1131 bench back bench 1 204 39 6 bench otherfurniture Furniture bench bench 02828884 n02828884 bench.n.01 34 seating +1132 baby changing station baby changing station 1 40 7 otherprop Objects 40 misc +1133 paip pip 1 286 40 7 fruit otherprop Objects n11685091 pip.n.03 39 objects +1134 bed arm pillows pillow 1 119 18 7 pillow pillow Objects pillow 03938244 n03938244 pillow.n.01 8 cushion +1135 shower tap shower tap 1 132 40 7 shower cap otherprop Objects 40 misc +1136 axe axe 1 40 7 otherprop Objects n02764044 ax.n.01 39 objects +1137 table shelf shelf 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +1138 washcloth washcloth 1 40 7 otherprop Objects n04554523 washcloth.n.01 39 objects +1139 coffee mug coffee mug 1 263 40 7 vessel otherprop Objects cup or mug 03797390 n03063599 coffee_mug.n.01 39 objects +1140 iron panel panel 1 559 40 7 sheet otherprop Objects n03882058 panel.n.01 35 board_panel +1141 door or window frame door/window frame 1 40 7 otherprop Objects 40 misc +1142 coffe table coffee table 1 356 39 6 coffee table otherfurniture Furniture table table table 04379243 n03063968 coffee_table.n.01 5 table +1143 dice dice 1 40 7 otherprop Objects n03191029 die.n.01 39 objects +1144 title title 1 40 7 otherprop Objects 40 misc +1145 toilet seat toilet 1 124 33 7 toilet toilet Objects toilet toilet n04446276 toilet.n.01 18 toilet +1146 coffee table leg coffee table 1 356 39 6 coffee table otherfurniture Furniture table table table 04379243 n03063968 coffee_table.n.01 5 table +1147 soft chair soft chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +1148 icebox icebox 1 17 24 6 refridgerator refridgerator Furniture n04070727 refrigerator.n.01 37 appliances +1149 showerfloor shower floor 1 11 2 5 floor floor Floor n04208936 shower.n.01 23 shower +1150 floor outside floor /outside 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1151 framing framing 1 40 7 otherprop Objects n03390983 frame.n.10 40 misc +1152 wall patch wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1153 decorative bowl decorative bowl 1 826 40 7 decorative bowl otherprop Objects bowl bowl 02880940 n02880940 bowl.n.03 39 objects +1154 dog bed dog bed 1 858 39 6 dog bed otherfurniture Furniture bed bed bed 02818832 n02818832 bed.n.01 11 bed +1155 pool sticks pool stick 1 529 40 7 stick otherprop Objects n03145522 cue.n.04 39 objects +1156 exericse equipment exercise equipment 1 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +1157 remove floor behind floor /otherroom 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1158 doorhandle door handle 1 758 40 7 handle otherprop Objects 40 misc +1159 dishrag dishrag 1 40 7 otherprop Objects n03207743 dishrag.n.01 39 objects +1160 behind behind 1 40 7 otherprop Objects 40 misc +1161 drinking fountain drinking fountain 1 40 7 otherprop Objects n03241335 drinking_fountain.n.01 40 misc +1162 bureau bureau 1 524 39 6 furniture otherfurniture Furniture dresser dresser n03015254 chest_of_drawers.n.01 13 chest_of_drawers +1163 night stand object object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1164 tub bathtub 1 136 36 7 bathtub bathtub Objects bathtub bathtub tub 02808440 n02808440 bathtub.n.01 25 bathtub +1165 parapet parapet 1 40 7 otherprop Objects 40 misc +1166 attic door attic door 1 28 8 12 door door Wall door 40 misc +1167 wall object object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1168 closet door knob closet door knob 1 652 40 7 knob otherprop Objects 40 misc +1169 bathroom accessory bathroom accessory 1 40 7 otherprop Objects n02671780 accessory.n.01 38 clothes +1170 teddy teddy bear 1 389 40 7 toy otherprop Objects n04399382 teddy.n.01 39 objects +1171 wall toilet paper wall toilet paper 1 15 26 7 paper paper Objects 40 misc +1172 coffee machine coffee machine 1 234 40 7 coffee machine otherprop Objects n03063338 coffee_maker.n.01 37 appliances +1173 excercise equipment / other room exercise equipment /otherroom 1 457 39 6 excercise equipment otherfurniture Furniture 40 misc +1174 storage cabinet storage cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1175 toilet stall door toilet stall door 1 28 8 12 door door Wall door 40 misc +1176 bin of posters bin 1 307 40 7 bin otherprop Objects n02839910 bin.n.01 39 objects +1177 photos picture 1 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +1178 push broom broom 1 328 40 7 broom otherprop Objects n02906734 broom.n.01 39 objects +1179 art stand art stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1180 shelf /w art shelf /w art 1 40 7 otherprop Objects 40 misc +1181 spiderman spiderman statue 1 40 7 otherprop Objects 40 misc +1182 mount mount 1 40 7 otherprop Objects 40 misc +1183 ac unit air conditioning 1 40 7 otherprop Objects n02686379 air_conditioner.n.01 39 objects +1184 newspaper newspaper 1 873 40 7 newspapers otherprop Objects 40 misc +1185 towel basket towel basket 1 39 40 7 basket otherprop Objects basket 02801938 n02801938 basket.n.01 39 objects +1186 base rail base rail 1 40 7 otherprop Objects 40 misc +1187 show pack show pack 1 40 7 otherprop Objects 40 misc +1188 coutertop countertop 1 7 12 6 counter counter Furniture n03118245 countertop.n.01 26 counter +1189 stair stepper stair stepper 1 40 7 otherprop Objects 40 misc +1190 car batteries car battery 1 40 7 otherprop Objects n02961225 car_battery.n.01 39 objects +1191 pad pad 1 40 7 otherprop Objects n03195485 diggings.n.02 40 misc +1192 old fireplace fireplace 1 372 38 7 fireplace otherstructure Objects n03346455 fireplace.n.01 27 fireplace +1193 desk lamp desk lamp 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +1194 easychair easy chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03262932 easy_chair.n.01 3 chair +1195 wees table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +1196 tabletop box tabletop box 1 26 29 7 box box Objects 40 misc +1197 canopy canopy 1 40 7 otherprop Objects 40 misc +1198 wall behind door wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1199 cabinet / other room cabinet /otherroom 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1200 objects / other room objects /otherroom 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1201 bar soap bar soap 1 133 40 7 soap otherprop Objects 40 misc +1202 column /otherroom column /otherroom 1 94 38 7 column otherstructure Objects n03074380 column.n.06 40 misc +1203 outer wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1204 wall control wall control 1 40 7 otherprop Objects 40 misc +1205 desktop computer tower 1 46 40 7 computer otherprop Objects n03082979 computer.n.01 39 objects +1206 fire screen fire screen 1 40 7 otherprop Objects n03347037 fire_screen.n.01 39 objects +1207 fireplace counter mantel 1 58 38 7 mantel otherstructure Objects mantel n03719343 mantel.n.01 27 fireplace +1208 bread bread 1 246 40 7 bread otherprop Objects n07679356 bread.n.01 40 misc +1209 piano bench piano bench 1 460 39 6 piano bench otherfurniture Furniture bench bench 02828884 n02828884 bench.n.01 34 seating +1210 window shutter window shutter 1 40 7 otherprop Objects n04211356 shutter.n.02 39 objects +1211 draw draw 1 40 7 otherprop Objects n09269882 draw.n.01 39 objects +1212 flower wage flower vase 1 78 40 7 vase otherprop Objects vase jar 03593526 n04522168 vase.n.01 39 objects +1213 ceiling/west wall ceiling/west wall 1 21 1 12 wall wall Wall 40 misc +1214 decorative quilt decorative quilt 1 575 40 7 quilt otherprop Objects 40 misc +1215 knife knife 1 259 40 7 knife otherprop Objects knife 03624134 n03624134 knife.n.02 39 objects +1216 projector opening projector opening 1 40 7 otherprop Objects 40 misc +1217 boader boarder 1 40 7 otherprop Objects 40 misc +1218 lights / deco lights/deco 1 40 7 otherprop Objects 40 misc +1219 tv3 wall tv 1 40 7 otherprop Objects 40 misc +1220 desk clutter desk clutter 1 40 7 otherprop Objects 40 misc +1221 curatain curtain 1 89 16 13 curtain curtain Window curtain n03151077 curtain.n.01 12 curtain +1222 shoe shelves shelving 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +1223 mixer mixer 1 40 7 otherprop Objects 40 misc +1224 ceiling fixture ceiling fixture 1 40 7 otherprop Objects 40 misc +1225 scuplture sculpture 1 294 40 7 sculpture otherprop Objects n04157320 sculpture.n.01 39 objects +1226 countertop /otherroom countertop /otherroom 1 7 12 6 counter counter Furniture n03118245 countertop.n.01 26 counter +1227 work bench workbench 1 204 39 6 bench otherfurniture Furniture bench table 04379243 n04600486 workbench.n.01 5 table +1228 wall desk wall desk 1 36 14 10 desk desk Table desk desk 40 misc +1229 shelf of cloth shelf 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +1230 telescope telescope 1 467 40 7 telescope otherprop Objects n04403638 telescope.n.01 39 objects +1231 shower hoses shower hose 1 669 40 7 shower hose otherprop Objects 40 misc +1232 step; step 1 38 7 otherstructure Objects n04314914 step.n.04 16 stairs +1233 bathtub platform bathtub platform 1 40 7 otherprop Objects 40 misc +1234 sauna seats seat 1 524 39 6 furniture otherfurniture Furniture n04161981 seat.n.03 34 seating +1235 bucker bucket 1 427 40 7 bucket otherprop Objects n02909870 bucket.n.01 39 objects +1236 ac air conditioner 1 79 38 7 air conditioner otherstructure Objects n02686379 air_conditioner.n.01 39 objects +1237 unknown / remove unknown/remove 1 40 7 otherprop Objects 40 misc +1238 art / statue art/statue 1 40 7 otherprop Objects 40 misc +1239 dinner table dinner table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +1240 storage storage 1 n03744276 memory.n.04 39 objects +1241 unknown kitchen stuff unknown kitchen stuff 1 40 7 otherprop Objects 40 misc +1242 alamari wardrobe 1 772 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 36 furniture +1243 cement drum cement drum 1 145 40 7 drum otherprop Objects 40 misc +1244 kitchen cupboard kitchen cabinet 1 3 3 6 cabinet cabinet Furniture n02933112 cabinet.n.01 7 cabinet +1245 platter platter 1 129 40 7 platter otherprop Objects 40 misc +1246 large chunk of art chunk of art 1 40 7 otherprop Objects 40 misc +1247 dushbin dustbin 1 307 40 7 bin otherprop Objects trash_bin 02747177 n02747177 ashcan.n.01 39 objects +1248 couch pillows pillow 1 119 18 7 pillow pillow Objects pillow 03938244 n03938244 pillow.n.01 8 cushion +1249 ceiling /other room ceiling /otherroom 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1250 object on cutting board object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1251 show wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1252 robes robe 1 40 7 otherprop Objects n04097866 robe.n.01 38 clothes +1253 cusion cushion 1 119 18 7 pillow pillow Objects n03151500 cushion.n.03 8 cushion +1254 column /outside column /outside 1 94 38 7 column otherstructure Objects n03074380 column.n.06 40 misc +1255 hanging clothes hanging clothes 1 141 21 7 clothes clothes Objects n02728440 apparel.n.01 38 clothes +1256 newspaper basket newspaper basket 1 39 40 7 basket otherprop Objects basket 02801938 n02801938 basket.n.01 39 objects +1257 towels in a bowl towel 1 135 27 7 towel towel Objects n04459362 towel.n.01 20 towel +1258 detached door door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1259 big door door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1260 wall molding molding 1 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +1261 bed / other room bed /otherroom 1 157 4 1 bed bed Bed bed bed bed 02818832 n02818832 bed.n.01 11 bed +1262 window /w pictures window /w pictures 1 40 7 otherprop Objects 40 misc +1263 ceiling corridor ceiling corridor 1 40 7 otherprop Objects 40 misc +1264 rotunda base rotunda 1 40 7 otherprop Objects 40 misc +1265 fire extinguisher' fire extinguisher 1 10 40 7 fire extinguisher otherprop Objects n03345837 fire_extinguisher.n.01 39 objects +1266 unicycle unicycle 1 40 7 otherprop Objects n04509417 unicycle.n.01 39 objects +1267 ceiling under balcony ceiling /otherroom 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1268 laundary machines laundry machine 1 220 40 7 machine otherprop Objects 40 misc +1269 nightstand /reflection nightstand /reflection 1 158 32 6 night stand night stand Furniture night_stand night_stand n03015254 chest_of_drawers.n.01 13 chest_of_drawers +1270 pile of magazines pile of magazines 1 40 7 otherprop Objects 40 misc +1271 wall of statue inset wall of statue inset 1 40 7 otherprop Objects 40 misc +1272 cosmetic cosmetic 1 40 7 otherprop Objects n03113152 cosmetic.n.01 39 objects +1273 unknown item object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1274 side steps 4 step 1 38 7 otherstructure Objects n04314914 step.n.04 16 stairs +1275 kitchen table kitchen table 1 19 7 10 table table Table table table table 04379243 n03620967 kitchen_table.n.01 5 table +1276 round footstool round footstool 1 40 7 otherprop Objects 40 misc +1277 firewood holder firewood holder 1 40 7 otherprop Objects 40 misc +1278 flag flag 1 405 40 7 flag otherprop Objects 40 misc +1279 window frame and shelves window frame and shelves 1 42 15 6 shelves shelves Furniture 40 misc +1280 towel ring towel ring 1 40 7 otherprop Objects n04460038 towel_ring.n.01 39 objects +1281 window frame /reflection window frame /reflection 1 477 38 7 window frame otherstructure Objects n04589593 window_frame.n.01 9 window +1282 basket of something basket of something 1 40 7 otherprop Objects 40 misc +1283 towel rod towel rod 1 134 38 7 towel rod otherstructure Objects 40 misc +1284 window /reflection window /reflection 1 59 9 13 window window Window n04587648 window.n.01 9 window +1285 paper storage paper storage 1 40 7 otherprop Objects 40 misc +1286 t.v tv 1 172 25 11 television television TV tv or monitor 03211117 n03211117 display.n.06 22 tv_monitor +1287 kitchen bell kitchen bell 1 40 7 otherprop Objects 40 misc +1288 ceiling from another room ceiling /otherroom 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1289 throw blanket throw blanket 1 312 40 7 blanket otherprop Objects 40 misc +1290 wall post wall post 1 40 7 otherprop Objects 40 misc +1291 sink pipe sink pipe 1 41 40 7 pipe otherprop Objects 40 misc +1292 workstation workstation 1 46 40 7 computer otherprop Objects n04603399 workstation.n.01 39 objects +1293 room 1 wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1294 storage space storage space 1 645 38 7 storage space otherstructure Objects n04328946 storage_space.n.01 40 misc +1295 mattress bed 1 157 4 1 bed bed Bed bed bed bed 02818832 n02818832 bed.n.01 11 bed +1296 kitchen handle kitchen handle 1 758 40 7 handle otherprop Objects n03485997 handle.n.01 39 objects +1297 bed side table nightstand 1 158 32 6 night stand night stand Furniture night_stand night_stand n03015254 chest_of_drawers.n.01 13 chest_of_drawers +1298 lookout lookout 1 40 7 otherprop Objects n03688943 lookout.n.03 40 misc +1299 bag of sand bag of sand 1 40 7 otherprop Objects 40 misc +1300 walll wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1301 kitchen countertop object' object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1302 bed rest headboard 1 161 39 6 headboard otherfurniture Furniture n03502200 headboard.n.01 11 bed +1303 furniture 2 furniture 1 524 39 6 furniture otherfurniture Furniture n03405725 furniture.n.01 36 furniture +1304 wall ac vent vent 1 25 38 7 air vent otherstructure Objects n04526241 vent.n.01 40 misc +1305 dumbbells dumbbell 1 40 7 otherprop Objects n03255030 dumbbell.n.01 33 gym_equipment +1306 weight weight 1 40 7 otherprop Objects n04571292 weight.n.02 33 gym_equipment +1307 fixture fixture 1 40 7 otherprop Objects n03354613 fixture.n.01 39 objects +1308 llight light 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +1309 show space show space 1 40 7 otherprop Objects 40 misc +1310 recessed cubby recessed cubby 1 40 7 otherprop Objects 40 misc +1311 oven hood range hood 1 380 38 7 range hood otherstructure Objects range_hood n04053677 range_hood.n.01 39 objects +1312 counter doors counter door 1 28 8 12 door door Wall door 40 misc +1313 roof grill roof grill 1 700 38 7 grill otherstructure Objects 40 misc +1314 shower dial shower dial 1 40 7 otherprop Objects 40 misc +1315 riser riser 1 40 7 otherprop Objects 40 misc +1316 plant ornament plant ornament 1 40 7 otherprop Objects 40 misc +1317 floor stand floor stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1318 fruit fruit 1 286 40 7 fruit otherprop Objects n13134947 fruit.n.01 39 objects +1319 frige refrigerator 1 17 24 6 refridgerator refridgerator Furniture n04070727 refrigerator.n.01 37 appliances +1320 stairwell stairwell 1 40 7 otherprop Objects n04298661 stairwell.n.01 16 stairs +1321 trash bag trash bag 1 55 37 7 bag bag Objects 40 misc +1322 cups cup 1 35 40 7 cup otherprop Objects cup cup or mug 03797390 n03797390 mug.n.04 39 objects +1323 photo mount photo mount 1 40 7 otherprop Objects 40 misc +1324 drawers for clothes drawers for clothes 1 141 21 7 clothes clothes Objects 40 misc +1325 strange ceiling ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1326 bilow billow 1 40 7 otherprop Objects 40 misc +1327 ceilng ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1328 bathroom towel bathroom towel 1 135 27 7 towel towel Objects n04459362 towel.n.01 20 towel +1329 john john 1 40 7 otherprop Objects toilet toilet n04446276 toilet.n.01 18 toilet +1330 refrigerator cabinet refrigerator cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1331 fish picture fish picture 1 64 11 8 picture picture Picture 40 misc +1332 painting frame painting frame 1 40 7 otherprop Objects 40 misc +1333 display table display table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +1334 round seat round seat 1 40 7 otherprop Objects 40 misc +1335 lower floor floor 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1336 lid lid 1 533 40 7 lid otherprop Objects 40 misc +1337 bonsai tree bonsai tree 1 40 7 otherprop Objects 40 misc +1338 window3 window 1 59 9 13 window window Window n04587648 window.n.01 9 window +1339 window2 window 1 59 9 13 window window Window n04587648 window.n.01 9 window +1340 window1 window 1 59 9 13 window window Window n04587648 window.n.01 9 window +1341 window5 window 1 59 9 13 window window Window n04587648 window.n.01 9 window +1342 shower floor surround shower floor 1 11 2 5 floor floor Floor n04208936 shower.n.01 23 shower +1343 folders folder 1 69 40 7 folder otherprop Objects n03376279 folder.n.02 39 objects +1344 decorative plant decorative plant 1 82 40 7 plant otherprop Objects plant 40 misc +1345 cooker unit cooker unit 1 40 7 otherprop Objects 40 misc +1346 towel holder towel holder 1 40 7 otherprop Objects 40 misc +1347 laundrybag laundry bag 1 55 37 7 bag bag Objects 40 misc +1348 bathroom utencil bathroom utensil 1 267 40 7 utensil otherprop Objects n04516672 utensil.n.01 39 objects +1349 wine bottles wine bottle 1 333 40 7 wine bottle otherprop Objects bottle wine bottle 04591713 n04591713 wine_bottle.n.01 39 objects +1350 high shelf high shelf 1 40 7 otherprop Objects 40 misc +1351 loveseat couch 1 83 6 9 sofa sofa Sofa sofa sofa sofa 04256520 n04256520 sofa.n.01 10 sofa +1352 couch pillow pillow 1 119 18 7 pillow pillow Objects pillow 03938244 n03938244 pillow.n.01 8 cushion +1353 ceiling molding ceiling molding 1 40 7 otherprop Objects 40 misc +1354 firewook firewood 1 40 7 otherprop Objects n15100644 firewood.n.01 40 misc +1355 door /otherroom door /otherroom 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1356 toaster? toaster 1 251 40 7 toaster otherprop Objects n04442312 toaster.n.02 37 appliances +1357 foodstand food stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1358 weight bench weight bench 1 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +1359 floor / room above floor /otherroom 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1360 mini fridge mini fridge 1 17 24 6 refridgerator refridgerator Furniture n03273913 electric_refrigerator.n.01 37 appliances +1361 bars bar 1 51 38 7 bar otherstructure Objects n02788689 bar.n.03 39 objects +1362 lion lion 1 594 40 7 cat otherprop Objects n02129165 lion.n.01 39 objects +1363 cuddly toy cuddly toy 1 389 40 7 toy otherprop Objects 40 misc +1364 toilet stall toilet stall 1 40 7 otherprop Objects 40 misc +1365 copier machine copier machine 1 220 40 7 machine otherprop Objects 40 misc +1366 raised platform platform 1 38 7 otherstructure Objects 40 misc +1367 bell bell 1 40 7 otherprop Objects 40 misc +1368 fireextinctioms fire extinguisher 1 10 40 7 fire extinguisher otherprop Objects n03345837 fire_extinguisher.n.01 39 objects +1369 table vase table vase 1 78 40 7 vase otherprop Objects vase 40 misc +1370 bedroom ceiling bedroom ceiling 1 4 22 3 ceiling ceiling Ceiling 40 misc +1371 ceiling \other room ceiling /otherroom 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1372 chair parts chair part 1 40 7 otherprop Objects 40 misc +1373 door / window door/window 1 40 7 otherprop Objects 40 misc +1374 closet mirror wall closet mirror wall 1 21 1 12 wall wall Wall 40 misc +1375 beam across top of arch beam across top of arch 1 40 7 otherprop Objects 40 misc +1376 fireplace utensils fireplace utensil 1 267 40 7 utensil otherprop Objects 40 misc +1377 liight light 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +1378 self side self side 1 40 7 otherprop Objects 40 misc +1379 kitchen faucet faucet 1 9 40 7 faucet otherprop Objects faucet 03325088 n03325088 faucet.n.01 39 objects +1380 candle holder candle holder 1 148 40 7 candlestick otherprop Objects n02948557 candlestick.n.01 39 objects +1381 mirror/other room mirror /otherroom 1 122 19 7 mirror mirror Objects n03773035 mirror.n.01 21 mirror +1382 hall top hall top 1 40 7 otherprop Objects 40 misc +1383 decoration / other room decoration /otherroom 1 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +1384 shower hose shower hose 1 669 40 7 shower hose otherprop Objects 40 misc +1385 bottle of detergent bottle of detergent 1 40 7 otherprop Objects 40 misc +1386 box opening box opening 1 40 7 otherprop Objects 40 misc +1387 hunting trohpy hunting trophy 1 547 40 7 trophy otherprop Objects 40 misc +1388 rack with pool cues rack 1 50 39 6 stand otherfurniture Furniture n04038440 rack.n.05 31 shelving +1389 stack stack 1 40 7 otherprop Objects 40 misc +1390 bed stand bed stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1391 planter curb planter curb 1 40 7 otherprop Objects 40 misc +1392 garage door motor garage door motor 1 40 7 otherprop Objects 40 misc +1393 entry entry 1 40 7 otherprop Objects n03290771 entrance.n.01 40 misc +1394 a-frame sign a-frame sign 1 208 40 7 sign otherprop Objects 40 misc +1395 cuboid cuboid 1 40 7 otherprop Objects 40 misc +1396 shelf with jars shelf 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +1397 wall top wall top 1 40 7 otherprop Objects 40 misc +1398 window4 window 1 59 9 13 window window Window n04587648 window.n.01 9 window +1399 bath faucet bath faucet 1 9 40 7 faucet otherprop Objects faucet 03325088 n03325088 faucet.n.01 39 objects +1400 statue / art statue/art 1 40 7 otherprop Objects 40 misc +1401 cabinet /w clutter cabinet /w clutter 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1402 workout bike workout bike 1 40 7 otherprop Objects 40 misc +1403 closet area for hanging clothes closet area for hanging clothes 1 141 21 7 clothes clothes Objects 40 misc +1404 vase \other room vase /otherroom 1 78 40 7 vase otherprop Objects vase jar 03593526 n04522168 vase.n.01 39 objects +1405 art / muscle shell art/muscle shell 1 40 7 otherprop Objects 40 misc +1406 center island island 1 456 38 7 kitchen island otherstructure Objects n03620600 kitchen_island.n.01 26 counter +1407 dedore decor 1 40 7 otherprop Objects n03579355 interior_decoration.n.01 39 objects +1408 dome roof dome roof 1 40 7 otherprop Objects 40 misc +1409 chair /w clutter chair /w clutter 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +1410 basket of objects (popcorn?) basket 1 39 40 7 basket otherprop Objects basket 02801938 n02801938 basket.n.01 39 objects +1411 outer side wall wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1412 top floor floor 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1413 pelt pelt 1 40 7 otherprop Objects n01895735 hide.n.02 39 objects +1414 washer-dryer washer-dryer 1 40 7 otherprop Objects 40 misc +1415 yellow egg shaped vase yellow egg shaped vase 1 78 40 7 vase otherprop Objects vase 40 misc +1416 entry table table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +1417 domino decoration domino decoration 1 40 7 otherprop Objects 40 misc +1418 pc tower pc tower 1 40 7 otherprop Objects tower 04460130 n04460130 tower.n.01 40 misc +1419 unknown/ other room unknown /otherroom 1 20 40 7 unknown otherprop Objects n08632096 unknown.n.01 41 unlabeled +1420 bathroom glass bathroom glass 1 612 38 7 glass otherstructure Objects n03438257 glass.n.02 39 objects +1421 chanellier chandelier 1 342 38 7 chandelier otherstructure Objects n03005285 chandelier.n.01 28 lighting +1422 makeup area makeup area 1 40 7 otherprop Objects 40 misc +1423 fireplace mirror fireplace mirror 1 122 19 7 mirror mirror Objects 40 misc +1424 jug jug 1 687 40 7 jug otherprop Objects bottle bottle 02876657 n03603722 jug.n.01 39 objects +1425 bathroom window bathroom window 1 59 9 13 window window Window n04587648 window.n.01 9 window +1426 falling light falling light 1 62 38 7 light otherstructure Objects 40 misc +1427 snow globe snow globe 1 347 40 7 globe otherprop Objects 40 misc +1428 unknown kitchen appliance unknown kitchen appliance 1 40 7 otherprop Objects 40 misc +1429 boxes /w books boxes /w books 1 85 23 2 books books Books 40 misc +1430 alarm alarm 1 525 40 7 alarm otherprop Objects clock 03046257 n02694662 alarm_clock.n.01 39 objects +1431 electirc outlet electric outlet 1 98 40 7 electrical outlet otherprop Objects n04548771 wall_socket.n.01 39 objects +1432 sauna floor floor 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1433 picture /otherroom picture /otherroom 1 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +1434 garage door railing garage door railing 1 497 38 7 railing otherstructure Objects 40 misc +1435 other ceiling ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1436 lightbox frame light box frame 1 40 7 otherprop Objects 40 misc +1437 air hole vent 1 25 38 7 air vent otherstructure Objects n04526241 vent.n.01 40 misc +1438 mantel mantel 1 58 38 7 mantel otherstructure Objects mantel n03719343 mantel.n.01 27 fireplace +1439 cooker cooker 4 267 40 7 utensil otherprop Objects n03101156 cooker.n.01 39 objects +1440 partial partial 1 40 7 otherprop Objects 40 misc +1441 amplifier amplifier 1 40 7 otherprop Objects n02705944 amplifier.n.01 39 objects +1442 emergency exit emergency exit 1 40 7 otherprop Objects n03345658 fire_escape.n.01 16 stairs +1443 barbecue barbecue 1 40 7 otherprop Objects n02790669 barbecue.n.03 40 misc +1444 unkown / other room unknown /otherroom 1 20 40 7 unknown otherprop Objects n08632096 unknown.n.01 41 unlabeled +1445 bath sink bath sink 1 24 34 7 sink sink Objects sink 40 misc +1446 spa bench spa bench 1 204 39 6 bench otherfurniture Furniture bench bench 02828884 n02828884 bench.n.01 34 seating +1447 handsoap hand soap 1 133 40 7 soap otherprop Objects n04253437 soap.n.01 39 objects +1448 folding stand folding stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1449 table plant table plant 1 82 40 7 plant otherprop Objects plant 40 misc +1450 hose outlet hose outlet 1 40 7 otherprop Objects 40 misc +1451 theater stage theater stage 1 40 7 otherprop Objects n04418818 theater_stage.n.01 39 objects +1452 bar cabinet bar cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1453 washer and dryer washing machine and dryer 1 40 7 otherprop Objects 37 appliances +1454 door trim molding molding 1 38 7 otherstructure Objects n02800354 baseboard.n.01 1 wall +1455 paintings picture 1 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +1456 unknown wall part unknown wall 1 21 1 12 wall wall Wall 40 misc +1457 wooden balcony balcony 1 40 7 otherprop Objects 40 misc +1458 shelves with shoes shelving 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +1459 stair landing landing 1 40 7 otherprop Objects n03638511 landing.n.01 2 floor +1460 rack of excercise weights rack 1 50 39 6 stand otherfurniture Furniture n04038440 rack.n.05 31 shelving +1461 unknown - hot tub? unknown - hot tub? 1 40 7 otherprop Objects 40 misc +1462 wall window window 1 59 9 13 window window Window n04587648 window.n.01 9 window +1463 ceiling smoke detector smoke detector 1 40 7 otherprop Objects 40 misc +1464 couch type chair sofa chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +1465 scales scale 1 639 40 7 scale otherprop Objects n04141975 scale.n.07 39 objects +1466 chair rail chair rail 1 40 7 otherprop Objects 40 misc +1467 kitchen backsplash backsplash 1 40 7 otherprop Objects 40 misc +1468 shelves / other room shelves /otherroom 1 42 15 6 shelves shelves Furniture 40 misc +1469 tools tool 1 40 7 otherprop Objects n04451818 tool.n.01 39 objects +1470 hoses for chemical tank hoses for chemical tank 1 40 7 otherprop Objects 40 misc +1471 wall behind / remove wall /otherroom 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1472 sauna heat rocks sauna heat rocks 1 40 7 otherprop Objects 40 misc +1473 camera camera 1 40 40 7 camera otherprop Objects camera 02942699 n02942699 camera.n.01 39 objects +1474 curtain lighter curtain lighter 1 40 7 otherprop Objects 40 misc +1475 table decor decoration 1 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +1476 brochure brochure 1 69 40 7 folder otherprop Objects n06413889 booklet.n.01 39 objects +1477 flowerwage flower vase 1 78 40 7 vase otherprop Objects vase jar 03593526 n04522168 vase.n.01 39 objects +1478 drill drill 1 40 7 otherprop Objects 40 misc +1479 book stand bookshelf 1 88 10 6 bookshelf bookshelf Furniture bookshelf bookshelf 02871439 n02871439 bookshelf.n.01 31 shelving +1480 china cabinet cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1481 remove / behind remove 1 0 0 0 void void void 0 void +1482 rotunda wall rotunda wall 1 21 1 12 wall wall Wall 40 misc +1483 stone support structure stone support structure 1 40 7 otherprop Objects 40 misc +1484 dresser decor decoration 1 40 7 otherprop Objects n03169390 decoration.n.01 39 objects +1485 roof / other room ceiling /otherroom 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1486 elliptical elliptical 1 457 39 6 excercise equipment otherfurniture Furniture n04285146 sports_equipment.n.01 33 gym_equipment +1487 furnace furnace 1 551 39 6 furnace otherfurniture Furniture n03404449 furnace.n.01 40 misc +1488 foosball table foosball table 1 510 39 6 foosball table otherfurniture Furniture table table table 04379243 n04379243 table.n.02 5 table +1489 maracas maraca 1 40 7 otherprop Objects n03720891 maraca.n.01 39 objects +1490 box of fruit box of fruit 1 286 40 7 fruit otherprop Objects 40 misc +1491 window screen window screen 1 40 7 otherprop Objects n04589890 window_screen.n.01 9 window +1492 door tag door tag 1 218 40 7 tag otherprop Objects 40 misc +1493 fire pit fire pit 1 40 7 otherprop Objects n09280113 fire_pit.n.01 39 objects +1494 box of tissues box of tissues 1 40 7 otherprop Objects 40 misc +1495 portal portal 1 46 40 7 computer otherprop Objects n06359657 portal_site.n.01 39 objects +1496 yoga mat yoga mat 1 205 40 7 yoga mat otherprop Objects 40 misc +1497 toy duck toy duck 1 887 40 7 duck otherprop Objects 40 misc +1498 motorcycle motorcycle 1 40 7 otherprop Objects motorcycle 03790512 n03790512 motorcycle.n.01 39 objects +1499 computer tower computer tower 1 46 40 7 computer otherprop Objects n03082979 computer.n.01 39 objects +1500 frame for door frame for door 1 28 8 12 door door Wall door 40 misc +1501 theater screen theater screen 1 40 7 otherprop Objects 40 misc +1502 swimming pool swimming pool 1 40 7 otherprop Objects n04371225 swimming_pool.n.01 40 misc +1503 shelf with clutter shelf 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +1504 spice rack spice rack 1 241 38 7 spice rack otherstructure Objects n04275175 spice_rack.n.01 31 shelving +1505 bed small bed small 1 40 7 otherprop Objects 40 misc +1506 object/other room object /otherroom 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1507 dinnerware dinnerware 1 40 7 otherprop Objects n03202622 dinnerware.n.01 39 objects +1508 storage shelves shelving 1 42 15 6 shelves shelves Furniture n04190052 shelf.n.01 31 shelving +1509 tanning bed tanning bed 1 157 4 1 bed bed Bed bed bed bed 02818832 n02818832 bed.n.01 11 bed +1510 grid door door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1511 background chair chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +1512 boarder boarder 1 40 7 otherprop Objects 40 misc +1513 closte closet 1 40 7 otherprop Objects n03148324 cupboard.n.01 7 cabinet +1514 doorframe / upstairs room doorframe /otherroom 1 615 38 7 door frame otherstructure Objects n03222722 doorframe.n.01 4 door +1515 rolling cart rolling cart 1 305 40 7 cart otherprop Objects 40 misc +1516 box of tissue box of tissue 1 648 40 7 tissue otherprop Objects 40 misc +1517 door nob door knob 1 27 40 7 door knob otherprop Objects 40 misc +1518 floor \other room floor /otherroom 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1519 table w/ clutter table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +1520 table vase soil table vase soil 1 40 7 otherprop Objects 40 misc +1521 light box light box 1 26 29 7 box box Objects 40 misc +1522 wines wine 1 766 40 7 wine otherprop Objects 40 misc +1523 sliding glass door sliding glass door 1 28 8 12 door door Wall door 40 misc +1524 toilet handle toilet handle 1 758 40 7 handle otherprop Objects 40 misc +1525 bath deco bath deco 1 40 7 otherprop Objects 40 misc +1526 shower hose/head shower hose/head 1 40 7 otherprop Objects 40 misc +1527 shower case shower case 1 851 40 7 case otherprop Objects 40 misc +1528 mortar mortar 1 40 7 otherprop Objects 40 misc +1529 watering can watering can 1 329 40 7 can otherprop Objects can 02946921 n02946921 can.n.01 39 objects +1530 scarf scarf 1 240 40 7 scarf otherprop Objects n04143897 scarf.n.01 38 clothes +1531 softer softer 1 40 7 otherprop Objects 40 misc +1532 soap dish cubby soap dish cubby 1 40 7 otherprop Objects 40 misc +1533 paintintg picture 1 64 11 8 picture picture Picture n03931044 picture.n.01 6 picture +1534 cleaning clutter cleaning clutter 1 40 7 otherprop Objects 40 misc +1535 fireplace tool set fireplace tool set 1 40 7 otherprop Objects 40 misc +1536 table stand table stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1537 shower valve shower valve 1 40 7 otherprop Objects 40 misc +1538 long sofa couch 1 83 6 9 sofa sofa Sofa sofa sofa sofa 04256520 n04256520 sofa.n.01 10 sofa +1539 wall soap shelf wall soap shelf 1 40 7 otherprop Objects 40 misc +1540 ceiling inset for fan ceiling inset for fan 1 74 40 7 fan otherprop Objects 40 misc +1541 beanbag chair beanbag chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +1542 door bottom rail door bottom rail 1 40 7 otherprop Objects 40 misc +1543 sowing machine sowing machine 1 220 40 7 machine otherprop Objects 40 misc +1544 couch chair sofa chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +1545 tabletop trinket tabletop trinket 1 844 40 7 trinket otherprop Objects 40 misc +1546 ceilling ceiling 3 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1547 flowerstand flower stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1548 wall stand wall stand 1 295 38 7 wall stand otherstructure Objects 40 misc +1549 shower cabin shower cabin 1 40 7 otherprop Objects 40 misc +1550 shower-bath cabinet shower-bath cabinet 1 3 3 6 cabinet cabinet Furniture cabinet 02933112 n02933112 cabinet.n.01 7 cabinet +1551 teddy bear stuffed animal 1 177 40 7 stuffed animal otherprop Objects n04399382 teddy.n.01 39 objects +1552 foor lamp floor lamp 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03367059 floor_lamp.n.01 28 lighting +1553 water fountain water fountain 1 339 38 7 water fountain otherstructure Objects n03241335 drinking_fountain.n.01 40 misc +1554 elephant sculpture elephant sculpture 1 294 40 7 sculpture otherprop Objects 40 misc +1555 grate grate 1 40 7 otherprop Objects 40 misc +1556 chicken chicken 1 40 7 otherprop Objects n01791625 chicken.n.02 39 objects +1557 bathroom fan bathroom fan 1 74 40 7 fan otherprop Objects n03320046 fan.n.01 39 objects +1558 roomba roomba 1 40 7 otherprop Objects 40 misc +1559 planter pot 1 16 40 7 pot otherprop Objects n03991062 pot.n.04 39 objects +1560 controls control 1 40 7 otherprop Objects n03096960 control.n.09 39 objects +1561 nightstand / other room nightstand /otherroom 1 158 32 6 night stand night stand Furniture night_stand night_stand n03015254 chest_of_drawers.n.01 13 chest_of_drawers +1562 worktop worktop 1 40 7 otherprop Objects 40 misc +1563 steps wall steps wall 1 21 1 12 wall wall Wall 40 misc +1564 door fireplace wall door 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1565 floor / upstairs room floor /otherroom 1 11 2 5 floor floor Floor n03365592 floor.n.01 2 floor +1566 oven and stove oven and stove 1 242 38 7 stove otherstructure Objects stove 04330267 40 misc +1567 chest drawer chest drawer 1 174 39 6 drawer otherfurniture Furniture 40 misc +1568 door outside door /outside 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1569 fireplace \other room fireplace /otherroom 1 372 38 7 fireplace otherstructure Objects n03346455 fireplace.n.01 27 fireplace +1570 hammock hammock 1 157 4 1 bed bed Bed bed bed bed 02818832 n03482252 hammock.n.02 11 bed +1571 dartboard dartboard 1 408 38 7 board otherstructure Objects n03162940 dartboard.n.01 39 objects +1572 fireplace brush fireplace brush 1 40 7 otherprop Objects 40 misc +1573 drum drum 1 145 40 7 drum otherprop Objects n03249569 drum.n.01 39 objects +1574 rotunda ceiling ceiling 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1575 ceiling light fixture connection ceiling light fixture connection 1 40 7 otherprop Objects 40 misc +1576 stair fencing banister 1 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +1577 piano stool piano stool 1 150 40 7 stool otherprop Objects stool n03801880 music_stool.n.01 19 stool +1578 swing door swing door 1 28 8 12 door door Wall door n04371979 swing_door.n.01 4 door +1579 bathtub utencils bathtub utensil 1 267 40 7 utensil otherprop Objects 40 misc +1580 access area access area 1 40 7 otherprop Objects 40 misc +1581 exercise mat roll exercise mat roll 1 40 7 otherprop Objects 40 misc +1582 ceiling duct ceiling duct 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1583 motorbike motorbike 1 40 7 otherprop Objects motorcycle 03790512 n03769722 minibike.n.01 39 objects +1584 floor elevator floor elevator 1 40 7 otherprop Objects 40 misc +1585 kitchen countertop items kitchen countertop items 1 40 7 otherprop Objects 40 misc +1586 entertainment set entertainment set 1 40 7 otherprop Objects 40 misc +1587 shower curtain bar shower curtain bar 1 51 38 7 bar otherstructure Objects 40 misc +1588 bedroom entry walls wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1589 oven vent oven vent 1 40 7 otherprop Objects 40 misc +1590 shower floor /otherroom shower floor /otherroom 1 11 2 5 floor floor Floor n04208936 shower.n.01 23 shower +1591 washing bowl sink 1 24 34 7 sink sink Objects sink n04223580 sink.n.01 15 sink +1592 desert plate desert plate 1 233 40 7 plate otherprop Objects 40 misc +1593 fuse panel fuse panel 1 40 7 otherprop Objects 40 misc +1594 barbers chair barbers chair 1 5 5 4 chair chair Chair chair chair chair 03001627 n03001627 chair.n.01 3 chair +1595 diploma diploma 1 40 7 otherprop Objects 40 misc +1596 wall behind stove wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1597 window6 window 1 59 9 13 window window Window n04587648 window.n.01 9 window +1598 ceiling /otherroom ceiling /otherroom 1 4 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 17 ceiling +1599 ventilation hood ventilation hood 1 40 7 otherprop Objects 40 misc +1600 dirt ground dirt ground 1 40 7 otherprop Objects 40 misc +1601 awning awning 1 40 7 otherprop Objects n02763901 awning.n.01 39 objects +1602 stiarcase step step 1 38 7 otherstructure Objects n04314914 step.n.04 16 stairs +1603 rack of weights rack of weights 1 40 7 otherprop Objects 40 misc +1604 shower tub shower tub 1 675 40 7 shower tube otherprop Objects bathtub bathtub tub 02808440 n02808440 bathtub.n.01 25 bathtub +1605 washing powder washing powder 1 40 7 otherprop Objects 40 misc +1606 toilet cleaner toilet cleaner 1 548 40 7 cleaner otherprop Objects 40 misc +1607 soap displensor shelf in shower soap dispenser shelf in shower 1 40 7 otherprop Objects 40 misc +1608 window or door window/door 1 40 7 otherprop Objects 40 misc +1609 storage boxes storage box 1 26 29 7 box box Objects 40 misc +1610 closet storage area closet storage area 1 40 7 otherprop Objects 40 misc +1611 stationary bike exercise bike 1 457 39 6 excercise equipment otherfurniture Furniture n03302671 exercise_bike.n.01 33 gym_equipment +1612 bedding bedding 1 40 7 otherprop Objects n02820210 bedclothes.n.01 39 objects +1613 stair handrail banister 1 453 38 7 banister otherstructure Objects n02788148 bannister.n.02 30 railing +1614 room divider partition 1 21 1 12 wall wall Wall n03894379 partition.n.01 40 misc +1615 window frames window frame 1 59 9 13 window window Window n04589593 window_frame.n.01 9 window +1616 wall electrics wall electronics 1 40 7 otherprop Objects 40 misc +1617 other step step 1 38 7 otherstructure Objects n04314914 step.n.04 16 stairs +1618 tabletop objects object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1619 foosball game table foosball game table 1 19 7 10 table table Table table table table 04379243 n04379243 table.n.02 5 table +1620 wall showcase wall showcase 1 40 7 otherprop Objects 40 misc +1621 kitchen utencil kitchen utensil 1 267 40 7 utensil otherprop Objects n03621049 kitchen_utensil.n.01 39 objects +1622 towels in a basket towels in a basket 1 39 40 7 basket otherprop Objects basket 02801938 n02801938 basket.n.01 39 objects +1623 lamp stand lamp stand 1 50 39 6 stand otherfurniture Furniture 40 misc +1624 toy giraffe toy giraffe 1 40 7 otherprop Objects 40 misc +1625 big door frame door frame 1 28 8 12 door door Wall door n03221720 door.n.01 4 door +1626 unknown / probably shrubbery unknown/probably shrubbery 1 40 7 otherprop Objects 40 misc +1627 wooden wall paneling paneling 1 21 1 12 wall wall Wall n03882611 paneling.n.01 1 wall +1628 bedstead bedstead 1 157 4 1 bed bed Bed n02822579 bedstead.n.01 11 bed +1629 fire dish fire dish 1 40 7 otherprop Objects 40 misc +1630 tray with tea cups tray 1 179 40 7 tray otherprop Objects n04476259 tray.n.01 39 objects +1631 television table tv stand 1 291 39 6 tv stand otherfurniture Furniture tv_stand n03290653 entertainment_center.n.01 36 furniture +1632 tissues tissue 1 648 40 7 tissue otherprop Objects 40 misc +1633 sitting area sitting area 1 40 7 otherprop Objects 40 misc +1634 wall vent vent 1 25 38 7 air vent otherstructure Objects n04526241 vent.n.01 40 misc +1635 utensils utensil 1 267 40 7 utensil otherprop Objects n04516672 utensil.n.01 39 objects +1636 pip pip 1 286 40 7 fruit otherprop Objects n11685091 pip.n.03 39 objects +1637 fireplace wall fireplace wall 1 21 1 12 wall wall Wall 40 misc +1638 stonework stonework 1 40 7 otherprop Objects n04326799 stonework.n.01 40 misc +1639 bottom of stairs bottom of stairs 1 215 38 7 stairs otherstructure Objects stairs 40 misc +1640 star star 1 40 7 otherprop Objects n09444783 star.n.03 39 objects +1641 art / deer statue statue 1 294 40 7 sculpture otherprop Objects n04306847 statue.n.01 39 objects +1642 unknown objects object 1 40 7 otherprop Objects n00002684 object.n.01 39 objects +1643 wall side wall 1 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1644 shower glass /shifted shower glass /shifted 1 612 38 7 glass otherstructure Objects 40 misc +1645 toilet stall partition partition 1 21 1 12 wall wall Wall n03894379 partition.n.01 40 misc +1646 ceiling design 1 40 7 otherprop Objects 40 misc +1647 conference table conference table 1 19 7 10 table table Table table table table 04379243 n03090000 conference_table.n.01 5 table +1648 edge edge 1 40 7 otherprop Objects 40 misc +1649 electrical box electrical box 1 26 29 7 box box Objects n03034244 circuit_breaker.n.01 39 objects +1650 entrance entrance 1 40 7 otherprop Objects n03290771 entrance.n.01 40 misc +1651 gap gap 2 40 7 otherprop Objects n09249034 col.n.01 39 objects +1652 garage door opener motor garage door opener motor 1 40 7 otherprop Objects 40 misc +1653 lamp base lamp 1 144 35 7 lamp lamp Objects lamp lamp 03636649 n03636649 lamp.n.02 28 lighting +1654 racks rack 2 50 39 6 stand otherfurniture Furniture n04038440 rack.n.05 31 shelving +1655 stringer stringer 1 40 7 otherprop Objects 40 misc +1656 supporting structure supporting structure 1 40 7 otherprop Objects n04361095 supporting_structure.n.01 40 misc +1657 typewriter typewriter 1 376 40 7 typewriter otherprop Objects printer 04004475 n04505036 typewriter.n.01 39 objects +1658 walls wall 4 21 1 12 wall wall Wall n04546855 wall.n.01 1 wall +1659 washbasin top washbasin 1 24 34 7 sink sink Objects sink n04553920 washbasin.n.01 15 sink diff --git a/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_test.txt b/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_test.txt new file mode 100644 index 0000000000000000000000000000000000000000..e378f66e2ed767778f9e62826da0ddac058c1682 --- /dev/null +++ b/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_test.txt @@ -0,0 +1,18 @@ +2t7WUuJeko7 +5ZKStnWn8Zo +ARNzJeq3xxb +fzynW3qQPVF +jtcxE69GiFV +pa4otMbVnkk +q9vSo1VnCiC +rqfALeAoiTq +UwV83HsGsw3 +wc2JMjhGNzB +WYY7iVyf5p8 +YFuZgdQ5vWj +yqstnuAEVhm +YVUC4YcDtcY +gxdoqLR6rwA +gYvKGZ5eRqb +RPmz2sHmrrY +Vt2qJdWjCF2 \ No newline at end of file diff --git a/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_train.txt b/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_train.txt new file mode 100644 index 0000000000000000000000000000000000000000..64afe8d80d754176a0b6c68791da55b16ea36c56 --- /dev/null +++ b/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_train.txt @@ -0,0 +1,61 @@ +17DRP5sb8fy +1LXtFkjw3qL +1pXnuDYAj8r +29hnd4uzFmX +5LpN3gDmAk7 +5q7pvUzZiYa +759xd9YjKW5 +7y3sRwLe3Va +82sE5b5pLXE +8WUmhLawc2A +aayBHfsNo7d +ac26ZMwG7aT +B6ByNegPMKs +b8cTxDM8gDG +cV4RVeZvu5T +D7N2EKCX4Sj +e9zR4mvMWw7 +EDJbREhghzL +GdvgFV5R1Z5 +gTV8FGcVJC9 +HxpKQynjfin +i5noydFURQK +JeFG25nYj2p +JF19kD82Mey +jh4fc5c5qoQ +kEZ7cmS4wCh +mJXqzFtmKg4 +p5wJjkQkbXX +Pm6F8kyY3z2 +pRbA3pwrgk9 +PuKPg4mmafe +PX4nDJXEHrG +qoiz87JEwZ2 +rPc6DW4iMge +s8pcmisQ38h +S9hNv5qa7GM +sKLMLpTHeUy +SN83YJsR3w2 +sT4fr6TAbpF +ULsKaCPVFJR +uNb9QFRL6hY +Uxmj2M2itWa +V2XKFyX4ASd +VFuaQ6m2Qom +VVfe2KiqLaN +Vvot9Ly1tCj +vyrNrziPKCB +VzqfbhrpDEA +XcA2TqTSSAj +2n8kARJN3HM +D7G3Y4RVNrH +dhjEzFoUFzH +E9uDoFAP3SH +gZ6f7yhEvPG +JmbYfDe2QKZ +r1Q1Z4BcV1o +r47D5H71a5s +ur6pFq6Qu1A +VLzqgDo317F +YmJkqBEsHnH +ZMojNkEp431 \ No newline at end of file diff --git a/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_val.txt b/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_val.txt new file mode 100644 index 0000000000000000000000000000000000000000..bcec005da85cc9f481208f9730a23da9bb9acc96 --- /dev/null +++ b/pointcept/datasets/preprocessing/matterport3d/meta_data/scenes_val.txt @@ -0,0 +1,11 @@ +2azQ1b91cZZ +8194nk5LbLH +EU6Fwq7SyZv +oLBMNvg9in8 +QUCTc6BB5sX +TbHJrupSAjP +X7HyMhZNoso +pLe4wQe7qrG +x8F5xyUWy9e +Z6MFQCViBuw +zsNo4HB9uLZ \ No newline at end of file diff --git a/pointcept/datasets/preprocessing/matterport3d/preprocess_matterport3d_mesh.py b/pointcept/datasets/preprocessing/matterport3d/preprocess_matterport3d_mesh.py new file mode 100644 index 0000000000000000000000000000000000000000..82d2850b73602924e6ce93d23976a35bbe8a2ae7 --- /dev/null +++ b/pointcept/datasets/preprocessing/matterport3d/preprocess_matterport3d_mesh.py @@ -0,0 +1,240 @@ +""" +Preprocessing Script for Matterport3D (Unzipping) +adatpted from https://github.com/pengsongyou/openscene/blob/main/scripts/preprocess/preprocess_3d_matterport.py + +Author: Chongjie Ye (chongjieye@link.cuhk.edu.cn) +Modified by: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import argparse +import glob +import plyfile +import numpy as np +import pandas as pd +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat +from pathlib import Path +import torch + +MATTERPORT_CLASS_REMAP = np.zeros(41) +MATTERPORT_CLASS_REMAP[1] = 1 +MATTERPORT_CLASS_REMAP[2] = 2 +MATTERPORT_CLASS_REMAP[3] = 3 +MATTERPORT_CLASS_REMAP[4] = 4 +MATTERPORT_CLASS_REMAP[5] = 5 +MATTERPORT_CLASS_REMAP[6] = 6 +MATTERPORT_CLASS_REMAP[7] = 7 +MATTERPORT_CLASS_REMAP[8] = 8 +MATTERPORT_CLASS_REMAP[9] = 9 +MATTERPORT_CLASS_REMAP[10] = 10 +MATTERPORT_CLASS_REMAP[11] = 11 +MATTERPORT_CLASS_REMAP[12] = 12 +MATTERPORT_CLASS_REMAP[14] = 13 +MATTERPORT_CLASS_REMAP[16] = 14 +MATTERPORT_CLASS_REMAP[22] = 21 # DIFFERENCE TO SCANNET! +MATTERPORT_CLASS_REMAP[24] = 15 +MATTERPORT_CLASS_REMAP[28] = 16 +MATTERPORT_CLASS_REMAP[33] = 17 +MATTERPORT_CLASS_REMAP[34] = 18 +MATTERPORT_CLASS_REMAP[36] = 19 +MATTERPORT_CLASS_REMAP[39] = 20 + +MATTERPORT_LABELS_21 = ( + "wall", + "floor", + "cabinet", + "bed", + "chair", + "sofa", + "table", + "door", + "window", + "bookshelf", + "picture", + "counter", + "desk", + "curtain", + "refrigerator", + "shower curtain", + "toilet", + "sink", + "bathtub", + "other", + "ceiling", +) +MATTERPORT_ALLOWED_NYU_CLASSES = [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 14, + 16, + 22, + 24, + 28, + 33, + 34, + 36, + 39, +] + + +def handle_process(mesh_path, output_path, mapping, train_scenes, val_scenes): + # Get the scene id and region name from the mesh path + scene_id = Path(mesh_path).parent.parent.name + region_id = Path(mesh_path).stem.removeprefix("region") + data_name = f"{scene_id}_{int(region_id):02d}" + + output_path = Path(output_path) + # Check which split the scene belongs to (train, val, or test) + if scene_id in train_scenes: + output_folder = output_path / "train" / data_name + split = "train" + elif scene_id in val_scenes: + output_folder = output_path / "val" / data_name + split = "val" + else: + output_folder = output_path / "test" / data_name + split = "test" + + # Create the output directory if it doesn't exist + os.makedirs(output_folder, exist_ok=True) + print(f"Processing: {data_name} in {split}") + + # Load the vertex data + with open(mesh_path, "rb") as f: + plydata = plyfile.PlyData.read(f) + vertex_data = plydata["vertex"].data + + # Get the coordinates, colors, and normals from the vertex data + coords = np.vstack([vertex_data["x"], vertex_data["y"], vertex_data["z"]]).T + colors = np.vstack( + [vertex_data["red"], vertex_data["green"], vertex_data["blue"]] + ).T + normals = np.vstack([vertex_data["nx"], vertex_data["ny"], vertex_data["nz"]]).T + + # Load the face data + face_data = plydata["face"].data + category_id = face_data["category_id"] + + # Replace -1 with 0 in category_id + category_id[category_id == -1] = 0 + + # Map the labels according to NYU40ID + mapped_labels = mapping[category_id] + + # Replace labels not in MATTERPORT_ALLOWED_NYU_CLASSES with 0 + mapped_labels[ + np.logical_not(np.isin(mapped_labels, MATTERPORT_ALLOWED_NYU_CLASSES)) + ] = 0 + + # Remap the labels to ScanNet 20 categories + ceiling + remapped_labels = MATTERPORT_CLASS_REMAP[mapped_labels].astype(int) + + # Calculate per-vertex labels + triangles = face_data["vertex_indices"] + vertex_labels = np.zeros((coords.shape[0], 22), dtype=np.int32) + # calculate per-vertex labels + for row_id in range(triangles.shape[0]): + for i in range(3): + vertex_labels[triangles[row_id][i], remapped_labels[row_id]] += 1 + + # Get the most frequent label for each vertex + vertex_labels = np.argmax(vertex_labels, axis=1) + vertex_labels -= 1 + + # Add the vertex labels to the data to be saved + # Prepare the data to be saved + data_dict = dict( + coord=coords.astype("float32"), + color=colors.astype("uint8"), + normal=normals.astype("float32"), + segment=vertex_labels.astype("int16"), + ) + + # Save processed data + for key in data_dict.keys(): + np.save(output_folder / f"{key}.npy", data_dict[key]) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the Matterport3D dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + opt = parser.parse_args() + meta_root = Path(os.path.dirname(__file__)) / "meta_data" + + # Load label map + category_mapping = pd.read_csv( + meta_root / "category_mapping.tsv", + sep="\t", + header=0, + ) + mapping = np.insert( + category_mapping[["nyu40id"]].to_numpy().astype(int).flatten(), 0, 0, axis=0 + ) + + # Load train/val splits + with open(meta_root / "scenes_train.txt") as train_file: + train_scenes = train_file.read().splitlines() + with open(meta_root / "scenes_val.txt") as val_file: + val_scenes = val_file.read().splitlines() + with open(meta_root / "scenes_test.txt") as test_file: + test_scenes = test_file.read().splitlines() + + # Create output directories + os.makedirs(opt.output_root, exist_ok=True) + train_output_dir = os.path.join(opt.output_root, "train") + os.makedirs(train_output_dir, exist_ok=True) + val_output_dir = os.path.join(opt.output_root, "val") + os.makedirs(val_output_dir, exist_ok=True) + test_output_dir = os.path.join(opt.output_root, "test") + os.makedirs(test_output_dir, exist_ok=True) + + # Load scene paths + scene_paths = sorted( + glob.glob( + os.path.join( + opt.dataset_root, "v1", "scans", "*", "region_segmentations", "*.ply" + ) + ) + ) + + # Preprocess data. + pool = ProcessPoolExecutor(max_workers=opt.num_workers) + print("Processing scenes...") + _ = list( + pool.map( + handle_process, + scene_paths, + repeat(opt.output_root), + repeat(mapping), + repeat(train_scenes), + repeat(val_scenes), + ) + ) diff --git a/pointcept/datasets/preprocessing/matterport3d/unzip_matterport3d_region_segmentation.py b/pointcept/datasets/preprocessing/matterport3d/unzip_matterport3d_region_segmentation.py new file mode 100644 index 0000000000000000000000000000000000000000..eee79908262e09aeb428a5e176cf799e8647a01a --- /dev/null +++ b/pointcept/datasets/preprocessing/matterport3d/unzip_matterport3d_region_segmentation.py @@ -0,0 +1,66 @@ +""" +Preprocessing Script for Matterport3D (Unzipping) +modified from official preprocess code. + +Author: Chongjie Ye (chongjieye@link.cuhk.edu.cn) +Modified by: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import argparse +import os +import zipfile +import glob +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat + + +def unzip_file(input_path, output_path): + print(f"Unzipping {input_path} ...") + os.makedirs(os.path.dirname(output_path), exist_ok=True) + with zipfile.ZipFile(input_path, "r") as zip_ref: + zip_ref.extractall(output_path) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description='Unzip all "region_segmentations.zip" files in a directory' + ) + parser.add_argument( + "--dataset_root", + type=str, + help="Path to input directory containing ZIP files", + required=True, + ) + parser.add_argument( + "--output_root", + type=str, + help="Path to output directory for extracted files", + default=None, + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + args = parser.parse_args() + if args.output_root is None: + args.output_root = args.dataset_root + args.output_root = os.path.join(args.output_root, "v1", "scans") + + file_list = glob.glob( + os.path.join(args.dataset_root, "v1", "scans", "*", "region_segmentations.zip") + ) + + # Preprocess data. + print("Unzipping region_segmentations.zip in Matterport3D...") + pool = ProcessPoolExecutor(max_workers=args.num_workers) + _ = list( + pool.map( + unzip_file, + file_list, + repeat(args.output_root), + ) + ) diff --git a/pointcept/datasets/preprocessing/nuscenes/preprocess_nuscenes_info.py b/pointcept/datasets/preprocessing/nuscenes/preprocess_nuscenes_info.py new file mode 100644 index 0000000000000000000000000000000000000000..fa5b1ea854d96b3e76deb12bd3b633a54eba1b00 --- /dev/null +++ b/pointcept/datasets/preprocessing/nuscenes/preprocess_nuscenes_info.py @@ -0,0 +1,606 @@ +""" +Preprocessing Script for nuScenes Informantion +modified from OpenPCDet (https://github.com/open-mmlab/OpenPCDet) + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +from pathlib import Path +import numpy as np +import argparse +import tqdm +import pickle +from functools import reduce +from pyquaternion import Quaternion +from nuscenes.nuscenes import NuScenes +from nuscenes.utils import splits +from nuscenes.utils.geometry_utils import transform_matrix + +map_name_from_general_to_detection = { + "human.pedestrian.adult": "pedestrian", + "human.pedestrian.child": "pedestrian", + "human.pedestrian.wheelchair": "ignore", + "human.pedestrian.stroller": "ignore", + "human.pedestrian.personal_mobility": "ignore", + "human.pedestrian.police_officer": "pedestrian", + "human.pedestrian.construction_worker": "pedestrian", + "animal": "ignore", + "vehicle.car": "car", + "vehicle.motorcycle": "motorcycle", + "vehicle.bicycle": "bicycle", + "vehicle.bus.bendy": "bus", + "vehicle.bus.rigid": "bus", + "vehicle.truck": "truck", + "vehicle.construction": "construction_vehicle", + "vehicle.emergency.ambulance": "ignore", + "vehicle.emergency.police": "ignore", + "vehicle.trailer": "trailer", + "movable_object.barrier": "barrier", + "movable_object.trafficcone": "traffic_cone", + "movable_object.pushable_pullable": "ignore", + "movable_object.debris": "ignore", + "static_object.bicycle_rack": "ignore", +} + + +cls_attr_dist = { + "barrier": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "bicycle": { + "cycle.with_rider": 2791, + "cycle.without_rider": 8946, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "bus": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 9092, + "vehicle.parked": 3294, + "vehicle.stopped": 3881, + }, + "car": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 114304, + "vehicle.parked": 330133, + "vehicle.stopped": 46898, + }, + "construction_vehicle": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 882, + "vehicle.parked": 11549, + "vehicle.stopped": 2102, + }, + "ignore": { + "cycle.with_rider": 307, + "cycle.without_rider": 73, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 165, + "vehicle.parked": 400, + "vehicle.stopped": 102, + }, + "motorcycle": { + "cycle.with_rider": 4233, + "cycle.without_rider": 8326, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "pedestrian": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 157444, + "pedestrian.sitting_lying_down": 13939, + "pedestrian.standing": 46530, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "traffic_cone": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 0, + "vehicle.parked": 0, + "vehicle.stopped": 0, + }, + "trailer": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 3421, + "vehicle.parked": 19224, + "vehicle.stopped": 1895, + }, + "truck": { + "cycle.with_rider": 0, + "cycle.without_rider": 0, + "pedestrian.moving": 0, + "pedestrian.sitting_lying_down": 0, + "pedestrian.standing": 0, + "vehicle.moving": 21339, + "vehicle.parked": 55626, + "vehicle.stopped": 11097, + }, +} + + +def get_available_scenes(nusc): + available_scenes = [] + for scene in nusc.scene: + scene_token = scene["token"] + scene_rec = nusc.get("scene", scene_token) + sample_rec = nusc.get("sample", scene_rec["first_sample_token"]) + sd_rec = nusc.get("sample_data", sample_rec["data"]["LIDAR_TOP"]) + has_more_frames = True + scene_not_exist = False + while has_more_frames: + lidar_path, boxes, _ = nusc.get_sample_data(sd_rec["token"]) + if not Path(lidar_path).exists(): + scene_not_exist = True + break + else: + break + if scene_not_exist: + continue + available_scenes.append(scene) + return available_scenes + + +def get_sample_data(nusc, sample_data_token, selected_anntokens=None): + """ + Returns the data path as well as all annotations related to that sample_data. + Note that the boxes are transformed into the current sensor"s coordinate frame. + Args: + nusc: + sample_data_token: Sample_data token. + selected_anntokens: If provided only return the selected annotation. + + Returns: + + """ + # Retrieve sensor & pose records + sd_record = nusc.get("sample_data", sample_data_token) + cs_record = nusc.get("calibrated_sensor", sd_record["calibrated_sensor_token"]) + sensor_record = nusc.get("sensor", cs_record["sensor_token"]) + pose_record = nusc.get("ego_pose", sd_record["ego_pose_token"]) + + data_path = nusc.get_sample_data_path(sample_data_token) + + if sensor_record["modality"] == "camera": + cam_intrinsic = np.array(cs_record["camera_intrinsic"]) + else: + cam_intrinsic = None + + # Retrieve all sample annotations and map to sensor coordinate system. + if selected_anntokens is not None: + boxes = list(map(nusc.get_box, selected_anntokens)) + else: + boxes = nusc.get_boxes(sample_data_token) + + # Make list of Box objects including coord system transforms. + box_list = [] + for box in boxes: + box.velocity = nusc.box_velocity(box.token) + # Move box to ego vehicle coord system + box.translate(-np.array(pose_record["translation"])) + box.rotate(Quaternion(pose_record["rotation"]).inverse) + + # Move box to sensor coord system + box.translate(-np.array(cs_record["translation"])) + box.rotate(Quaternion(cs_record["rotation"]).inverse) + + box_list.append(box) + + return data_path, box_list, cam_intrinsic + + +def quaternion_yaw(q: Quaternion) -> float: + """ + Calculate the yaw angle from a quaternion. + Note that this only works for a quaternion that represents a box in lidar or global coordinate frame. + It does not work for a box in the camera frame. + :param q: Quaternion of interest. + :return: Yaw angle in radians. + """ + + # Project into xy plane. + v = np.dot(q.rotation_matrix, np.array([1, 0, 0])) + + # Measure yaw using arctan. + yaw = np.arctan2(v[1], v[0]) + + return yaw + + +def obtain_sensor2top( + nusc, sensor_token, l2e_t, l2e_r_mat, e2g_t, e2g_r_mat, sensor_type="lidar" +): + """Obtain the info with RT matric from general sensor to Top LiDAR. + + Args: + nusc (class): Dataset class in the nuScenes dataset. + sensor_token (str): Sample data token corresponding to the + specific sensor type. + l2e_t (np.ndarray): Translation from lidar to ego in shape (1, 3). + l2e_r_mat (np.ndarray): Rotation matrix from lidar to ego + in shape (3, 3). + e2g_t (np.ndarray): Translation from ego to global in shape (1, 3). + e2g_r_mat (np.ndarray): Rotation matrix from ego to global + in shape (3, 3). + sensor_type (str): Sensor to calibrate. Default: "lidar". + + Returns: + sweep (dict): Sweep information after transformation. + """ + sd_rec = nusc.get("sample_data", sensor_token) + cs_record = nusc.get("calibrated_sensor", sd_rec["calibrated_sensor_token"]) + pose_record = nusc.get("ego_pose", sd_rec["ego_pose_token"]) + data_path = str(nusc.get_sample_data_path(sd_rec["token"])) + # if os.getcwd() in data_path: # path from lyftdataset is absolute path + # data_path = data_path.split(f"{os.getcwd()}/")[-1] # relative path + sweep = { + "data_path": data_path, + "type": sensor_type, + "sample_data_token": sd_rec["token"], + "sensor2ego_translation": cs_record["translation"], + "sensor2ego_rotation": cs_record["rotation"], + "ego2global_translation": pose_record["translation"], + "ego2global_rotation": pose_record["rotation"], + "timestamp": sd_rec["timestamp"], + } + l2e_r_s = sweep["sensor2ego_rotation"] + l2e_t_s = sweep["sensor2ego_translation"] + e2g_r_s = sweep["ego2global_rotation"] + e2g_t_s = sweep["ego2global_translation"] + + # obtain the RT from sensor to Top LiDAR + # sweep->ego->global->ego'->lidar + l2e_r_s_mat = Quaternion(l2e_r_s).rotation_matrix + e2g_r_s_mat = Quaternion(e2g_r_s).rotation_matrix + R = (l2e_r_s_mat.T @ e2g_r_s_mat.T) @ ( + np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T + ) + T = (l2e_t_s @ e2g_r_s_mat.T + e2g_t_s) @ ( + np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T + ) + T -= ( + e2g_t @ (np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(l2e_r_mat).T) + + l2e_t @ np.linalg.inv(l2e_r_mat).T + ).squeeze(0) + sweep["sensor2lidar_rotation"] = R.T # points @ R.T + T + sweep["sensor2lidar_translation"] = T + return sweep + + +def fill_trainval_infos( + data_path, nusc, train_scenes, test=False, max_sweeps=10, with_camera=False +): + train_nusc_infos = [] + val_nusc_infos = [] + progress_bar = tqdm.tqdm( + total=len(nusc.sample), desc="create_info", dynamic_ncols=True + ) + + ref_chan = "LIDAR_TOP" # The radar channel from which we track back n sweeps to aggregate the point cloud. + chan = "LIDAR_TOP" # The reference channel of the current sample_rec that the point clouds are mapped to. + + for index, sample in enumerate(nusc.sample): + progress_bar.update() + + ref_sd_token = sample["data"][ref_chan] + ref_sd_rec = nusc.get("sample_data", ref_sd_token) + ref_cs_rec = nusc.get( + "calibrated_sensor", ref_sd_rec["calibrated_sensor_token"] + ) + ref_pose_rec = nusc.get("ego_pose", ref_sd_rec["ego_pose_token"]) + ref_time = 1e-6 * ref_sd_rec["timestamp"] + + ref_lidar_path, ref_boxes, _ = get_sample_data(nusc, ref_sd_token) + + ref_cam_front_token = sample["data"]["CAM_FRONT"] + ref_cam_path, _, ref_cam_intrinsic = nusc.get_sample_data(ref_cam_front_token) + + # Homogeneous transform from ego car frame to reference frame + ref_from_car = transform_matrix( + ref_cs_rec["translation"], Quaternion(ref_cs_rec["rotation"]), inverse=True + ) + + # Homogeneous transformation matrix from global to _current_ ego car frame + car_from_global = transform_matrix( + ref_pose_rec["translation"], + Quaternion(ref_pose_rec["rotation"]), + inverse=True, + ) + info = { + "lidar_path": Path(ref_lidar_path).relative_to(data_path).__str__(), + "lidar_token": ref_sd_token, + "cam_front_path": Path(ref_cam_path).relative_to(data_path).__str__(), + "cam_intrinsic": ref_cam_intrinsic, + "token": sample["token"], + "sweeps": [], + "ref_from_car": ref_from_car, + "car_from_global": car_from_global, + "timestamp": ref_time, + } + if with_camera: + info["cams"] = dict() + l2e_r = ref_cs_rec["rotation"] + l2e_t = (ref_cs_rec["translation"],) + e2g_r = ref_pose_rec["rotation"] + e2g_t = ref_pose_rec["translation"] + l2e_r_mat = Quaternion(l2e_r).rotation_matrix + e2g_r_mat = Quaternion(e2g_r).rotation_matrix + + # obtain 6 image's information per frame + camera_types = [ + "CAM_FRONT", + "CAM_FRONT_RIGHT", + "CAM_FRONT_LEFT", + "CAM_BACK", + "CAM_BACK_LEFT", + "CAM_BACK_RIGHT", + ] + for cam in camera_types: + cam_token = sample["data"][cam] + cam_path, _, camera_intrinsics = nusc.get_sample_data(cam_token) + cam_info = obtain_sensor2top( + nusc, cam_token, l2e_t, l2e_r_mat, e2g_t, e2g_r_mat, cam + ) + cam_info["data_path"] = ( + Path(cam_info["data_path"]).relative_to(data_path).__str__() + ) + cam_info.update(camera_intrinsics=camera_intrinsics) + info["cams"].update({cam: cam_info}) + + sample_data_token = sample["data"][chan] + curr_sd_rec = nusc.get("sample_data", sample_data_token) + sweeps = [] + while len(sweeps) < max_sweeps - 1: + if curr_sd_rec["prev"] == "": + if len(sweeps) == 0: + sweep = { + "lidar_path": Path(ref_lidar_path) + .relative_to(data_path) + .__str__(), + "sample_data_token": curr_sd_rec["token"], + "transform_matrix": None, + "time_lag": curr_sd_rec["timestamp"] * 0, + } + sweeps.append(sweep) + else: + sweeps.append(sweeps[-1]) + else: + curr_sd_rec = nusc.get("sample_data", curr_sd_rec["prev"]) + + # Get past pose + current_pose_rec = nusc.get("ego_pose", curr_sd_rec["ego_pose_token"]) + global_from_car = transform_matrix( + current_pose_rec["translation"], + Quaternion(current_pose_rec["rotation"]), + inverse=False, + ) + + # Homogeneous transformation matrix from sensor coordinate frame to ego car frame. + current_cs_rec = nusc.get( + "calibrated_sensor", curr_sd_rec["calibrated_sensor_token"] + ) + car_from_current = transform_matrix( + current_cs_rec["translation"], + Quaternion(current_cs_rec["rotation"]), + inverse=False, + ) + + tm = reduce( + np.dot, + [ref_from_car, car_from_global, global_from_car, car_from_current], + ) + + lidar_path = nusc.get_sample_data_path(curr_sd_rec["token"]) + + time_lag = ref_time - 1e-6 * curr_sd_rec["timestamp"] + + sweep = { + "lidar_path": Path(lidar_path).relative_to(data_path).__str__(), + "sample_data_token": curr_sd_rec["token"], + "transform_matrix": tm, + "global_from_car": global_from_car, + "car_from_current": car_from_current, + "time_lag": time_lag, + } + sweeps.append(sweep) + + info["sweeps"] = sweeps + + assert len(info["sweeps"]) == max_sweeps - 1, ( + f"sweep {curr_sd_rec['token']} only has {len(info['sweeps'])} sweeps, " + f"you should duplicate to sweep num {max_sweeps - 1}" + ) + + if not test: + # processing gt bbox + annotations = [ + nusc.get("sample_annotation", token) for token in sample["anns"] + ] + + # the filtering gives 0.5~1 map improvement + num_lidar_pts = np.array([anno["num_lidar_pts"] for anno in annotations]) + num_radar_pts = np.array([anno["num_radar_pts"] for anno in annotations]) + mask = num_lidar_pts + num_radar_pts > 0 + + locs = np.array([b.center for b in ref_boxes]).reshape(-1, 3) + dims = np.array([b.wlh for b in ref_boxes]).reshape(-1, 3)[ + :, [1, 0, 2] + ] # wlh == > dxdydz (lwh) + velocity = np.array([b.velocity for b in ref_boxes]).reshape(-1, 3) + rots = np.array([quaternion_yaw(b.orientation) for b in ref_boxes]).reshape( + -1, 1 + ) + names = np.array([b.name for b in ref_boxes]) + tokens = np.array([b.token for b in ref_boxes]) + gt_boxes = np.concatenate([locs, dims, rots, velocity[:, :2]], axis=1) + + assert len(annotations) == len(gt_boxes) == len(velocity) + + info["gt_boxes"] = gt_boxes[mask, :] + info["gt_boxes_velocity"] = velocity[mask, :] + info["gt_names"] = np.array( + [map_name_from_general_to_detection[name] for name in names] + )[mask] + info["gt_boxes_token"] = tokens[mask] + info["num_lidar_pts"] = num_lidar_pts[mask] + info["num_radar_pts"] = num_radar_pts[mask] + + # processing gt segment + segment_path = nusc.get("lidarseg", ref_sd_token)["filename"] + info["gt_segment_path"] = segment_path + + if sample["scene_token"] in train_scenes: + train_nusc_infos.append(info) + else: + val_nusc_infos.append(info) + + progress_bar.close() + return train_nusc_infos, val_nusc_infos + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", required=True, help="Path to the nuScenes dataset." + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where processed information located.", + ) + parser.add_argument( + "--max_sweeps", default=10, type=int, help="Max number of sweeps. Default: 10." + ) + parser.add_argument( + "--with_camera", + action="store_true", + default=False, + help="Whether use camera or not.", + ) + config = parser.parse_args() + + print(f"Loading nuScenes tables for version v1.0-trainval...") + nusc_trainval = NuScenes( + version="v1.0-trainval", dataroot=config.dataset_root, verbose=False + ) + available_scenes_trainval = get_available_scenes(nusc_trainval) + available_scene_names_trainval = [s["name"] for s in available_scenes_trainval] + print("total scene num:", len(nusc_trainval.scene)) + print("exist scene num:", len(available_scenes_trainval)) + assert len(available_scenes_trainval) == len(nusc_trainval.scene) == 850 + + print(f"Loading nuScenes tables for version v1.0-test...") + nusc_test = NuScenes( + version="v1.0-test", dataroot=config.dataset_root, verbose=False + ) + available_scenes_test = get_available_scenes(nusc_test) + available_scene_names_test = [s["name"] for s in available_scenes_test] + print("total scene num:", len(nusc_test.scene)) + print("exist scene num:", len(available_scenes_test)) + assert len(available_scenes_test) == len(nusc_test.scene) == 150 + + train_scenes = splits.train + train_scenes = set( + [ + available_scenes_trainval[available_scene_names_trainval.index(s)]["token"] + for s in train_scenes + ] + ) + test_scenes = splits.test + test_scenes = set( + [ + available_scenes_test[available_scene_names_test.index(s)]["token"] + for s in test_scenes + ] + ) + print(f"Filling trainval information...") + train_nusc_infos, val_nusc_infos = fill_trainval_infos( + config.dataset_root, + nusc_trainval, + train_scenes, + test=False, + max_sweeps=config.max_sweeps, + with_camera=config.with_camera, + ) + print(f"Filling test information...") + test_nusc_infos, _ = fill_trainval_infos( + config.dataset_root, + nusc_test, + test_scenes, + test=True, + max_sweeps=config.max_sweeps, + with_camera=config.with_camera, + ) + + print(f"Saving nuScenes information...") + os.makedirs(os.path.join(config.output_root, "info"), exist_ok=True) + print( + f"train sample: {len(train_nusc_infos)}, val sample: {len(val_nusc_infos)}, test sample: {len(test_nusc_infos)}" + ) + with open( + os.path.join( + config.output_root, + "info", + f"nuscenes_infos_{config.max_sweeps}sweeps_train.pkl", + ), + "wb", + ) as f: + pickle.dump(train_nusc_infos, f) + with open( + os.path.join( + config.output_root, + "info", + f"nuscenes_infos_{config.max_sweeps}sweeps_val.pkl", + ), + "wb", + ) as f: + pickle.dump(val_nusc_infos, f) + with open( + os.path.join( + config.output_root, + "info", + f"nuscenes_infos_{config.max_sweeps}sweeps_test.pkl", + ), + "wb", + ) as f: + pickle.dump(test_nusc_infos, f) diff --git a/pointcept/datasets/preprocessing/partnete/meta_info.json b/pointcept/datasets/preprocessing/partnete/meta_info.json new file mode 100644 index 0000000000000000000000000000000000000000..b2d50d77e150302cb6f2f11c1c8a0b8b33ccb3ef --- /dev/null +++ b/pointcept/datasets/preprocessing/partnete/meta_info.json @@ -0,0 +1,195 @@ +{ + "Bottle": [ + "lid" + ], + "Box": [ + "lid" + ], + "Bucket": [ + "handle" + ], + "Camera": [ + "button", + "lens" + ], + "Cart": [ + "wheel" + ], + "Chair": [ + "arm", + "back", + "leg", + "seat", + "wheel" + ], + "Clock": [ + "hand" + ], + "CoffeeMachine": [ + "button", + "container", + "knob", + "lid" + ], + "Dishwasher": [ + "door", + "handle" + ], + "Dispenser": [ + "head", + "lid" + ], + "Display": [ + "base", + "screen", + "support" + ], + "Door": [ + "frame", + "door", + "handle" + ], + "Eyeglasses": [ + "body", + "leg" + ], + "Faucet": [ + "spout", + "switch" + ], + "FoldingChair": [ + "seat" + ], + "Globe": [ + "sphere" + ], + "Kettle": [ + "lid", + "handle", + "spout" + ], + "Keyboard": [ + "cord", + "key" + ], + "KitchenPot": [ + "lid", + "handle" + ], + "Knife": [ + "blade" + ], + "Lamp": [ + "base", + "body", + "bulb", + "shade" + ], + "Laptop": [ + "keyboard", + "screen", + "shaft", + "touchpad", + "camera" + ], + "Lighter": [ + "lid", + "wheel", + "button" + ], + "Microwave": [ + "display", + "door", + "handle", + "button" + ], + "Mouse": [ + "button", + "cord", + "wheel" + ], + "Oven": [ + "door", + "knob" + ], + "Pen": [ + "cap", + "button" + ], + "Phone": [ + "lid", + "button" + ], + "Pliers": [ + "leg" + ], + "Printer": [ + "button" + ], + "Refrigerator": [ + "door", + "handle" + ], + "Remote": [ + "button" + ], + "Safe": [ + "door", + "switch", + "button" + ], + "Scissors": [ + "blade", + "handle", + "screw" + ], + "Stapler": [ + "body", + "lid" + ], + "StorageFurniture": [ + "door", + "drawer", + "handle" + ], + "Suitcase": [ + "handle", + "wheel" + ], + "Switch": [ + "switch" + ], + "Table": [ + "door", + "drawer", + "leg", + "tabletop", + "wheel", + "handle" + ], + "Toaster": [ + "button", + "slider" + ], + "Toilet": [ + "lid", + "seat", + "button" + ], + "TrashCan": [ + "footpedal", + "lid", + "door" + ], + "USB": [ + "cap", + "rotation" + ], + "WashingMachine": [ + "door", + "button" + ], + "Window": [ + "window" + ] +} \ No newline at end of file diff --git a/pointcept/datasets/preprocessing/partnete/preprocess_partnete.py b/pointcept/datasets/preprocessing/partnete/preprocess_partnete.py new file mode 100644 index 0000000000000000000000000000000000000000..05bc373747154e9390b9903c4b39f6cf68c241b5 --- /dev/null +++ b/pointcept/datasets/preprocessing/partnete/preprocess_partnete.py @@ -0,0 +1,83 @@ +""" +PartNetE Datasets preprocessing + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import shutil +import argparse +import numpy as np +import trimesh +import open3d as o3d + + +def process_folder(target_dir): + ply_path = os.path.join(target_dir, "pc.ply") + label_path = os.path.join(target_dir, "label.npy") + + if os.path.exists(ply_path): + mesh = trimesh.load(ply_path, process=False) + coords = np.array(mesh.vertices, dtype=np.float32) + np.save(os.path.join(target_dir, "coord.npy"), coords) + + if ( + hasattr(mesh, "vertex_normals") + and mesh.vertex_normals is not None + and len(mesh.vertex_normals) == len(coords) + ): + normals = np.array(mesh.vertex_normals, dtype=np.float32) + else: + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coords) + search_param = o3d.geometry.KDTreeSearchParamHybrid(radius=0.1, max_nn=30) + pcd.estimate_normals(search_param=search_param) + normals = np.asarray(pcd.normals).astype(np.float32) + + np.save(os.path.join(target_dir, "normal.npy"), normals) + + if ( + hasattr(mesh.visual, "vertex_colors") + and mesh.visual.vertex_colors is not None + ): + colors = np.array(mesh.visual.vertex_colors[:, :3], dtype=np.uint8) + np.save(os.path.join(target_dir, "color.npy"), colors) + elif hasattr(mesh, "colors") and mesh.colors is not None: + colors = np.array(mesh.colors[:, :3], dtype=np.uint8) + np.save(os.path.join(target_dir, "color.npy"), colors) + else: + pass + + label_data = np.load(label_path, allow_pickle=True).item() + segment = label_data["semantic_seg"] + instance = label_data["instance_seg"] + assert coords.shape[0] == segment.shape[0] + segment_path = os.path.join(target_dir, "segment.npy") + instance_path = os.path.join(target_dir, "instance.npy") + np.save(segment_path, segment) + np.save(instance_path, instance) + else: + print(f"Warning: pc.ply not found in {target_dir}") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Preprocess PartNetE dataset by splitting PLY files and copying labels." + ) + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + args = parser.parse_args() + subfolders = ["few_shot", "test"] + total_processed_count = 0 + + for subfolder in subfolders: + current_root = os.path.join(args.dataset_root, subfolder) + processed_in_subfolder = 0 + for dirpath, dirnames, filenames in os.walk(current_root): + if "pc.ply" in filenames and "label.npy" in filenames: + process_folder(dirpath) + processed_in_subfolder += 1 diff --git a/pointcept/datasets/preprocessing/s3dis/preprocess_s3dis.py b/pointcept/datasets/preprocessing/s3dis/preprocess_s3dis.py new file mode 100644 index 0000000000000000000000000000000000000000..1bf83f5c7fdef136d5dd7e53b285ac56ad065719 --- /dev/null +++ b/pointcept/datasets/preprocessing/s3dis/preprocess_s3dis.py @@ -0,0 +1,233 @@ +""" +Preprocessing Script for S3DIS +Parsing normal vectors has a large consumption of memory. Please reduce max_workers if memory is limited. + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import argparse +import glob +import numpy as np + +try: + import open3d +except ImportError: + import warnings + + warnings.warn("Please install open3d for parsing normal") + +try: + import trimesh +except ImportError: + import warnings + + warnings.warn("Please install trimesh for parsing normal") + +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat + +area_mesh_dict = {} + + +def parse_room( + room, angle, dataset_root, output_root, align_angle=True, parse_normal=False +): + print("Parsing: {}".format(room)) + classes = [ + "ceiling", + "floor", + "wall", + "beam", + "column", + "window", + "door", + "table", + "chair", + "sofa", + "bookcase", + "board", + "clutter", + ] + class2label = {cls: i for i, cls in enumerate(classes)} + source_dir = os.path.join(dataset_root, room) + save_path = os.path.join(output_root, room) + os.makedirs(save_path, exist_ok=True) + object_path_list = sorted(glob.glob(os.path.join(source_dir, "Annotations/*.txt"))) + + room_coords = [] + room_colors = [] + room_normals = [] + room_semantic_gt = [] + room_instance_gt = [] + + for object_id, object_path in enumerate(object_path_list): + object_name = os.path.basename(object_path).split("_")[0] + obj = np.loadtxt(object_path) + coords = obj[:, :3] + colors = obj[:, 3:6] + # note: in some room there is 'stairs' class + class_name = object_name if object_name in classes else "clutter" + semantic_gt = np.repeat(class2label[class_name], coords.shape[0]) + semantic_gt = semantic_gt.reshape([-1, 1]) + instance_gt = np.repeat(object_id, coords.shape[0]) + instance_gt = instance_gt.reshape([-1, 1]) + + room_coords.append(coords) + room_colors.append(colors) + room_semantic_gt.append(semantic_gt) + room_instance_gt.append(instance_gt) + + room_coords = np.ascontiguousarray(np.vstack(room_coords)) + + if parse_normal: + x_min, z_max, y_min = np.min(room_coords, axis=0) + x_max, z_min, y_max = np.max(room_coords, axis=0) + z_max = -z_max + z_min = -z_min + max_bound = np.array([x_max, y_max, z_max]) + 0.1 + min_bound = np.array([x_min, y_min, z_min]) - 0.1 + bbox = open3d.geometry.AxisAlignedBoundingBox( + min_bound=min_bound, max_bound=max_bound + ) + # crop room + room_mesh = ( + area_mesh_dict[os.path.dirname(room)] + .crop(bbox) + .transform( + np.array([[1, 0, 0, 0], [0, 0, -1, 0], [0, 1, 0, 0], [0, 0, 0, 1]]) + ) + ) + vertices = np.array(room_mesh.vertices) + faces = np.array(room_mesh.triangles) + vertex_normals = np.array(room_mesh.vertex_normals) + room_mesh = trimesh.Trimesh( + vertices=vertices, faces=faces, vertex_normals=vertex_normals + ) + closest_points, distances, face_id = room_mesh.nearest.on_surface(room_coords) + room_normals = room_mesh.face_normals[face_id] + + if align_angle: + angle = (2 - angle / 180) * np.pi + rot_cos, rot_sin = np.cos(angle), np.sin(angle) + rot_t = np.array([[rot_cos, -rot_sin, 0], [rot_sin, rot_cos, 0], [0, 0, 1]]) + room_center = (np.max(room_coords, axis=0) + np.min(room_coords, axis=0)) / 2 + room_coords = (room_coords - room_center) @ np.transpose(rot_t) + room_center + if parse_normal: + room_normals = room_normals @ np.transpose(rot_t) + + room_colors = np.ascontiguousarray(np.vstack(room_colors)) + room_semantic_gt = np.ascontiguousarray(np.vstack(room_semantic_gt)) + room_instance_gt = np.ascontiguousarray(np.vstack(room_instance_gt)) + np.save(os.path.join(save_path, "coord.npy"), room_coords.astype(np.float32)) + np.save(os.path.join(save_path, "color.npy"), room_colors.astype(np.uint8)) + np.save(os.path.join(save_path, "segment.npy"), room_semantic_gt.astype(np.int16)) + np.save(os.path.join(save_path, "instance.npy"), room_instance_gt.astype(np.int16)) + + if parse_normal: + np.save(os.path.join(save_path, "normal.npy"), room_normals.astype(np.float32)) + + +def main_process(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--splits", + required=True, + nargs="+", + choices=["Area_1", "Area_2", "Area_3", "Area_4", "Area_5", "Area_6"], + help="Splits need to process ([Area_1, Area_2, Area_3, Area_4, Area_5, Area_6]).", + ) + parser.add_argument( + "--dataset_root", required=True, help="Path to Stanford3dDataset_v1.2 dataset" + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where area folders will be located", + ) + parser.add_argument( + "--raw_root", + default=None, + help="Path to Stanford2d3dDataset_noXYZ dataset (optional)", + ) + parser.add_argument( + "--align_angle", action="store_true", help="Whether align room angles" + ) + parser.add_argument( + "--parse_normal", action="store_true", help="Whether process normal" + ) + parser.add_argument( + "--num_workers", default=1, type=int, help="Num workers for preprocessing." + ) + args = parser.parse_args() + + if args.parse_normal: + assert args.raw_root is not None + + room_list = [] + angle_list = [] + + # Load room information + print("Loading room information ...") + for split in args.splits: + area_info = np.loadtxt( + os.path.join( + args.dataset_root, + split, + f"{split}_alignmentAngle.txt", + ), + dtype=str, + ) + room_list += [os.path.join(split, room_info[0]) for room_info in area_info] + angle_list += [int(room_info[1]) for room_info in area_info] + + if args.parse_normal: + # load raw mesh file to extract normal + print("Loading raw mesh file ...") + for split in args.splits: + if split != "Area_5": + mesh_dir = os.path.join(args.raw_root, split, "3d", "rgb.obj") + mesh = open3d.io.read_triangle_mesh(mesh_dir) + mesh.triangle_uvs.clear() + else: + mesh_a_dir = os.path.join(args.raw_root, f"{split}a", "3d", "rgb.obj") + mesh_b_dir = os.path.join(args.raw_root, f"{split}b", "3d", "rgb.obj") + mesh_a = open3d.io.read_triangle_mesh(mesh_a_dir) + mesh_a.triangle_uvs.clear() + mesh_b = open3d.io.read_triangle_mesh(mesh_b_dir) + mesh_b.triangle_uvs.clear() + mesh_b = mesh_b.transform( + np.array( + [ + [0, 0, -1, -4.09703582], + [0, 1, 0, 0], + [1, 0, 0, -6.22617759], + [0, 0, 0, 1], + ] + ) + ) + mesh = mesh_a + mesh_b + area_mesh_dict[split] = mesh + print(f"{split} mesh is loaded") + + # Preprocess data. + print("Processing scenes...") + pool = ProcessPoolExecutor( + max_workers=args.num_workers + ) # peak 110G memory when parsing normal. + _ = list( + pool.map( + parse_room, + room_list, + angle_list, + repeat(args.dataset_root), + repeat(args.output_root), + repeat(args.align_angle), + repeat(args.parse_normal), + ) + ) + + +if __name__ == "__main__": + main_process() diff --git a/pointcept/datasets/preprocessing/sampling_chunking_data.py b/pointcept/datasets/preprocessing/sampling_chunking_data.py new file mode 100644 index 0000000000000000000000000000000000000000..96536d415370bf28c0f1cc89312b2fde719c9a58 --- /dev/null +++ b/pointcept/datasets/preprocessing/sampling_chunking_data.py @@ -0,0 +1,149 @@ +""" +Chunking Data + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import argparse +import numpy as np +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat +from pathlib import Path + + +def chunking_scene( + name, + dataset_root, + split, + grid_size=None, + chunk_range=(6, 6), + chunk_stride=(3, 3), + chunk_minimum_size=10000, +): + print(f"Chunking scene {name} in {split} split") + dataset_root = Path(dataset_root) + scene_path = dataset_root / split / name + assets = os.listdir(scene_path) + data_dict = dict() + for asset in assets: + if not asset.endswith(".npy"): + continue + data_dict[asset[:-4]] = np.load(scene_path / asset) + coord = data_dict["coord"] - data_dict["coord"].min(axis=0) + + if grid_size is not None: + grid_coord = np.floor(coord / grid_size).astype(int) + _, idx = np.unique(grid_coord, axis=0, return_index=True) + coord = coord[idx] + for key in data_dict.keys(): + data_dict[key] = data_dict[key][idx] + + bev_range = coord.max(axis=0)[:2] + x, y = np.meshgrid( + np.arange(0, bev_range[0] + chunk_stride[0] - chunk_range[0], chunk_stride[0]), + np.arange(0, bev_range[0] + chunk_stride[0] - chunk_range[0], chunk_stride[0]), + indexing="ij", + ) + chunks = np.concatenate([x.reshape([-1, 1]), y.reshape([-1, 1])], axis=-1) + chunk_idx = 0 + for chunk in chunks: + mask = ( + (coord[:, 0] >= chunk[0]) + & (coord[:, 0] < chunk[0] + chunk_range[0]) + & (coord[:, 1] >= chunk[1]) + & (coord[:, 1] < chunk[1] + chunk_range[1]) + ) + if np.sum(mask) < chunk_minimum_size: + continue + + chunk_data_name = f"{name}_{chunk_idx}" + if grid_size is not None: + chunk_split_name = ( + f"{split}_" + f"grid{grid_size * 100:.0f}mm_" + f"chunk{chunk_range[0]}x{chunk_range[1]}_" + f"stride{chunk_stride[0]}x{chunk_stride[1]}" + ) + else: + chunk_split_name = ( + f"{split}_" + f"chunk{chunk_range[0]}x{chunk_range[1]}_" + f"stride{chunk_stride[0]}x{chunk_stride[1]}" + ) + + chunk_save_path = dataset_root / chunk_split_name / chunk_data_name + chunk_save_path.mkdir(parents=True, exist_ok=True) + for key in data_dict.keys(): + np.save(chunk_save_path / f"{key}.npy", data_dict[key][mask]) + chunk_idx += 1 + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the Pointcept processed ScanNet++ dataset.", + ) + parser.add_argument( + "--split", + required=True, + default="train", + type=str, + help="Split need to process.", + ) + parser.add_argument( + "--grid_size", + default=None, + type=float, + help="Grid size for initial grid sampling", + ) + parser.add_argument( + "--chunk_range", + default=[6, 6], + type=int, + nargs="+", + help="Range of each chunk, e.g. --chunk_range 6 6", + ) + parser.add_argument( + "--chunk_stride", + default=[3, 3], + type=int, + nargs="+", + help="Stride of each chunk, e.g. --chunk_stride 3 3", + ) + parser.add_argument( + "--chunk_minimum_size", + default=10000, + type=int, + help="Minimum number of points in each chunk", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + + config = parser.parse_args() + config.dataset_root = Path(config.dataset_root) + data_list = os.listdir(config.dataset_root / config.split) + + print("Processing scenes...") + pool = ProcessPoolExecutor(max_workers=config.num_workers) + _ = list( + pool.map( + chunking_scene, + data_list, + repeat(config.dataset_root), + repeat(config.split), + repeat(config.grid_size), + repeat(config.chunk_range), + repeat(config.chunk_stride), + repeat(config.chunk_minimum_size), + ) + ) + pool.shutdown() diff --git a/pointcept/datasets/preprocessing/scannet/dino/prepare_scene_list.py b/pointcept/datasets/preprocessing/scannet/dino/prepare_scene_list.py new file mode 100644 index 0000000000000000000000000000000000000000..7b43f15002ee1f8167a964cd2e9cc9c5627e2266 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/dino/prepare_scene_list.py @@ -0,0 +1,27 @@ +import os +import argparse +from pathlib import Path + +import numpy as np + +if __name__ == "__main__": + num_train_list = 12 + num_val_list = 3 + meta_root = Path(os.path.dirname(__file__)).parent / "meta_data" + + # Load train/val splits + train_scenes = np.loadtxt(meta_root / "scannetv2_train.txt", dtype=str) + val_scenes = np.loadtxt(meta_root / "scannetv2_val.txt", dtype=str) + + for i in range(num_train_list): + np.savetxt( + meta_root / f"scannetv2_train_{i}.txt", + train_scenes[i::num_train_list], + fmt="%s", + ) + for i in range(num_val_list): + np.savetxt( + meta_root / f"scannetv2_val_{i}.txt", + val_scenes[i::num_val_list], + fmt="%s", + ) diff --git a/pointcept/datasets/preprocessing/scannet/dino/preprocess_dino_feature.py b/pointcept/datasets/preprocessing/scannet/dino/preprocess_dino_feature.py new file mode 100644 index 0000000000000000000000000000000000000000..1eb954161681ca5c3609671d7c4170c2f3dcad9f --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/dino/preprocess_dino_feature.py @@ -0,0 +1,362 @@ +import os +import argparse +import einops +import torch +import torch.nn.functional as F +import torchvision +import tqdm +import cv2 +import camtools as ct +import open3d as o3d +import zlib +import imageio +import struct +import numpy as np +import torch_scatter +from pathlib import Path + + +class RGBDFrame: + def __init__(self, file_handle): + self.camera_to_world = np.asarray( + struct.unpack("f" * 16, file_handle.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.timestamp_color = struct.unpack("Q", file_handle.read(8))[0] + self.timestamp_depth = struct.unpack("Q", file_handle.read(8))[0] + self.color_size_bytes = struct.unpack("Q", file_handle.read(8))[0] + self.depth_size_bytes = struct.unpack("Q", file_handle.read(8))[0] + self.color_data = b"".join( + struct.unpack( + "c" * self.color_size_bytes, file_handle.read(self.color_size_bytes) + ) + ) + self.depth_data = b"".join( + struct.unpack( + "c" * self.depth_size_bytes, file_handle.read(self.depth_size_bytes) + ) + ) + + def decompress_depth(self, compression_type): + if compression_type == "zlib_ushort": + return self.decompress_depth_zlib() + else: + raise + + def decompress_depth_zlib(self): + return zlib.decompress(self.depth_data) + + def decompress_color(self, compression_type): + if compression_type == "jpeg": + return self.decompress_color_jpeg() + else: + raise + + def decompress_color_jpeg(self): + return imageio.v2.imread(self.color_data) + + +class SensorData: + COMPRESSION_TYPE_COLOR = { + -1: "unknown", + 0: "raw", + 1: "png", + 2: "jpeg", + } + COMPRESSION_TYPE_DEPTH = { + -1: "unknown", + 0: "raw_ushort", + 1: "zlib_ushort", + 2: "occi_ushort", + } + + def __init__(self, filename): + self.version = 4 + f = open(filename, "rb") + version = struct.unpack("I", f.read(4))[0] + assert self.version == version + strlen = struct.unpack("Q", f.read(8))[0] + self.sensor_name = b"".join(struct.unpack("c" * strlen, f.read(strlen))) + self.intrinsic_color = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.extrinsic_color = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.intrinsic_depth = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.extrinsic_depth = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.color_compression_type = self.COMPRESSION_TYPE_COLOR[ + struct.unpack("i", f.read(4))[0] + ] + self.depth_compression_type = self.COMPRESSION_TYPE_DEPTH[ + struct.unpack("i", f.read(4))[0] + ] + self.color_width = struct.unpack("I", f.read(4))[0] + self.color_height = struct.unpack("I", f.read(4))[0] + self.depth_width = struct.unpack("I", f.read(4))[0] + self.depth_height = struct.unpack("I", f.read(4))[0] + self.depth_shift = struct.unpack("f", f.read(4))[0] + self.num_frames = struct.unpack("Q", f.read(8))[0] + self.file_handle = f + + def export( + self, + frame_skip=20, + export_color=True, + export_depth=True, + export_pose=True, + ): + for i in range(self.num_frames): + if i % frame_skip != 0: + self.file_handle.seek(16 * 4 + 8 + 8, 1) # skip pose, timestamp + color_size_bytes = struct.unpack("Q", self.file_handle.read(8))[0] + depth_size_bytes = struct.unpack("Q", self.file_handle.read(8))[0] + self.file_handle.seek(color_size_bytes + depth_size_bytes, 1) + continue + else: + frame = RGBDFrame(self.file_handle) + data_dict = {} + if export_color: + color = frame.decompress_color(self.color_compression_type) + data_dict["color"] = color + if export_depth: + depth = frame.decompress_depth(self.depth_compression_type) + depth = np.frombuffer(depth, dtype=np.uint16).reshape( + self.depth_height, self.depth_width + ) + data_dict["depth"] = depth + if export_pose: + pose = frame.camera_to_world + data_dict["pose"] = pose + yield data_dict + + def __del__(self): + self.file_handle.close() + + +def ray_distance_to_z_depth(ray_depth, K): + height, width = ray_depth.shape + + u = np.arange(width) + v = np.arange(height) + u_grid, v_grid = np.meshgrid(u, v) + + fx = K[0, 0] + fy = K[1, 1] + cx = K[0, 2] + cy = K[1, 2] + + u_norm = (u_grid - cx) / fx + v_norm = (v_grid - cy) / fy + + norm_square = u_norm**2 + v_norm**2 + + z_depth = ray_depth / np.sqrt(norm_square + 1) + return z_depth + + +def center_crop(image, crop_ratio=1.0, patch_size=None): + if len(image.shape) == 2: + height, width = image.shape + elif len(image.shape) == 3: + height, width, _ = image.shape + else: + raise ValueError("Invalid image shape") + if patch_size is not None: + crop_h = int(height * crop_ratio // patch_size * patch_size) + crop_w = int(width * crop_ratio // patch_size * patch_size) + else: + crop_h = int(height * crop_ratio) + crop_w = int(width * crop_ratio) + + # Calculate the cropping box + start_h = (height - crop_h) // 2 + start_w = (width - crop_w) // 2 + + # Perform the center crop + cropped_image = image[start_h : start_h + crop_h, start_w : start_w + crop_w] + + return cropped_image + + +def parsing_scene( + scene_path, + output_root, + split, + model, + frame_skip=20, + grid_size=0.08, + crop_ratio=0.95, + device="cuda", +): + print(f"Parsing scene: {scene_path.name}") + device = torch.device(device) + scene_path = Path(scene_path) + sensor_reader = SensorData(scene_path / f"{scene_path.name}.sens") + mesh = o3d.io.read_triangle_mesh( + str(scene_path / f"{scene_path.name}_vh_clean_2.ply") + ) + transform = torchvision.transforms.Compose( + [ + torchvision.transforms.ToTensor(), + torchvision.transforms.Normalize( + mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225) + ), + ] + ) + scene_coord = [] + scene_feat = [] + scene_count = [] + for data in tqdm.tqdm( + sensor_reader.export(frame_skip=frame_skip), + total=sensor_reader.num_frames // frame_skip, + ): + height, width = data["depth"].shape + K = sensor_reader.intrinsic_depth[:3, :3] + T = data["pose"] + if np.isnan(T).any() or np.isinf(T).any(): + continue + depth = ct.raycast.mesh_to_depth( + mesh=mesh, K=K, T=np.linalg.inv(T), height=height, width=width + ) + depth = ray_distance_to_z_depth(depth, K) + depth = center_crop(depth, crop_ratio, model.patch_size) + height_, width_ = depth.shape + pixel = np.transpose(np.indices((width_, height_)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + depth = depth.reshape((-1, 1)) + valid = ~np.isinf(depth).squeeze(-1) + coord = depth[valid] * (np.linalg.inv(K) @ pixel[valid].T).T # coord_camera + coord = coord @ T[:3, :3].T + T[:3, 3] + + color = cv2.resize( + data["color"], (width, height), interpolation=cv2.INTER_LINEAR + ) + color = center_crop(color, crop_ratio, model.patch_size) + with torch.inference_mode(): + color_t = transform(color).unsqueeze(0).to(device) + feat_t = model.forward_features(color_t)["x_norm_patchtokens"] + feat_t = einops.rearrange( + feat_t, "1 (h w) c -> 1 c h w", w=width_ // model.patch_size + ) + feat_t = F.interpolate(feat_t, (height_, width_), mode="bilinear") + feat_t = einops.rearrange(feat_t, "1 c h w -> (h w) c")[valid] + coord_t = torch.tensor(coord, dtype=torch.float32).to(device) + scene_coord.append(coord_t) + scene_feat.append(feat_t) + scene_count.append( + torch.ones(coord_t.shape[0], dtype=torch.long, device=device) + ) + scene_coord = torch.concatenate(scene_coord, dim=0) + scene_feat = torch.concatenate(scene_feat, dim=0) + scene_count = torch.concatenate(scene_count, dim=0) + + # grid sampling + grid_coord = torch.floor_divide(scene_coord, grid_size).to(torch.int32) + grid_coord, cluster = torch.unique( + grid_coord, sorted=True, return_inverse=True, dim=0 + ) + scene_coord = [ + torch_scatter.scatter(scene_coord, cluster, reduce="mean", dim=0) + ] + scene_feat = [ + torch_scatter.scatter(scene_feat, cluster, reduce="sum", dim=0) + ] + scene_count = [ + torch_scatter.scatter(scene_count, cluster, reduce="sum", dim=0) + ] + + # color = color.reshape((-1, 3))[valid] + # pcd = o3d.geometry.PointCloud() + # pcd.points = o3d.utility.Vector3dVector(coord) + # pcd.colors = o3d.utility.Vector3dVector(color / 255) + # o3d.visualization.draw_geometries([pcd]) + + scene_coord = scene_coord[0] + scene_feat = scene_feat[0] / scene_count[0].unsqueeze(-1) + + scene_coord = scene_coord.half().cpu().numpy() + scene_feat = scene_feat.half().cpu().numpy() + np.savez( + Path(output_root) / split / f"{scene_path.name}.npz", + coord=scene_coord, + feat=scene_feat, + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--scene_list", + required=True, + help="Path to scene list need to process", + ) + parser.add_argument( + "--frame_skip", + default=10, + help="Frame skip for processing", + ) + parser.add_argument( + "--grid_size", + default=0.08, + help="Grid size for sampling", + ) + parser.add_argument( + "--crop_ratio", + default=0.95, + help="Crop ratio for center crop", + ) + + args = parser.parse_args() + scene_list = np.loadtxt(args.scene_list, dtype=str) + if "train" in args.scene_list: + split = "train" + folder = "scans" + elif "val" in args.scene_list: + split = "val" + folder = "scans" + else: + split = "test" + folder = "scans_test" + + os.makedirs(Path(args.output_root) / split, exist_ok=True) + + device = torch.device("cuda") + model = torch.hub.load("facebookresearch/dinov2", "dinov2_vitg14").to(device) + model.eval() + for scene in scene_list: + parsing_scene( + scene_path=Path(args.dataset_root) / folder / scene, + output_root=args.output_root, + split=split, + frame_skip=args.frame_skip, + grid_size=args.grid_size, + crop_ratio=args.crop_ratio, + model=model, + device="cuda", + ) + + # parsing_scene( + # scene_path=Path("/mnt/e/datasets/raw/scannet/scans/scene0230_00"), + # output_root=args.output_root, + # split=split, + # frame_skip=args.frame_skip, + # grid_size=args.grid_size, + # crop_ratio=args.crop_ratio, + # model=model, + # device="cuda", + # ) diff --git a/pointcept/datasets/preprocessing/scannet/extract_partition.py b/pointcept/datasets/preprocessing/scannet/extract_partition.py new file mode 100644 index 0000000000000000000000000000000000000000..432a490fdb9588b5c060b22b900d050ec3a4e4b5 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/extract_partition.py @@ -0,0 +1,71 @@ +import json +import shutil +import argparse +import torch +import glob +import os.path + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--processed_root", + required=True, + help="Path to the processed ScanNet dataset, add partition to test data dict", + ) + parser.add_argument( + "--segmentor_root", + required=True, + help="Path to Felzenswalb and Huttenlocher's Graph Based Image Segmentation binary", + ) + parser.add_argument( + "--split", + default="test", + choices=["test", "val"], + help="Split to process. [test / val]", + ) + config = parser.parse_args() + if config.split == "test": + raw_split = "scans_test" + else: + raw_split = "scans" + + scene_list = glob.glob(os.path.join(config.processed_root, config.split, "*.pth")) + os.makedirs(os.path.join(config.processed_root, "tmp"), exist_ok=True) + + for scene in scene_list: + scene_name = os.path.basename(scene).split(".")[0] + raw_scene = os.path.join( + config.dataset_root, + raw_split, + scene_name, + f"{scene_name}_vh_clean_2.ply", + ) + tmp_scene = os.path.join( + config.processed_root, + "tmp", + f"{scene_name}_vh_clean_2.ply", + ) + # copy original scene to tmp folder + shutil.copy(raw_scene, tmp_scene) + # run segmentor + process = os.popen(f"{config.segmentor_root} {tmp_scene}") + print(process.read()) + process.close() + # load partition file + partition_file = tmp_scene.replace(".ply", ".0.010000.segs.json") + with open(partition_file) as f: + partition = json.load(f)["segIndices"] + data_dict = torch.load(scene) + data_dict["partition"] = partition + torch.save(data_dict, scene) + # clean tmp + os.remove(partition_file) + os.remove(tmp_scene) + print(f"Adding partition information to {scene_name}") + + os.rmdir(os.path.join(config.processed_root, "tmp")) diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/classes_ObjClassification-ShapeNetCore55.txt b/pointcept/datasets/preprocessing/scannet/meta_data/classes_ObjClassification-ShapeNetCore55.txt new file mode 100644 index 0000000000000000000000000000000000000000..e53f5bcb2c1480f42ee9327940246258aa434f88 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/classes_ObjClassification-ShapeNetCore55.txt @@ -0,0 +1,17 @@ +1 trash +3 basket +4 bathtub +5 bed +9 shelf +13 cabinet +18 chair +20 keyboard +22 tv +30 lamp +31 laptop +35 microwave +39 pillow +42 printer +47 sofa +48 stove +49 table diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/classes_SemVoxLabel-nyu40id.txt b/pointcept/datasets/preprocessing/scannet/meta_data/classes_SemVoxLabel-nyu40id.txt new file mode 100644 index 0000000000000000000000000000000000000000..48e228766391e0f0234c2eed086e31f738068a4b --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/classes_SemVoxLabel-nyu40id.txt @@ -0,0 +1,20 @@ +1 wall +2 floor +3 cabinet +4 bed +5 chair +6 sofa +7 table +8 door +9 window +10 bookshelf +11 picture +12 counter +14 desk +16 curtain +24 refridgerator +28 shower curtain +33 toilet +34 sink +36 bathtub +39 otherfurniture \ No newline at end of file diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannet200_constants.py b/pointcept/datasets/preprocessing/scannet/meta_data/scannet200_constants.py new file mode 100644 index 0000000000000000000000000000000000000000..0404fd6aa8ad14ad729354ce184d4b51834bfd1b --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannet200_constants.py @@ -0,0 +1,704 @@ +# ScanNet Benchmark constants +VALID_CLASS_IDS_20 = ( + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 14, + 16, + 24, + 28, + 33, + 34, + 36, + 39, +) + +CLASS_LABELS_20 = ( + "wall", + "floor", + "cabinet", + "bed", + "chair", + "sofa", + "table", + "door", + "window", + "bookshelf", + "picture", + "counter", + "desk", + "curtain", + "refrigerator", + "shower curtain", + "toilet", + "sink", + "bathtub", + "otherfurniture", +) + +SCANNET_COLOR_MAP_20 = { + 0: (0.0, 0.0, 0.0), + 1: (174.0, 199.0, 232.0), + 2: (152.0, 223.0, 138.0), + 3: (31.0, 119.0, 180.0), + 4: (255.0, 187.0, 120.0), + 5: (188.0, 189.0, 34.0), + 6: (140.0, 86.0, 75.0), + 7: (255.0, 152.0, 150.0), + 8: (214.0, 39.0, 40.0), + 9: (197.0, 176.0, 213.0), + 10: (148.0, 103.0, 189.0), + 11: (196.0, 156.0, 148.0), + 12: (23.0, 190.0, 207.0), + 14: (247.0, 182.0, 210.0), + 15: (66.0, 188.0, 102.0), + 16: (219.0, 219.0, 141.0), + 17: (140.0, 57.0, 197.0), + 18: (202.0, 185.0, 52.0), + 19: (51.0, 176.0, 203.0), + 20: (200.0, 54.0, 131.0), + 21: (92.0, 193.0, 61.0), + 22: (78.0, 71.0, 183.0), + 23: (172.0, 114.0, 82.0), + 24: (255.0, 127.0, 14.0), + 25: (91.0, 163.0, 138.0), + 26: (153.0, 98.0, 156.0), + 27: (140.0, 153.0, 101.0), + 28: (158.0, 218.0, 229.0), + 29: (100.0, 125.0, 154.0), + 30: (178.0, 127.0, 135.0), + 32: (146.0, 111.0, 194.0), + 33: (44.0, 160.0, 44.0), + 34: (112.0, 128.0, 144.0), + 35: (96.0, 207.0, 209.0), + 36: (227.0, 119.0, 194.0), + 37: (213.0, 92.0, 176.0), + 38: (94.0, 106.0, 211.0), + 39: (82.0, 84.0, 163.0), + 40: (100.0, 85.0, 144.0), +} + +# ScanNet200 Benchmark constants +VALID_CLASS_IDS_200 = ( + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 21, + 22, + 23, + 24, + 26, + 27, + 28, + 29, + 31, + 32, + 33, + 34, + 35, + 36, + 38, + 39, + 40, + 41, + 42, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 54, + 55, + 56, + 57, + 58, + 59, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 82, + 84, + 86, + 87, + 88, + 89, + 90, + 93, + 95, + 96, + 97, + 98, + 99, + 100, + 101, + 102, + 103, + 104, + 105, + 106, + 107, + 110, + 112, + 115, + 116, + 118, + 120, + 121, + 122, + 125, + 128, + 130, + 131, + 132, + 134, + 136, + 138, + 139, + 140, + 141, + 145, + 148, + 154, + 155, + 156, + 157, + 159, + 161, + 163, + 165, + 166, + 168, + 169, + 170, + 177, + 180, + 185, + 188, + 191, + 193, + 195, + 202, + 208, + 213, + 214, + 221, + 229, + 230, + 232, + 233, + 242, + 250, + 261, + 264, + 276, + 283, + 286, + 300, + 304, + 312, + 323, + 325, + 331, + 342, + 356, + 370, + 392, + 395, + 399, + 408, + 417, + 488, + 540, + 562, + 570, + 572, + 581, + 609, + 748, + 776, + 1156, + 1163, + 1164, + 1165, + 1166, + 1167, + 1168, + 1169, + 1170, + 1171, + 1172, + 1173, + 1174, + 1175, + 1176, + 1178, + 1179, + 1180, + 1181, + 1182, + 1183, + 1184, + 1185, + 1186, + 1187, + 1188, + 1189, + 1190, + 1191, +) + +CLASS_LABELS_200 = ( + "wall", + "chair", + "floor", + "table", + "door", + "couch", + "cabinet", + "shelf", + "desk", + "office chair", + "bed", + "pillow", + "sink", + "picture", + "window", + "toilet", + "bookshelf", + "monitor", + "curtain", + "book", + "armchair", + "coffee table", + "box", + "refrigerator", + "lamp", + "kitchen cabinet", + "towel", + "clothes", + "tv", + "nightstand", + "counter", + "dresser", + "stool", + "cushion", + "plant", + "ceiling", + "bathtub", + "end table", + "dining table", + "keyboard", + "bag", + "backpack", + "toilet paper", + "printer", + "tv stand", + "whiteboard", + "blanket", + "shower curtain", + "trash can", + "closet", + "stairs", + "microwave", + "stove", + "shoe", + "computer tower", + "bottle", + "bin", + "ottoman", + "bench", + "board", + "washing machine", + "mirror", + "copier", + "basket", + "sofa chair", + "file cabinet", + "fan", + "laptop", + "shower", + "paper", + "person", + "paper towel dispenser", + "oven", + "blinds", + "rack", + "plate", + "blackboard", + "piano", + "suitcase", + "rail", + "radiator", + "recycling bin", + "container", + "wardrobe", + "soap dispenser", + "telephone", + "bucket", + "clock", + "stand", + "light", + "laundry basket", + "pipe", + "clothes dryer", + "guitar", + "toilet paper holder", + "seat", + "speaker", + "column", + "bicycle", + "ladder", + "bathroom stall", + "shower wall", + "cup", + "jacket", + "storage bin", + "coffee maker", + "dishwasher", + "paper towel roll", + "machine", + "mat", + "windowsill", + "bar", + "toaster", + "bulletin board", + "ironing board", + "fireplace", + "soap dish", + "kitchen counter", + "doorframe", + "toilet paper dispenser", + "mini fridge", + "fire extinguisher", + "ball", + "hat", + "shower curtain rod", + "water cooler", + "paper cutter", + "tray", + "shower door", + "pillar", + "ledge", + "toaster oven", + "mouse", + "toilet seat cover dispenser", + "furniture", + "cart", + "storage container", + "scale", + "tissue box", + "light switch", + "crate", + "power outlet", + "decoration", + "sign", + "projector", + "closet door", + "vacuum cleaner", + "candle", + "plunger", + "stuffed animal", + "headphones", + "dish rack", + "broom", + "guitar case", + "range hood", + "dustpan", + "hair dryer", + "water bottle", + "handicap bar", + "purse", + "vent", + "shower floor", + "water pitcher", + "mailbox", + "bowl", + "paper bag", + "alarm clock", + "music stand", + "projector screen", + "divider", + "laundry detergent", + "bathroom counter", + "object", + "bathroom vanity", + "closet wall", + "laundry hamper", + "bathroom stall door", + "ceiling light", + "trash bin", + "dumbbell", + "stair rail", + "tube", + "bathroom cabinet", + "cd case", + "closet rod", + "coffee kettle", + "structure", + "shower head", + "keyboard piano", + "case of water bottles", + "coat rack", + "storage organizer", + "folded chair", + "fire alarm", + "power strip", + "calendar", + "poster", + "potted plant", + "luggage", + "mattress", +) + +SCANNET_COLOR_MAP_200 = { + 0: (0.0, 0.0, 0.0), + 1: (174.0, 199.0, 232.0), + 2: (188.0, 189.0, 34.0), + 3: (152.0, 223.0, 138.0), + 4: (255.0, 152.0, 150.0), + 5: (214.0, 39.0, 40.0), + 6: (91.0, 135.0, 229.0), + 7: (31.0, 119.0, 180.0), + 8: (229.0, 91.0, 104.0), + 9: (247.0, 182.0, 210.0), + 10: (91.0, 229.0, 110.0), + 11: (255.0, 187.0, 120.0), + 13: (141.0, 91.0, 229.0), + 14: (112.0, 128.0, 144.0), + 15: (196.0, 156.0, 148.0), + 16: (197.0, 176.0, 213.0), + 17: (44.0, 160.0, 44.0), + 18: (148.0, 103.0, 189.0), + 19: (229.0, 91.0, 223.0), + 21: (219.0, 219.0, 141.0), + 22: (192.0, 229.0, 91.0), + 23: (88.0, 218.0, 137.0), + 24: (58.0, 98.0, 137.0), + 26: (177.0, 82.0, 239.0), + 27: (255.0, 127.0, 14.0), + 28: (237.0, 204.0, 37.0), + 29: (41.0, 206.0, 32.0), + 31: (62.0, 143.0, 148.0), + 32: (34.0, 14.0, 130.0), + 33: (143.0, 45.0, 115.0), + 34: (137.0, 63.0, 14.0), + 35: (23.0, 190.0, 207.0), + 36: (16.0, 212.0, 139.0), + 38: (90.0, 119.0, 201.0), + 39: (125.0, 30.0, 141.0), + 40: (150.0, 53.0, 56.0), + 41: (186.0, 197.0, 62.0), + 42: (227.0, 119.0, 194.0), + 44: (38.0, 100.0, 128.0), + 45: (120.0, 31.0, 243.0), + 46: (154.0, 59.0, 103.0), + 47: (169.0, 137.0, 78.0), + 48: (143.0, 245.0, 111.0), + 49: (37.0, 230.0, 205.0), + 50: (14.0, 16.0, 155.0), + 51: (196.0, 51.0, 182.0), + 52: (237.0, 80.0, 38.0), + 54: (138.0, 175.0, 62.0), + 55: (158.0, 218.0, 229.0), + 56: (38.0, 96.0, 167.0), + 57: (190.0, 77.0, 246.0), + 58: (208.0, 49.0, 84.0), + 59: (208.0, 193.0, 72.0), + 62: (55.0, 220.0, 57.0), + 63: (10.0, 125.0, 140.0), + 64: (76.0, 38.0, 202.0), + 65: (191.0, 28.0, 135.0), + 66: (211.0, 120.0, 42.0), + 67: (118.0, 174.0, 76.0), + 68: (17.0, 242.0, 171.0), + 69: (20.0, 65.0, 247.0), + 70: (208.0, 61.0, 222.0), + 71: (162.0, 62.0, 60.0), + 72: (210.0, 235.0, 62.0), + 73: (45.0, 152.0, 72.0), + 74: (35.0, 107.0, 149.0), + 75: (160.0, 89.0, 237.0), + 76: (227.0, 56.0, 125.0), + 77: (169.0, 143.0, 81.0), + 78: (42.0, 143.0, 20.0), + 79: (25.0, 160.0, 151.0), + 80: (82.0, 75.0, 227.0), + 82: (253.0, 59.0, 222.0), + 84: (240.0, 130.0, 89.0), + 86: (123.0, 172.0, 47.0), + 87: (71.0, 194.0, 133.0), + 88: (24.0, 94.0, 205.0), + 89: (134.0, 16.0, 179.0), + 90: (159.0, 32.0, 52.0), + 93: (213.0, 208.0, 88.0), + 95: (64.0, 158.0, 70.0), + 96: (18.0, 163.0, 194.0), + 97: (65.0, 29.0, 153.0), + 98: (177.0, 10.0, 109.0), + 99: (152.0, 83.0, 7.0), + 100: (83.0, 175.0, 30.0), + 101: (18.0, 199.0, 153.0), + 102: (61.0, 81.0, 208.0), + 103: (213.0, 85.0, 216.0), + 104: (170.0, 53.0, 42.0), + 105: (161.0, 192.0, 38.0), + 106: (23.0, 241.0, 91.0), + 107: (12.0, 103.0, 170.0), + 110: (151.0, 41.0, 245.0), + 112: (133.0, 51.0, 80.0), + 115: (184.0, 162.0, 91.0), + 116: (50.0, 138.0, 38.0), + 118: (31.0, 237.0, 236.0), + 120: (39.0, 19.0, 208.0), + 121: (223.0, 27.0, 180.0), + 122: (254.0, 141.0, 85.0), + 125: (97.0, 144.0, 39.0), + 128: (106.0, 231.0, 176.0), + 130: (12.0, 61.0, 162.0), + 131: (124.0, 66.0, 140.0), + 132: (137.0, 66.0, 73.0), + 134: (250.0, 253.0, 26.0), + 136: (55.0, 191.0, 73.0), + 138: (60.0, 126.0, 146.0), + 139: (153.0, 108.0, 234.0), + 140: (184.0, 58.0, 125.0), + 141: (135.0, 84.0, 14.0), + 145: (139.0, 248.0, 91.0), + 148: (53.0, 200.0, 172.0), + 154: (63.0, 69.0, 134.0), + 155: (190.0, 75.0, 186.0), + 156: (127.0, 63.0, 52.0), + 157: (141.0, 182.0, 25.0), + 159: (56.0, 144.0, 89.0), + 161: (64.0, 160.0, 250.0), + 163: (182.0, 86.0, 245.0), + 165: (139.0, 18.0, 53.0), + 166: (134.0, 120.0, 54.0), + 168: (49.0, 165.0, 42.0), + 169: (51.0, 128.0, 133.0), + 170: (44.0, 21.0, 163.0), + 177: (232.0, 93.0, 193.0), + 180: (176.0, 102.0, 54.0), + 185: (116.0, 217.0, 17.0), + 188: (54.0, 209.0, 150.0), + 191: (60.0, 99.0, 204.0), + 193: (129.0, 43.0, 144.0), + 195: (252.0, 100.0, 106.0), + 202: (187.0, 196.0, 73.0), + 208: (13.0, 158.0, 40.0), + 213: (52.0, 122.0, 152.0), + 214: (128.0, 76.0, 202.0), + 221: (187.0, 50.0, 115.0), + 229: (180.0, 141.0, 71.0), + 230: (77.0, 208.0, 35.0), + 232: (72.0, 183.0, 168.0), + 233: (97.0, 99.0, 203.0), + 242: (172.0, 22.0, 158.0), + 250: (155.0, 64.0, 40.0), + 261: (118.0, 159.0, 30.0), + 264: (69.0, 252.0, 148.0), + 276: (45.0, 103.0, 173.0), + 283: (111.0, 38.0, 149.0), + 286: (184.0, 9.0, 49.0), + 300: (188.0, 174.0, 67.0), + 304: (53.0, 206.0, 53.0), + 312: (97.0, 235.0, 252.0), + 323: (66.0, 32.0, 182.0), + 325: (236.0, 114.0, 195.0), + 331: (241.0, 154.0, 83.0), + 342: (133.0, 240.0, 52.0), + 356: (16.0, 205.0, 144.0), + 370: (75.0, 101.0, 198.0), + 392: (237.0, 95.0, 251.0), + 395: (191.0, 52.0, 49.0), + 399: (227.0, 254.0, 54.0), + 408: (49.0, 206.0, 87.0), + 417: (48.0, 113.0, 150.0), + 488: (125.0, 73.0, 182.0), + 540: (229.0, 32.0, 114.0), + 562: (158.0, 119.0, 28.0), + 570: (60.0, 205.0, 27.0), + 572: (18.0, 215.0, 201.0), + 581: (79.0, 76.0, 153.0), + 609: (134.0, 13.0, 116.0), + 748: (192.0, 97.0, 63.0), + 776: (108.0, 163.0, 18.0), + 1156: (95.0, 220.0, 156.0), + 1163: (98.0, 141.0, 208.0), + 1164: (144.0, 19.0, 193.0), + 1165: (166.0, 36.0, 57.0), + 1166: (212.0, 202.0, 34.0), + 1167: (23.0, 206.0, 34.0), + 1168: (91.0, 211.0, 236.0), + 1169: (79.0, 55.0, 137.0), + 1170: (182.0, 19.0, 117.0), + 1171: (134.0, 76.0, 14.0), + 1172: (87.0, 185.0, 28.0), + 1173: (82.0, 224.0, 187.0), + 1174: (92.0, 110.0, 214.0), + 1175: (168.0, 80.0, 171.0), + 1176: (197.0, 63.0, 51.0), + 1178: (175.0, 199.0, 77.0), + 1179: (62.0, 180.0, 98.0), + 1180: (8.0, 91.0, 150.0), + 1181: (77.0, 15.0, 130.0), + 1182: (154.0, 65.0, 96.0), + 1183: (197.0, 152.0, 11.0), + 1184: (59.0, 155.0, 45.0), + 1185: (12.0, 147.0, 145.0), + 1186: (54.0, 35.0, 219.0), + 1187: (210.0, 73.0, 181.0), + 1188: (221.0, 124.0, 77.0), + 1189: (149.0, 214.0, 66.0), + 1190: (72.0, 185.0, 134.0), + 1191: (42.0, 94.0, 198.0), +} + +# For instance segmentation the non-object categories +VALID_PANOPTIC_IDS = (1, 3) + +CLASS_LABELS_PANOPTIC = ("wall", "floor") diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannet200_splits.py b/pointcept/datasets/preprocessing/scannet/meta_data/scannet200_splits.py new file mode 100644 index 0000000000000000000000000000000000000000..39ccc3c60bf289199342332e455fadb5b22129ee --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannet200_splits.py @@ -0,0 +1,625 @@ +# This file contains the HEAD - COMMON - TAIL split category ids for ScanNet 200 + +HEAD_CATS_SCANNET_200 = [ + "tv stand", + "curtain", + "blinds", + "shower curtain", + "bookshelf", + "tv", + "kitchen cabinet", + "pillow", + "lamp", + "dresser", + "monitor", + "object", + "ceiling", + "board", + "stove", + "closet wall", + "couch", + "office chair", + "kitchen counter", + "shower", + "closet", + "doorframe", + "sofa chair", + "mailbox", + "nightstand", + "washing machine", + "picture", + "book", + "sink", + "recycling bin", + "table", + "backpack", + "shower wall", + "toilet", + "copier", + "counter", + "stool", + "refrigerator", + "window", + "file cabinet", + "chair", + "wall", + "plant", + "coffee table", + "stairs", + "armchair", + "cabinet", + "bathroom vanity", + "bathroom stall", + "mirror", + "blackboard", + "trash can", + "stair rail", + "box", + "towel", + "door", + "clothes", + "whiteboard", + "bed", + "floor", + "bathtub", + "desk", + "wardrobe", + "clothes dryer", + "radiator", + "shelf", +] +COMMON_CATS_SCANNET_200 = [ + "cushion", + "end table", + "dining table", + "keyboard", + "bag", + "toilet paper", + "printer", + "blanket", + "microwave", + "shoe", + "computer tower", + "bottle", + "bin", + "ottoman", + "bench", + "basket", + "fan", + "laptop", + "person", + "paper towel dispenser", + "oven", + "rack", + "piano", + "suitcase", + "rail", + "container", + "telephone", + "stand", + "light", + "laundry basket", + "pipe", + "seat", + "column", + "bicycle", + "ladder", + "jacket", + "storage bin", + "coffee maker", + "dishwasher", + "machine", + "mat", + "windowsill", + "bulletin board", + "fireplace", + "mini fridge", + "water cooler", + "shower door", + "pillar", + "ledge", + "furniture", + "cart", + "decoration", + "closet door", + "vacuum cleaner", + "dish rack", + "range hood", + "projector screen", + "divider", + "bathroom counter", + "laundry hamper", + "bathroom stall door", + "ceiling light", + "trash bin", + "bathroom cabinet", + "structure", + "storage organizer", + "potted plant", + "mattress", +] +TAIL_CATS_SCANNET_200 = [ + "paper", + "plate", + "soap dispenser", + "bucket", + "clock", + "guitar", + "toilet paper holder", + "speaker", + "cup", + "paper towel roll", + "bar", + "toaster", + "ironing board", + "soap dish", + "toilet paper dispenser", + "fire extinguisher", + "ball", + "hat", + "shower curtain rod", + "paper cutter", + "tray", + "toaster oven", + "mouse", + "toilet seat cover dispenser", + "storage container", + "scale", + "tissue box", + "light switch", + "crate", + "power outlet", + "sign", + "projector", + "candle", + "plunger", + "stuffed animal", + "headphones", + "broom", + "guitar case", + "dustpan", + "hair dryer", + "water bottle", + "handicap bar", + "purse", + "vent", + "shower floor", + "water pitcher", + "bowl", + "paper bag", + "alarm clock", + "music stand", + "laundry detergent", + "dumbbell", + "tube", + "cd case", + "closet rod", + "coffee kettle", + "shower head", + "keyboard piano", + "case of water bottles", + "coat rack", + "folded chair", + "fire alarm", + "power strip", + "calendar", + "poster", + "luggage", +] + + +# Given the different size of the official train and val sets, not all ScanNet200 categories are present in the validation set. +# Here we list of categories with labels and IDs present in both train and validation set, and the remaining categories those are present in train, but not in val +# We dont evaluate on unseen validation categories in this benchmark + +VALID_CLASS_IDS_200_VALIDATION = ( + "wall", + "chair", + "floor", + "table", + "door", + "couch", + "cabinet", + "shelf", + "desk", + "office chair", + "bed", + "pillow", + "sink", + "picture", + "window", + "toilet", + "bookshelf", + "monitor", + "curtain", + "book", + "armchair", + "coffee table", + "box", + "refrigerator", + "lamp", + "kitchen cabinet", + "towel", + "clothes", + "tv", + "nightstand", + "counter", + "dresser", + "stool", + "cushion", + "plant", + "ceiling", + "bathtub", + "end table", + "dining table", + "keyboard", + "bag", + "backpack", + "toilet paper", + "printer", + "tv stand", + "whiteboard", + "blanket", + "shower curtain", + "trash can", + "closet", + "stairs", + "microwave", + "stove", + "shoe", + "computer tower", + "bottle", + "bin", + "ottoman", + "bench", + "board", + "washing machine", + "mirror", + "copier", + "basket", + "sofa chair", + "file cabinet", + "fan", + "laptop", + "shower", + "paper", + "person", + "paper towel dispenser", + "oven", + "blinds", + "rack", + "plate", + "blackboard", + "piano", + "suitcase", + "rail", + "radiator", + "recycling bin", + "container", + "wardrobe", + "soap dispenser", + "telephone", + "bucket", + "clock", + "stand", + "light", + "laundry basket", + "pipe", + "clothes dryer", + "guitar", + "toilet paper holder", + "seat", + "speaker", + "column", + "ladder", + "bathroom stall", + "shower wall", + "cup", + "jacket", + "storage bin", + "coffee maker", + "dishwasher", + "paper towel roll", + "machine", + "mat", + "windowsill", + "bar", + "toaster", + "bulletin board", + "ironing board", + "fireplace", + "soap dish", + "kitchen counter", + "doorframe", + "toilet paper dispenser", + "mini fridge", + "fire extinguisher", + "ball", + "hat", + "shower curtain rod", + "water cooler", + "paper cutter", + "tray", + "shower door", + "pillar", + "ledge", + "toaster oven", + "mouse", + "toilet seat cover dispenser", + "furniture", + "cart", + "scale", + "tissue box", + "light switch", + "crate", + "power outlet", + "decoration", + "sign", + "projector", + "closet door", + "vacuum cleaner", + "plunger", + "stuffed animal", + "headphones", + "dish rack", + "broom", + "range hood", + "dustpan", + "hair dryer", + "water bottle", + "handicap bar", + "vent", + "shower floor", + "water pitcher", + "mailbox", + "bowl", + "paper bag", + "projector screen", + "divider", + "laundry detergent", + "bathroom counter", + "object", + "bathroom vanity", + "closet wall", + "laundry hamper", + "bathroom stall door", + "ceiling light", + "trash bin", + "dumbbell", + "stair rail", + "tube", + "bathroom cabinet", + "closet rod", + "coffee kettle", + "shower head", + "keyboard piano", + "case of water bottles", + "coat rack", + "folded chair", + "fire alarm", + "power strip", + "calendar", + "poster", + "potted plant", + "mattress", +) + +CLASS_LABELS_200_VALIDATION = ( + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 21, + 22, + 23, + 24, + 26, + 27, + 28, + 29, + 31, + 32, + 33, + 34, + 35, + 36, + 38, + 39, + 40, + 41, + 42, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 54, + 55, + 56, + 57, + 58, + 59, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 82, + 84, + 86, + 87, + 88, + 89, + 90, + 93, + 95, + 96, + 97, + 98, + 99, + 100, + 101, + 102, + 103, + 104, + 105, + 106, + 107, + 110, + 112, + 115, + 116, + 118, + 120, + 122, + 125, + 128, + 130, + 131, + 132, + 134, + 136, + 138, + 139, + 140, + 141, + 145, + 148, + 154, + 155, + 156, + 157, + 159, + 161, + 163, + 165, + 166, + 168, + 169, + 170, + 177, + 180, + 185, + 188, + 191, + 193, + 195, + 202, + 208, + 213, + 214, + 229, + 230, + 232, + 233, + 242, + 250, + 261, + 264, + 276, + 283, + 300, + 304, + 312, + 323, + 325, + 342, + 356, + 370, + 392, + 395, + 408, + 417, + 488, + 540, + 562, + 570, + 609, + 748, + 776, + 1156, + 1163, + 1164, + 1165, + 1166, + 1167, + 1168, + 1169, + 1170, + 1171, + 1172, + 1173, + 1175, + 1176, + 1179, + 1180, + 1181, + 1182, + 1184, + 1185, + 1186, + 1187, + 1188, + 1189, + 1191, +) + +VALID_CLASS_IDS_200_TRAIN_ONLY = ( + "bicycle", + "storage container", + "candle", + "guitar case", + "purse", + "alarm clock", + "music stand", + "cd case", + "structure", + "storage organizer", + "luggage", +) + +CLASS_LABELS_200_TRAIN_ONLY = ( + 121, + 221, + 286, + 331, + 399, + 572, + 581, + 1174, + 1178, + 1183, + 1190, +) diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannet_means.npz b/pointcept/datasets/preprocessing/scannet/meta_data/scannet_means.npz new file mode 100644 index 0000000000000000000000000000000000000000..d9bbb4f7c3b72dbe81fbeb86f594066b883fafaf --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannet_means.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:df5c2bd40e8518e982c7d7b4b39020b07ac774695038bf49cb28b44e5760457e +size 676 diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_test.txt b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_test.txt new file mode 100644 index 0000000000000000000000000000000000000000..b9e7d9205321e8ca047a527466f4b7100c9c9d2c --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_test.txt @@ -0,0 +1,312 @@ +scene0568_00 +scene0568_01 +scene0568_02 +scene0304_00 +scene0488_00 +scene0488_01 +scene0412_00 +scene0412_01 +scene0217_00 +scene0019_00 +scene0019_01 +scene0414_00 +scene0575_00 +scene0575_01 +scene0575_02 +scene0426_00 +scene0426_01 +scene0426_02 +scene0426_03 +scene0549_00 +scene0549_01 +scene0578_00 +scene0578_01 +scene0578_02 +scene0665_00 +scene0665_01 +scene0050_00 +scene0050_01 +scene0050_02 +scene0257_00 +scene0025_00 +scene0025_01 +scene0025_02 +scene0583_00 +scene0583_01 +scene0583_02 +scene0701_00 +scene0701_01 +scene0701_02 +scene0580_00 +scene0580_01 +scene0565_00 +scene0169_00 +scene0169_01 +scene0655_00 +scene0655_01 +scene0655_02 +scene0063_00 +scene0221_00 +scene0221_01 +scene0591_00 +scene0591_01 +scene0591_02 +scene0678_00 +scene0678_01 +scene0678_02 +scene0462_00 +scene0427_00 +scene0595_00 +scene0193_00 +scene0193_01 +scene0164_00 +scene0164_01 +scene0164_02 +scene0164_03 +scene0598_00 +scene0598_01 +scene0598_02 +scene0599_00 +scene0599_01 +scene0599_02 +scene0328_00 +scene0300_00 +scene0300_01 +scene0354_00 +scene0458_00 +scene0458_01 +scene0423_00 +scene0423_01 +scene0423_02 +scene0307_00 +scene0307_01 +scene0307_02 +scene0606_00 +scene0606_01 +scene0606_02 +scene0432_00 +scene0432_01 +scene0608_00 +scene0608_01 +scene0608_02 +scene0651_00 +scene0651_01 +scene0651_02 +scene0430_00 +scene0430_01 +scene0689_00 +scene0357_00 +scene0357_01 +scene0574_00 +scene0574_01 +scene0574_02 +scene0329_00 +scene0329_01 +scene0329_02 +scene0153_00 +scene0153_01 +scene0616_00 +scene0616_01 +scene0671_00 +scene0671_01 +scene0618_00 +scene0382_00 +scene0382_01 +scene0490_00 +scene0621_00 +scene0607_00 +scene0607_01 +scene0149_00 +scene0695_00 +scene0695_01 +scene0695_02 +scene0695_03 +scene0389_00 +scene0377_00 +scene0377_01 +scene0377_02 +scene0342_00 +scene0139_00 +scene0629_00 +scene0629_01 +scene0629_02 +scene0496_00 +scene0633_00 +scene0633_01 +scene0518_00 +scene0652_00 +scene0406_00 +scene0406_01 +scene0406_02 +scene0144_00 +scene0144_01 +scene0494_00 +scene0278_00 +scene0278_01 +scene0316_00 +scene0609_00 +scene0609_01 +scene0609_02 +scene0609_03 +scene0084_00 +scene0084_01 +scene0084_02 +scene0696_00 +scene0696_01 +scene0696_02 +scene0351_00 +scene0351_01 +scene0643_00 +scene0644_00 +scene0645_00 +scene0645_01 +scene0645_02 +scene0081_00 +scene0081_01 +scene0081_02 +scene0647_00 +scene0647_01 +scene0535_00 +scene0353_00 +scene0353_01 +scene0353_02 +scene0559_00 +scene0559_01 +scene0559_02 +scene0593_00 +scene0593_01 +scene0246_00 +scene0653_00 +scene0653_01 +scene0064_00 +scene0064_01 +scene0356_00 +scene0356_01 +scene0356_02 +scene0030_00 +scene0030_01 +scene0030_02 +scene0222_00 +scene0222_01 +scene0338_00 +scene0338_01 +scene0338_02 +scene0378_00 +scene0378_01 +scene0378_02 +scene0660_00 +scene0553_00 +scene0553_01 +scene0553_02 +scene0527_00 +scene0663_00 +scene0663_01 +scene0663_02 +scene0664_00 +scene0664_01 +scene0664_02 +scene0334_00 +scene0334_01 +scene0334_02 +scene0046_00 +scene0046_01 +scene0046_02 +scene0203_00 +scene0203_01 +scene0203_02 +scene0088_00 +scene0088_01 +scene0088_02 +scene0088_03 +scene0086_00 +scene0086_01 +scene0086_02 +scene0670_00 +scene0670_01 +scene0256_00 +scene0256_01 +scene0256_02 +scene0249_00 +scene0441_00 +scene0658_00 +scene0704_00 +scene0704_01 +scene0187_00 +scene0187_01 +scene0131_00 +scene0131_01 +scene0131_02 +scene0207_00 +scene0207_01 +scene0207_02 +scene0461_00 +scene0011_00 +scene0011_01 +scene0343_00 +scene0251_00 +scene0077_00 +scene0077_01 +scene0684_00 +scene0684_01 +scene0550_00 +scene0686_00 +scene0686_01 +scene0686_02 +scene0208_00 +scene0500_00 +scene0500_01 +scene0552_00 +scene0552_01 +scene0648_00 +scene0648_01 +scene0435_00 +scene0435_01 +scene0435_02 +scene0435_03 +scene0690_00 +scene0690_01 +scene0693_00 +scene0693_01 +scene0693_02 +scene0700_00 +scene0700_01 +scene0700_02 +scene0699_00 +scene0231_00 +scene0231_01 +scene0231_02 +scene0697_00 +scene0697_01 +scene0697_02 +scene0697_03 +scene0474_00 +scene0474_01 +scene0474_02 +scene0474_03 +scene0474_04 +scene0474_05 +scene0355_00 +scene0355_01 +scene0146_00 +scene0146_01 +scene0146_02 +scene0196_00 +scene0702_00 +scene0702_01 +scene0702_02 +scene0314_00 +scene0277_00 +scene0277_01 +scene0277_02 +scene0095_00 +scene0095_01 +scene0015_00 +scene0100_00 +scene0100_01 +scene0100_02 +scene0558_00 +scene0558_01 +scene0558_02 +scene0685_00 +scene0685_01 +scene0685_02 diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_train.txt b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_train.txt new file mode 100644 index 0000000000000000000000000000000000000000..7520948c8170df9ae1a9e8a40bc444fcc7cc0772 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_train.txt @@ -0,0 +1,1045 @@ +scene0191_00 +scene0191_01 +scene0191_02 +scene0119_00 +scene0230_00 +scene0528_00 +scene0528_01 +scene0705_00 +scene0705_01 +scene0705_02 +scene0415_00 +scene0415_01 +scene0415_02 +scene0007_00 +scene0141_00 +scene0141_01 +scene0141_02 +scene0515_00 +scene0515_01 +scene0515_02 +scene0447_00 +scene0447_01 +scene0447_02 +scene0531_00 +scene0503_00 +scene0285_00 +scene0069_00 +scene0584_00 +scene0584_01 +scene0584_02 +scene0581_00 +scene0581_01 +scene0581_02 +scene0620_00 +scene0620_01 +scene0263_00 +scene0263_01 +scene0481_00 +scene0481_01 +scene0020_00 +scene0020_01 +scene0291_00 +scene0291_01 +scene0291_02 +scene0469_00 +scene0469_01 +scene0469_02 +scene0659_00 +scene0659_01 +scene0024_00 +scene0024_01 +scene0024_02 +scene0564_00 +scene0117_00 +scene0027_00 +scene0027_01 +scene0027_02 +scene0028_00 +scene0330_00 +scene0418_00 +scene0418_01 +scene0418_02 +scene0233_00 +scene0233_01 +scene0673_00 +scene0673_01 +scene0673_02 +scene0673_03 +scene0673_04 +scene0673_05 +scene0585_00 +scene0585_01 +scene0362_00 +scene0362_01 +scene0362_02 +scene0362_03 +scene0035_00 +scene0035_01 +scene0358_00 +scene0358_01 +scene0358_02 +scene0037_00 +scene0194_00 +scene0321_00 +scene0293_00 +scene0293_01 +scene0623_00 +scene0623_01 +scene0592_00 +scene0592_01 +scene0569_00 +scene0569_01 +scene0413_00 +scene0313_00 +scene0313_01 +scene0313_02 +scene0480_00 +scene0480_01 +scene0401_00 +scene0517_00 +scene0517_01 +scene0517_02 +scene0032_00 +scene0032_01 +scene0613_00 +scene0613_01 +scene0613_02 +scene0306_00 +scene0306_01 +scene0052_00 +scene0052_01 +scene0052_02 +scene0053_00 +scene0444_00 +scene0444_01 +scene0055_00 +scene0055_01 +scene0055_02 +scene0560_00 +scene0589_00 +scene0589_01 +scene0589_02 +scene0610_00 +scene0610_01 +scene0610_02 +scene0364_00 +scene0364_01 +scene0383_00 +scene0383_01 +scene0383_02 +scene0006_00 +scene0006_01 +scene0006_02 +scene0275_00 +scene0451_00 +scene0451_01 +scene0451_02 +scene0451_03 +scene0451_04 +scene0451_05 +scene0135_00 +scene0065_00 +scene0065_01 +scene0065_02 +scene0104_00 +scene0674_00 +scene0674_01 +scene0448_00 +scene0448_01 +scene0448_02 +scene0502_00 +scene0502_01 +scene0502_02 +scene0440_00 +scene0440_01 +scene0440_02 +scene0071_00 +scene0072_00 +scene0072_01 +scene0072_02 +scene0509_00 +scene0509_01 +scene0509_02 +scene0649_00 +scene0649_01 +scene0602_00 +scene0694_00 +scene0694_01 +scene0101_00 +scene0101_01 +scene0101_02 +scene0101_03 +scene0101_04 +scene0101_05 +scene0218_00 +scene0218_01 +scene0579_00 +scene0579_01 +scene0579_02 +scene0039_00 +scene0039_01 +scene0493_00 +scene0493_01 +scene0242_00 +scene0242_01 +scene0242_02 +scene0083_00 +scene0083_01 +scene0127_00 +scene0127_01 +scene0662_00 +scene0662_01 +scene0662_02 +scene0018_00 +scene0087_00 +scene0087_01 +scene0087_02 +scene0332_00 +scene0332_01 +scene0332_02 +scene0628_00 +scene0628_01 +scene0628_02 +scene0134_00 +scene0134_01 +scene0134_02 +scene0238_00 +scene0238_01 +scene0092_00 +scene0092_01 +scene0092_02 +scene0092_03 +scene0092_04 +scene0022_00 +scene0022_01 +scene0467_00 +scene0392_00 +scene0392_01 +scene0392_02 +scene0424_00 +scene0424_01 +scene0424_02 +scene0646_00 +scene0646_01 +scene0646_02 +scene0098_00 +scene0098_01 +scene0044_00 +scene0044_01 +scene0044_02 +scene0510_00 +scene0510_01 +scene0510_02 +scene0571_00 +scene0571_01 +scene0166_00 +scene0166_01 +scene0166_02 +scene0563_00 +scene0172_00 +scene0172_01 +scene0388_00 +scene0388_01 +scene0215_00 +scene0215_01 +scene0252_00 +scene0287_00 +scene0668_00 +scene0572_00 +scene0572_01 +scene0572_02 +scene0026_00 +scene0224_00 +scene0113_00 +scene0113_01 +scene0551_00 +scene0381_00 +scene0381_01 +scene0381_02 +scene0371_00 +scene0371_01 +scene0460_00 +scene0118_00 +scene0118_01 +scene0118_02 +scene0417_00 +scene0008_00 +scene0634_00 +scene0521_00 +scene0123_00 +scene0123_01 +scene0123_02 +scene0045_00 +scene0045_01 +scene0511_00 +scene0511_01 +scene0114_00 +scene0114_01 +scene0114_02 +scene0070_00 +scene0029_00 +scene0029_01 +scene0029_02 +scene0129_00 +scene0103_00 +scene0103_01 +scene0002_00 +scene0002_01 +scene0132_00 +scene0132_01 +scene0132_02 +scene0124_00 +scene0124_01 +scene0143_00 +scene0143_01 +scene0143_02 +scene0604_00 +scene0604_01 +scene0604_02 +scene0507_00 +scene0105_00 +scene0105_01 +scene0105_02 +scene0428_00 +scene0428_01 +scene0311_00 +scene0140_00 +scene0140_01 +scene0182_00 +scene0182_01 +scene0182_02 +scene0142_00 +scene0142_01 +scene0399_00 +scene0399_01 +scene0012_00 +scene0012_01 +scene0012_02 +scene0060_00 +scene0060_01 +scene0370_00 +scene0370_01 +scene0370_02 +scene0310_00 +scene0310_01 +scene0310_02 +scene0661_00 +scene0650_00 +scene0152_00 +scene0152_01 +scene0152_02 +scene0158_00 +scene0158_01 +scene0158_02 +scene0482_00 +scene0482_01 +scene0600_00 +scene0600_01 +scene0600_02 +scene0393_00 +scene0393_01 +scene0393_02 +scene0562_00 +scene0174_00 +scene0174_01 +scene0157_00 +scene0157_01 +scene0161_00 +scene0161_01 +scene0161_02 +scene0159_00 +scene0254_00 +scene0254_01 +scene0115_00 +scene0115_01 +scene0115_02 +scene0162_00 +scene0163_00 +scene0163_01 +scene0523_00 +scene0523_01 +scene0523_02 +scene0459_00 +scene0459_01 +scene0175_00 +scene0085_00 +scene0085_01 +scene0279_00 +scene0279_01 +scene0279_02 +scene0201_00 +scene0201_01 +scene0201_02 +scene0283_00 +scene0456_00 +scene0456_01 +scene0429_00 +scene0043_00 +scene0043_01 +scene0419_00 +scene0419_01 +scene0419_02 +scene0368_00 +scene0368_01 +scene0348_00 +scene0348_01 +scene0348_02 +scene0442_00 +scene0178_00 +scene0380_00 +scene0380_01 +scene0380_02 +scene0165_00 +scene0165_01 +scene0165_02 +scene0181_00 +scene0181_01 +scene0181_02 +scene0181_03 +scene0333_00 +scene0614_00 +scene0614_01 +scene0614_02 +scene0404_00 +scene0404_01 +scene0404_02 +scene0185_00 +scene0126_00 +scene0126_01 +scene0126_02 +scene0519_00 +scene0236_00 +scene0236_01 +scene0189_00 +scene0075_00 +scene0267_00 +scene0192_00 +scene0192_01 +scene0192_02 +scene0281_00 +scene0420_00 +scene0420_01 +scene0420_02 +scene0195_00 +scene0195_01 +scene0195_02 +scene0597_00 +scene0597_01 +scene0597_02 +scene0041_00 +scene0041_01 +scene0111_00 +scene0111_01 +scene0111_02 +scene0666_00 +scene0666_01 +scene0666_02 +scene0200_00 +scene0200_01 +scene0200_02 +scene0536_00 +scene0536_01 +scene0536_02 +scene0390_00 +scene0280_00 +scene0280_01 +scene0280_02 +scene0344_00 +scene0344_01 +scene0205_00 +scene0205_01 +scene0205_02 +scene0484_00 +scene0484_01 +scene0009_00 +scene0009_01 +scene0009_02 +scene0302_00 +scene0302_01 +scene0209_00 +scene0209_01 +scene0209_02 +scene0210_00 +scene0210_01 +scene0395_00 +scene0395_01 +scene0395_02 +scene0683_00 +scene0601_00 +scene0601_01 +scene0214_00 +scene0214_01 +scene0214_02 +scene0477_00 +scene0477_01 +scene0439_00 +scene0439_01 +scene0468_00 +scene0468_01 +scene0468_02 +scene0546_00 +scene0466_00 +scene0466_01 +scene0220_00 +scene0220_01 +scene0220_02 +scene0122_00 +scene0122_01 +scene0130_00 +scene0110_00 +scene0110_01 +scene0110_02 +scene0327_00 +scene0156_00 +scene0266_00 +scene0266_01 +scene0001_00 +scene0001_01 +scene0228_00 +scene0199_00 +scene0219_00 +scene0464_00 +scene0232_00 +scene0232_01 +scene0232_02 +scene0299_00 +scene0299_01 +scene0530_00 +scene0363_00 +scene0453_00 +scene0453_01 +scene0570_00 +scene0570_01 +scene0570_02 +scene0183_00 +scene0239_00 +scene0239_01 +scene0239_02 +scene0373_00 +scene0373_01 +scene0241_00 +scene0241_01 +scene0241_02 +scene0188_00 +scene0622_00 +scene0622_01 +scene0244_00 +scene0244_01 +scene0691_00 +scene0691_01 +scene0206_00 +scene0206_01 +scene0206_02 +scene0247_00 +scene0247_01 +scene0061_00 +scene0061_01 +scene0082_00 +scene0250_00 +scene0250_01 +scene0250_02 +scene0501_00 +scene0501_01 +scene0501_02 +scene0320_00 +scene0320_01 +scene0320_02 +scene0320_03 +scene0631_00 +scene0631_01 +scene0631_02 +scene0255_00 +scene0255_01 +scene0255_02 +scene0047_00 +scene0265_00 +scene0265_01 +scene0265_02 +scene0004_00 +scene0336_00 +scene0336_01 +scene0058_00 +scene0058_01 +scene0260_00 +scene0260_01 +scene0260_02 +scene0243_00 +scene0603_00 +scene0603_01 +scene0093_00 +scene0093_01 +scene0093_02 +scene0109_00 +scene0109_01 +scene0434_00 +scene0434_01 +scene0434_02 +scene0290_00 +scene0627_00 +scene0627_01 +scene0470_00 +scene0470_01 +scene0137_00 +scene0137_01 +scene0137_02 +scene0270_00 +scene0270_01 +scene0270_02 +scene0271_00 +scene0271_01 +scene0504_00 +scene0274_00 +scene0274_01 +scene0274_02 +scene0036_00 +scene0036_01 +scene0276_00 +scene0276_01 +scene0272_00 +scene0272_01 +scene0499_00 +scene0698_00 +scene0698_01 +scene0051_00 +scene0051_01 +scene0051_02 +scene0051_03 +scene0108_00 +scene0245_00 +scene0369_00 +scene0369_01 +scene0369_02 +scene0284_00 +scene0289_00 +scene0289_01 +scene0286_00 +scene0286_01 +scene0286_02 +scene0286_03 +scene0031_00 +scene0031_01 +scene0031_02 +scene0545_00 +scene0545_01 +scene0545_02 +scene0557_00 +scene0557_01 +scene0557_02 +scene0533_00 +scene0533_01 +scene0116_00 +scene0116_01 +scene0116_02 +scene0611_00 +scene0611_01 +scene0688_00 +scene0294_00 +scene0294_01 +scene0294_02 +scene0295_00 +scene0295_01 +scene0296_00 +scene0296_01 +scene0596_00 +scene0596_01 +scene0596_02 +scene0532_00 +scene0532_01 +scene0637_00 +scene0638_00 +scene0121_00 +scene0121_01 +scene0121_02 +scene0040_00 +scene0040_01 +scene0197_00 +scene0197_01 +scene0197_02 +scene0410_00 +scene0410_01 +scene0305_00 +scene0305_01 +scene0615_00 +scene0615_01 +scene0703_00 +scene0703_01 +scene0555_00 +scene0297_00 +scene0297_01 +scene0297_02 +scene0582_00 +scene0582_01 +scene0582_02 +scene0023_00 +scene0094_00 +scene0013_00 +scene0013_01 +scene0013_02 +scene0136_00 +scene0136_01 +scene0136_02 +scene0407_00 +scene0407_01 +scene0062_00 +scene0062_01 +scene0062_02 +scene0386_00 +scene0318_00 +scene0554_00 +scene0554_01 +scene0497_00 +scene0213_00 +scene0258_00 +scene0323_00 +scene0323_01 +scene0324_00 +scene0324_01 +scene0016_00 +scene0016_01 +scene0016_02 +scene0681_00 +scene0398_00 +scene0398_01 +scene0227_00 +scene0090_00 +scene0066_00 +scene0262_00 +scene0262_01 +scene0155_00 +scene0155_01 +scene0155_02 +scene0352_00 +scene0352_01 +scene0352_02 +scene0038_00 +scene0038_01 +scene0038_02 +scene0335_00 +scene0335_01 +scene0335_02 +scene0261_00 +scene0261_01 +scene0261_02 +scene0261_03 +scene0640_00 +scene0640_01 +scene0640_02 +scene0080_00 +scene0080_01 +scene0080_02 +scene0403_00 +scene0403_01 +scene0282_00 +scene0282_01 +scene0282_02 +scene0682_00 +scene0173_00 +scene0173_01 +scene0173_02 +scene0522_00 +scene0687_00 +scene0345_00 +scene0345_01 +scene0612_00 +scene0612_01 +scene0411_00 +scene0411_01 +scene0411_02 +scene0625_00 +scene0625_01 +scene0211_00 +scene0211_01 +scene0211_02 +scene0211_03 +scene0676_00 +scene0676_01 +scene0179_00 +scene0498_00 +scene0498_01 +scene0498_02 +scene0547_00 +scene0547_01 +scene0547_02 +scene0269_00 +scene0269_01 +scene0269_02 +scene0366_00 +scene0680_00 +scene0680_01 +scene0588_00 +scene0588_01 +scene0588_02 +scene0588_03 +scene0346_00 +scene0346_01 +scene0359_00 +scene0359_01 +scene0014_00 +scene0120_00 +scene0120_01 +scene0212_00 +scene0212_01 +scene0212_02 +scene0176_00 +scene0049_00 +scene0259_00 +scene0259_01 +scene0586_00 +scene0586_01 +scene0586_02 +scene0309_00 +scene0309_01 +scene0125_00 +scene0455_00 +scene0177_00 +scene0177_01 +scene0177_02 +scene0326_00 +scene0372_00 +scene0171_00 +scene0171_01 +scene0374_00 +scene0654_00 +scene0654_01 +scene0445_00 +scene0445_01 +scene0475_00 +scene0475_01 +scene0475_02 +scene0349_00 +scene0349_01 +scene0234_00 +scene0669_00 +scene0669_01 +scene0375_00 +scene0375_01 +scene0375_02 +scene0387_00 +scene0387_01 +scene0387_02 +scene0312_00 +scene0312_01 +scene0312_02 +scene0384_00 +scene0385_00 +scene0385_01 +scene0385_02 +scene0000_00 +scene0000_01 +scene0000_02 +scene0376_00 +scene0376_01 +scene0376_02 +scene0301_00 +scene0301_01 +scene0301_02 +scene0322_00 +scene0542_00 +scene0079_00 +scene0079_01 +scene0099_00 +scene0099_01 +scene0476_00 +scene0476_01 +scene0476_02 +scene0394_00 +scene0394_01 +scene0147_00 +scene0147_01 +scene0067_00 +scene0067_01 +scene0067_02 +scene0397_00 +scene0397_01 +scene0337_00 +scene0337_01 +scene0337_02 +scene0431_00 +scene0223_00 +scene0223_01 +scene0223_02 +scene0010_00 +scene0010_01 +scene0402_00 +scene0268_00 +scene0268_01 +scene0268_02 +scene0679_00 +scene0679_01 +scene0405_00 +scene0128_00 +scene0408_00 +scene0408_01 +scene0190_00 +scene0107_00 +scene0076_00 +scene0167_00 +scene0361_00 +scene0361_01 +scene0361_02 +scene0216_00 +scene0202_00 +scene0303_00 +scene0303_01 +scene0303_02 +scene0446_00 +scene0446_01 +scene0089_00 +scene0089_01 +scene0089_02 +scene0360_00 +scene0150_00 +scene0150_01 +scene0150_02 +scene0421_00 +scene0421_01 +scene0421_02 +scene0454_00 +scene0626_00 +scene0626_01 +scene0626_02 +scene0186_00 +scene0186_01 +scene0538_00 +scene0479_00 +scene0479_01 +scene0479_02 +scene0656_00 +scene0656_01 +scene0656_02 +scene0656_03 +scene0525_00 +scene0525_01 +scene0525_02 +scene0308_00 +scene0396_00 +scene0396_01 +scene0396_02 +scene0624_00 +scene0292_00 +scene0292_01 +scene0632_00 +scene0253_00 +scene0021_00 +scene0325_00 +scene0325_01 +scene0437_00 +scene0437_01 +scene0438_00 +scene0590_00 +scene0590_01 +scene0400_00 +scene0400_01 +scene0541_00 +scene0541_01 +scene0541_02 +scene0677_00 +scene0677_01 +scene0677_02 +scene0443_00 +scene0315_00 +scene0288_00 +scene0288_01 +scene0288_02 +scene0422_00 +scene0672_00 +scene0672_01 +scene0184_00 +scene0449_00 +scene0449_01 +scene0449_02 +scene0048_00 +scene0048_01 +scene0138_00 +scene0452_00 +scene0452_01 +scene0452_02 +scene0667_00 +scene0667_01 +scene0667_02 +scene0463_00 +scene0463_01 +scene0078_00 +scene0078_01 +scene0078_02 +scene0636_00 +scene0457_00 +scene0457_01 +scene0457_02 +scene0465_00 +scene0465_01 +scene0577_00 +scene0151_00 +scene0151_01 +scene0339_00 +scene0573_00 +scene0573_01 +scene0154_00 +scene0096_00 +scene0096_01 +scene0096_02 +scene0235_00 +scene0168_00 +scene0168_01 +scene0168_02 +scene0594_00 +scene0587_00 +scene0587_01 +scene0587_02 +scene0587_03 +scene0229_00 +scene0229_01 +scene0229_02 +scene0512_00 +scene0106_00 +scene0106_01 +scene0106_02 +scene0472_00 +scene0472_01 +scene0472_02 +scene0489_00 +scene0489_01 +scene0489_02 +scene0425_00 +scene0425_01 +scene0641_00 +scene0526_00 +scene0526_01 +scene0317_00 +scene0317_01 +scene0544_00 +scene0017_00 +scene0017_01 +scene0017_02 +scene0042_00 +scene0042_01 +scene0042_02 +scene0576_00 +scene0576_01 +scene0576_02 +scene0347_00 +scene0347_01 +scene0347_02 +scene0436_00 +scene0226_00 +scene0226_01 +scene0485_00 +scene0486_00 +scene0487_00 +scene0487_01 +scene0619_00 +scene0097_00 +scene0367_00 +scene0367_01 +scene0491_00 +scene0492_00 +scene0492_01 +scene0005_00 +scene0005_01 +scene0543_00 +scene0543_01 +scene0543_02 +scene0657_00 +scene0341_00 +scene0341_01 diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_val.txt b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_val.txt new file mode 100644 index 0000000000000000000000000000000000000000..965ff258035f857446c30b10e9a6be49f71d3dc7 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv1_val.txt @@ -0,0 +1,156 @@ +scene0534_00 +scene0534_01 +scene0319_00 +scene0273_00 +scene0273_01 +scene0225_00 +scene0198_00 +scene0003_00 +scene0003_01 +scene0003_02 +scene0409_00 +scene0409_01 +scene0331_00 +scene0331_01 +scene0505_00 +scene0505_01 +scene0505_02 +scene0505_03 +scene0505_04 +scene0506_00 +scene0057_00 +scene0057_01 +scene0074_00 +scene0074_01 +scene0074_02 +scene0091_00 +scene0112_00 +scene0112_01 +scene0112_02 +scene0240_00 +scene0102_00 +scene0102_01 +scene0513_00 +scene0514_00 +scene0514_01 +scene0537_00 +scene0516_00 +scene0516_01 +scene0495_00 +scene0617_00 +scene0133_00 +scene0520_00 +scene0520_01 +scene0635_00 +scene0635_01 +scene0054_00 +scene0473_00 +scene0473_01 +scene0524_00 +scene0524_01 +scene0379_00 +scene0471_00 +scene0471_01 +scene0471_02 +scene0566_00 +scene0248_00 +scene0248_01 +scene0248_02 +scene0529_00 +scene0529_01 +scene0529_02 +scene0391_00 +scene0264_00 +scene0264_01 +scene0264_02 +scene0675_00 +scene0675_01 +scene0350_00 +scene0350_01 +scene0350_02 +scene0450_00 +scene0068_00 +scene0068_01 +scene0237_00 +scene0237_01 +scene0365_00 +scene0365_01 +scene0365_02 +scene0605_00 +scene0605_01 +scene0539_00 +scene0539_01 +scene0539_02 +scene0540_00 +scene0540_01 +scene0540_02 +scene0170_00 +scene0170_01 +scene0170_02 +scene0433_00 +scene0340_00 +scene0340_01 +scene0340_02 +scene0160_00 +scene0160_01 +scene0160_02 +scene0160_03 +scene0160_04 +scene0059_00 +scene0059_01 +scene0059_02 +scene0056_00 +scene0056_01 +scene0478_00 +scene0478_01 +scene0548_00 +scene0548_01 +scene0548_02 +scene0204_00 +scene0204_01 +scene0204_02 +scene0033_00 +scene0145_00 +scene0483_00 +scene0508_00 +scene0508_01 +scene0508_02 +scene0180_00 +scene0148_00 +scene0556_00 +scene0556_01 +scene0416_00 +scene0416_01 +scene0416_02 +scene0416_03 +scene0416_04 +scene0073_00 +scene0073_01 +scene0073_02 +scene0073_03 +scene0034_00 +scene0034_01 +scene0034_02 +scene0639_00 +scene0561_00 +scene0561_01 +scene0298_00 +scene0692_00 +scene0692_01 +scene0692_02 +scene0692_03 +scene0692_04 +scene0642_00 +scene0642_01 +scene0642_02 +scene0642_03 +scene0630_00 +scene0630_01 +scene0630_02 +scene0630_03 +scene0630_04 +scene0630_05 +scene0630_06 +scene0706_00 +scene0567_00 +scene0567_01 diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2-labels-old.combined.tsv b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2-labels-old.combined.tsv new file mode 100644 index 0000000000000000000000000000000000000000..05c006e98066aa78d126bebcfb3654200d351b93 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2-labels-old.combined.tsv @@ -0,0 +1,608 @@ +id raw_category category count nyu40id eigen13id nyuClass nyu40class eigen13class ModelNet40 ModelNet10 ShapeNetCore55 synsetoffset wnsynsetid wnsynsetkey mpcat40 mpcat40index +1 wall wall 8277 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +2 chair chair 4646 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +22 books book 1678 23 2 book books Books n02870526 book.n.11 objects 39 +3 floor floor 1553 2 5 floor floor Floor n03365592 floor.n.01 floor 2 +5 door door 1483 8 12 door door Wall door n03221720 door.n.01 door 4 +1163 object object 1313 40 7 otherprop Objects objects 39 +16 window window 1209 9 13 window window Window n04587648 window.n.01 window 9 +4 table table 1170 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +56 trash can trash can 1090 39 6 garbage bin otherfurniture Furniture trash_bin 2747177 n02747177 ashcan.n.01 objects 39 +13 pillow pillow 937 18 7 pillow pillow Objects pillow 3938244 n03938244 pillow.n.01 cushion 8 +15 picture picture 862 11 8 picture picture Picture n03931044 picture.n.01 picture 6 +41 ceiling ceiling 806 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 ceiling 17 +26 box box 775 29 7 box box Objects n02883344 box.n.01 objects 39 +161 doorframe doorframe 768 8 12 door door Wall door doorframe.n.01 door 4 +19 monitor monitor 765 40 7 monitor otherprop Objects monitor monitor tv or monitor 3211117 n03782190 monitor.n.04 objects 39 +7 cabinet cabinet 731 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +9 desk desk 680 14 10 desk desk Table desk desk table 4379243 n03179701 desk.n.01 table 5 +8 shelf shelf 641 15 6 shelves shelves Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +10 office chair office chair 595 5 4 chair chair Chair chair chair chair 3001627 n04373704 swivel_chair.n.01 chair 3 +31 towel towel 570 27 7 towel towel Objects n04459362 towel.n.01 towel 20 +6 couch couch 502 6 9 sofa sofa Sofa sofa sofa sofa 4256520 n04256520 sofa.n.01 sofa 10 +14 sink sink 488 34 7 sink sink Objects sink n04223580 sink.n.01 sink 15 +48 backpack backpack 479 40 7 backpack otherprop Objects n02769748 backpack.n.01 objects 39 +28 lamp lamp 419 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +11 bed bed 370 4 1 bed bed Bed bed bed bed 2818832 n02818832 bed.n.01 bed 11 +18 bookshelf bookshelf 360 10 6 bookshelf bookshelf Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +71 mirror mirror 349 19 7 mirror mirror Objects n03773035 mirror.n.01 mirror 21 +21 curtain curtain 347 16 13 curtain curtain Window curtain n03151077 curtain.n.01 curtain 12 +40 plant plant 331 40 7 plant otherprop Objects plant n00017222 plant.n.02 plant 14 +52 whiteboard whiteboard 327 30 7 whiteboard whiteboard Objects n03211616 display_panel.n.01 board_panel 35 +96 radiator radiator 322 39 6 radiator otherfurniture Furniture n04041069 radiator.n.02 misc 40 +22 book book 318 23 2 book books Books n02870526 book.n.11 objects 39 +29 kitchen cabinet kitchen cabinet 310 3 6 cabinet cabinet Furniture n02933112 cabinet.n.01 cabinet 7 +49 toilet paper toilet paper 291 40 7 toilet paper otherprop Objects n15075141 toilet_tissue.n.01 objects 39 +29 kitchen cabinets kitchen cabinet 289 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +23 armchair armchair 281 5 4 chair chair Chair chair chair chair 3001627 n02738535 armchair.n.01 chair 3 +63 shoes shoe 272 40 7 shoe otherprop Objects n04199027 shoe.n.01 clothes 38 +24 coffee table coffee table 258 7 10 coffee table table Table table table table 4379243 n03063968 coffee_table.n.01 table 5 +17 toilet toilet 256 33 7 toilet toilet Objects toilet toilet n04446276 toilet.n.01 toilet 18 +47 bag bag 252 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +32 clothes clothes 248 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +46 keyboard keyboard 246 40 7 keyboard otherprop Objects keyboard computer keyboard 3085013 n03085013 computer_keyboard.n.01 objects 39 +65 bottle bottle 226 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +97 recycling bin recycling bin 225 39 6 garbage bin otherfurniture Furniture trash_bin 2747177 n02747177 ashcan.n.01 objects 39 +34 nightstand nightstand 224 32 6 night stand night stand Furniture night_stand night_stand n03015254 chest_of_drawers.n.01 chest_of_drawers 13 +38 stool stool 221 40 7 stool otherprop Objects stool n04326896 stool.n.01 stool 19 +33 tv tv 219 25 11 television television TV tv or monitor 3211117 n03211117 display.n.06 tv_monitor 22 +75 file cabinet file cabinet 217 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +36 dresser dresser 213 17 6 dresser dresser Furniture dresser dresser n03015254 chest_of_drawers.n.01 chest_of_drawers 13 +64 computer tower computer tower 203 40 7 computer otherprop Objects n03082979 computer.n.01 objects 39 +32 clothing clothes 165 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +101 telephone telephone 164 40 7 telephone otherprop Objects telephone 4401088 n04401088 telephone.n.01 objects 39 +130 cup cup 157 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 +27 refrigerator refrigerator 154 24 6 refridgerator refridgerator Furniture n04070727 refrigerator.n.01 appliances 37 +44 end table end table 147 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +131 jacket jacket 146 40 7 jacket otherprop Objects n03589791 jacket.n.01 clothes 38 +55 shower curtain shower curtain 144 28 7 shower curtain shower curtain Objects curtain n04209239 shower_curtain.n.01 curtain 12 +42 bathtub bathtub 144 36 7 bathtub bathtub Objects bathtub bathtub tub 2808440 n02808440 bathtub.n.01 bathtub 25 +59 microwave microwave 141 40 7 microwave otherprop Objects microwave 3761084 n03761084 microwave.n.02 appliances 37 +159 kitchen counter kitchen counter 140 12 6 counter counter Furniture table table table 4379243 n03116530 counter.n.01 counter 26 +74 sofa chair sofa chair 129 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +82 paper towel dispenser paper towel dispenser 129 40 7 paper towel dispenser otherprop Objects objects 39 +1164 bathroom vanity bathroom vanity 126 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 table 5 +93 suitcase suitcase 118 40 7 luggage otherprop Objects n02773838 bag.n.06 objects 39 +77 laptop laptop 111 40 7 laptop otherprop Objects laptop laptop 3642806 n03642806 laptop.n.01 objects 39 +67 ottoman ottoman 111 39 6 ottoman otherfurniture Furniture stool n03380724 footstool.n.01 stool 19 +128 shower walls shower wall 109 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +50 printer printer 106 40 7 printer otherprop Objects printer 4004475 n04004475 printer.n.03 appliances 37 +35 counter counter 104 12 6 counter counter Furniture table table table 4379243 n03116530 counter.n.01 counter 26 +69 board board 100 38 7 board otherstructure Objects board_panel 35 +100 soap dispenser soap dispenser 99 40 7 otherprop Objects n04254120 soap_dispenser.n.01 objects 39 +62 stove stove 95 38 7 stove otherstructure Objects stove 4330267 n04330267 stove.n.02 appliances 37 +105 light light 93 38 7 light otherstructure Objects n03665366 light.n.02 lighting 28 +1165 closet wall closet wall 90 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +165 mini fridge mini fridge 87 24 6 refridgerator refridgerator Furniture n03273913 electric_refrigerator.n.01 appliances 37 +7 cabinets cabinet 79 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +5 doors door 76 8 12 door door Wall door n03221720 door.n.01 door 4 +76 fan fan 75 40 7 fan otherprop Objects n03320046 fan.n.01 misc 40 +230 tissue box tissue box 73 40 7 tissue box otherprop Objects n02883344 box.n.01 objects 39 +54 blanket blanket 72 40 7 blanket otherprop Objects n02849154 blanket.n.01 objects 39 +125 bathroom stall bathroom stall 71 38 7 otherstructure Objects n02873839 booth.n.02 misc 40 +72 copier copier 70 40 7 otherprop Objects n03257586 duplicator.n.01 appliances 37 +68 bench bench 66 39 6 bench otherfurniture Furniture bench bench 2828884 n02828884 bench.n.01 seating 34 +145 bar bar 66 38 7 bar otherstructure Objects n02788689 bar.n.03 misc 40 +157 soap dish soap dish 65 40 7 soap dish otherprop Objects n04254009 soap_dish.n.01 objects 39 +1166 laundry hamper laundry hamper 65 40 7 laundry basket otherprop Objects objects 39 +132 storage bin storage bin 63 40 7 storage bin otherprop Objects objects 39 +1167 bathroom stall door bathroom stall door 62 8 12 door door Wall door n03221720 door.n.01 door 4 +232 light switch light switch 61 38 7 light switch otherstructure Objects n04372370 switch.n.01 misc 40 +134 coffee maker coffee maker 61 40 7 otherprop Objects n03063338 coffee_maker.n.01 appliances 37 +51 tv stand tv stand 61 39 6 tv stand otherfurniture Furniture tv_stand n03290653 entertainment_center.n.01 furniture 36 +250 decoration decoration 60 40 7 otherprop Objects n03169390 decoration.n.01 misc 40 +1168 ceiling light ceiling light 59 38 7 light otherstructure Objects n03665366 light.n.02 lighting 28 +342 range hood range hood 59 38 7 range hood otherstructure Objects range_hood n04053677 range_hood.n.01 misc 40 +89 blackboard blackboard 58 38 7 blackboard otherstructure Objects n02846511 blackboard.n.01 board_panel 35 +103 clock clock 58 40 7 clock otherprop Objects clock 3046257 n03046257 clock.n.01 objects 39 +99 wardrobe closet wardrobe 54 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 furniture 36 +95 rail rail 53 38 7 railing otherstructure Objects n04047401 railing.n.01 railing 30 +154 bulletin board bulletin board 53 38 7 board otherstructure Objects n03211616 display_panel.n.01 board_panel 35 +140 mat mat 52 20 5 floor mat floor mat Floor n03727837 mat.n.01 floor 2 +1169 trash bin trash bin 52 39 6 garbage bin otherfurniture Furniture trash_bin 2747177 n02747177 ashcan.n.01 objects 39 +193 ledge ledge 51 38 7 otherstructure Objects n09337253 ledge.n.01 misc 40 +116 seat seat 49 39 6 furniture otherfurniture Furniture n04161981 seat.n.03 furniture 36 +202 mouse mouse 49 40 7 mouse otherprop Objects n03793489 mouse.n.04 objects 39 +73 basket basket 48 40 7 basket otherprop Objects basket 2801938 n02801938 basket.n.01 objects 39 +78 shower shower 48 38 7 otherstructure Objects n04208936 shower.n.01 shower 23 +1170 dumbbell dumbbell 48 40 7 otherprop Objects n03255030 dumbbell.n.01 objects 39 +79 paper paper 46 26 7 paper paper Objects n14974264 paper.n.01 objects 39 +80 person person 46 31 7 person person Objects person n05217688 person.n.02 misc 40 +141 windowsill windowsill 45 38 7 otherstructure Objects n04590263 windowsill.n.01 window 9 +57 closet closet 45 39 6 wardrobe otherfurniture Furniture wardrobe misc 40 +102 bucket bucket 45 40 7 bucket otherprop Objects n02909870 bucket.n.01 misc 40 +261 sign sign 44 40 7 sign otherprop Objects n04217882 signboard.n.01 objects 39 +118 speaker speaker 43 40 7 speaker otherprop Objects speaker 3691459 n03691459 loudspeaker.n.01 objects 39 +136 dishwasher dishwasher 43 38 7 dishwasher otherstructure Objects dishwasher 3207941 n03207941 dishwasher.n.01 appliances 37 +98 container container 43 40 7 container otherprop Objects n03094503 container.n.01 objects 39 +1171 stair rail stair rail 42 38 7 banister otherstructure Objects n02788148 bannister.n.02 railing 30 +170 shower curtain rod shower curtain rod 42 40 7 otherprop Objects curtain 12 +1172 tube tube 41 40 7 otherprop Objects misc 40 +1173 bathroom cabinet bathroom cabinet 39 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +79 papers paper 39 26 7 paper paper Objects n14974264 paper.n.01 objects 39 +221 storage container storage container 39 40 7 container otherprop Objects objects 39 +570 paper bag paper bag 39 37 7 bag bag Objects n04122825 sack.n.01 objects 39 +138 paper towel roll paper towel roll 39 40 7 paper towel otherprop Objects n03887697 paper_towel.n.01 towel 20 +168 ball ball 39 40 7 ball otherprop Objects objects 39 +276 closet doors closet door 38 8 12 door door Wall door n03221720 door.n.01 door 4 +106 laundry basket laundry basket 37 40 7 laundry basket otherprop Objects basket 2801938 n03050864 clothes_hamper.n.01 objects 39 +214 cart cart 37 40 7 cart otherprop Objects n03484083 handcart.n.01 shelving 31 +276 closet door closet door 35 8 12 door door Wall door n03221720 door.n.01 door 4 +323 dish rack dish rack 35 40 7 dish rack otherprop Objects n03207630 dish_rack.n.01 objects 39 +58 stairs stairs 35 38 7 stairs otherstructure Objects n04298308 stairway.n.01 stairs 16 +86 blinds blinds 35 13 13 blinds blinds Window n02851099 blind.n.03 blinds 32 +2 stack of chairs chair 35 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +399 purse purse 34 40 7 purse otherprop Objects n02774152 bag.n.04 objects 39 +121 bicycle bicycle 33 40 7 bicycle otherprop Objects bicycle 2834778 n02834778 bicycle.n.01 objects 39 +185 tray tray 32 40 7 tray otherprop Objects n04476259 tray.n.01 objects 39 +300 plunger plunger 30 40 7 otherprop Objects n03970156 plunger.n.03 objects 39 +180 paper cutter paper cutter 30 40 7 paper cutter otherprop Objects n03886940 paper_cutter.n.01 objects 39 +163 toilet paper dispenser toilet paper dispenser 29 40 7 otherprop Objects objects 39 +26 boxes box 29 29 7 box box Objects n02883344 box.n.01 objects 39 +66 bin bin 28 40 7 bin otherprop Objects n02839910 bin.n.01 objects 39 +208 toilet seat cover dispenser toilet seat cover dispenser 28 40 7 otherprop Objects objects 39 +112 guitar guitar 28 40 7 guitar otherprop Objects guitar guitar 3467517 n03467517 guitar.n.01 objects 39 +540 mailboxes mailbox 28 29 7 box box Objects mailbox 3710193 n03710193 mailbox.n.01 misc 40 +395 handicap bar handicap bar 27 38 7 bar otherstructure Objects misc 40 +166 fire extinguisher fire extinguisher 27 40 7 fire extinguisher otherprop Objects n03345837 fire_extinguisher.n.01 misc 40 +122 ladder ladder 27 39 6 ladder otherfurniture Furniture stairs n03632277 ladder.n.01 stairs 16 +120 column column 26 38 7 column otherstructure Objects n03074380 column.n.06 column 24 +107 pipe pipe 25 40 7 pipe otherprop Objects n03944672 pipe.n.02 misc 40 +283 vacuum cleaner vacuum cleaner 25 40 7 otherprop Objects n04517823 vacuum.n.04 objects 39 +88 plate plate 24 40 7 plate otherprop Objects n03959485 plate.n.04 objects 39 +90 piano piano 24 39 6 piano otherfurniture Furniture piano piano 3928116 n03928116 piano.n.01 furniture 36 +177 water cooler water cooler 24 39 6 water cooler otherfurniture Furniture n04559166 water_cooler.n.01 misc 40 +1174 cd case cd case 24 40 7 otherprop Objects objects 39 +562 bowl bowl 24 40 7 bowl otherprop Objects bowl bowl 2880940 n02880940 bowl.n.03 objects 39 +1175 closet rod closet rod 24 40 7 otherprop Objects n04100174 rod.n.01 misc 40 +1156 bathroom counter bathroom counter 24 12 6 counter counter Furniture table table table 4379243 n03116530 counter.n.01 counter 26 +84 oven oven 23 38 7 oven otherstructure Objects n03862676 oven.n.01 appliances 37 +104 stand stand 23 39 6 stand otherfurniture Furniture table table table 4379243 n04301000 stand.n.04 table 5 +229 scale scale 23 40 7 scale otherprop Objects n04141975 scale.n.07 objects 39 +70 washing machine washing machine 23 39 6 washing machine otherfurniture Furniture washing_machine 4554684 n04554684 washer.n.03 appliances 37 +325 broom broom 22 40 7 broom otherprop Objects n02906734 broom.n.01 objects 39 +169 hat hat 22 40 7 hat otherprop Objects n03497657 hat.n.01 clothes 38 +128 shower wall shower wall 22 1 12 wall wall Wall n04208936 shower.n.01 wall 1 +331 guitar case guitar case 21 40 7 guitar case otherprop Objects objects 39 +87 rack rack 21 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +488 water pitcher water pitcher 21 40 7 pitcher otherprop Objects n03950228 pitcher.n.02 objects 39 +776 laundry detergent laundry detergent 21 40 7 otherprop Objects objects 39 +370 hair dryer hair dryer 21 40 7 hair dryer otherprop Objects n03483316 hand_blower.n.01 objects 39 +191 pillar pillar 21 38 7 column otherstructure Objects n03073977 column.n.07 column 24 +748 divider divider 20 40 7 otherprop Objects wall 1 +242 power outlet power outlet 19 40 7 otherprop Objects misc 40 +45 dining table dining table 19 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +417 shower floor shower floor 19 2 5 floor floor Floor n04208936 shower.n.01 floor 2 +70 washing machines washing machine 19 39 6 washing machine otherfurniture Furniture washing_machine 4554684 n04554684 washer.n.03 appliances 37 +188 shower door shower door 19 8 12 door door Wall door n04208936 shower.n.01 door 4 +1176 coffee kettle coffee kettle 18 40 7 pot otherprop Objects n03612814 kettle.n.01 objects 39 +1177 wardrobe cabinet wardrobe 18 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 furniture 36 +1178 structure structure 18 38 7 otherstructure Objects misc 40 +18 bookshelves bookshelf 17 10 6 bookshelf bookshelf Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +110 clothes dryer clothes dryer 17 39 6 otherfurniture Furniture n03251766 dryer.n.01 appliances 37 +148 toaster toaster 17 40 7 toaster otherprop Objects n04442312 toaster.n.02 appliances 37 +63 shoe shoe 17 40 7 shoe otherprop Objects n04199027 shoe.n.01 clothes 38 +155 ironing board ironing board 16 39 6 ironing board otherfurniture Furniture n03586090 ironing_board.n.01 objects 39 +572 alarm clock alarm clock 16 40 7 alarm clock otherprop Objects clock 3046257 n02694662 alarm_clock.n.01 objects 39 +1179 shower head shower head 15 38 7 otherstructure Objects shower 23 +28 lamp base lamp 15 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +392 water bottle water bottle 15 40 7 bottle otherprop Objects bottle bottle 2876657 n04557648 water_bottle.n.01 objects 39 +1180 keyboard piano keyboard piano 15 39 6 piano otherfurniture Furniture piano piano 3928116 n03928116 piano.n.01 furniture 36 +609 projector screen projector screen 15 38 7 projector screen otherstructure Objects misc 40 +1181 case of water bottles case of water bottles 15 40 7 otherprop Objects objects 39 +195 toaster oven toaster oven 14 40 7 toaster oven otherprop Objects n04442441 toaster_oven.n.01 appliances 37 +581 music stand music stand 14 39 6 music stand otherfurniture Furniture n03801760 music_stand.n.01 furniture 36 +58 staircase stairs 14 38 7 stairs otherstructure Objects n04298308 stairway.n.01 stairs 16 +1182 coat rack coat rack 14 40 7 otherprop Objects n03059103 coatrack.n.01 shelving 3 +1183 storage organizer storage organizer 14 40 7 otherprop Objects shelving 3 +139 machine machine 14 40 7 machine otherprop Objects n03699975 machine.n.01 appliances 37 +1184 folded chair folded chair 14 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +1185 fire alarm fire alarm 14 40 7 otherprop Objects n03343737 fire_alarm.n.02 misc 40 +156 fireplace fireplace 13 38 7 fireplace otherstructure Objects n03346455 fireplace.n.01 fireplace 27 +408 vent vent 13 40 7 otherprop Objects n04526241 vent.n.01 misc 40 +213 furniture furniture 13 39 6 furniture otherfurniture Furniture n03405725 furniture.n.01 furniture 36 +1186 power strip power strip 13 40 7 otherprop Objects objects 39 +1187 calendar calendar 13 40 7 otherprop Objects objects 39 +1188 poster poster 13 11 8 picture picture Picture n03931044 picture.n.01 picture 6 +115 toilet paper holder toilet paper holder 13 40 7 toilet paper holder otherprop Objects objects 39 +1189 potted plant potted plant 12 40 7 plant otherprop Objects plant n00017222 plant.n.02 plant 14 +304 stuffed animal stuffed animal 12 40 7 stuffed animal otherprop Objects n04399382 teddy.n.01 objects 39 +1190 luggage luggage 12 40 7 luggage otherprop Objects n02774630 baggage.n.01 objects 39 +21 curtains curtain 12 16 13 curtain curtain Window curtain n03151077 curtain.n.01 curtain 12 +312 headphones headphones 12 40 7 otherprop Objects n03261776 earphone.n.01 objects 39 +233 crate crate 12 39 6 crate otherfurniture Furniture n03127925 crate.n.01 objects 39 +286 candle candle 12 40 7 candle otherprop Objects lamp n02948072 candle.n.01 objects 39 +264 projector projector 12 40 7 projector otherprop Objects n04009552 projector.n.02 objects 39 +110 clothes dryers clothes dryer 12 39 6 otherfurniture Furniture n03251766 dryer.n.01 appliances 37 +1191 mattress mattress 12 4 1 bed bed Bed bed bed bed 2818832 n02818832 bed.n.01 bed 11 +356 dustpan dustpan 12 40 7 otherprop Objects n03259009 dustpan.n.02 objects 39 +25 drawer drawer 11 39 6 drawer otherfurniture Furniture n03233905 drawer.n.01 furniture 36 +750 rod rod 11 40 7 otherprop Objects pistol 3948459 n03427202 gat.n.01 misc 40 +269 globe globe 11 40 7 globe otherprop Objects objects 39 +307 footrest footrest 11 39 6 foot rest otherfurniture Furniture stool n03380724 footstool.n.01 stool 19 +410 piano bench piano bench 11 39 6 piano bench otherfurniture Furniture bench bench 2828884 n02828884 bench.n.01 seating 34 +730 breakfast bar breakfast bar 11 38 7 bar otherstructure Objects counter 26 +216 step stool step stool 11 40 7 step stool otherprop Objects stool n04315713 step_stool.n.01 stool 19 +1192 hand rail hand rail 11 38 7 railing otherstructure Objects railing 30 +119 vending machine vending machine 11 40 7 machine otherprop Objects n04525305 vending_machine.n.01 appliances 37 +682 ceiling fan ceiling fan 11 40 7 fan otherprop Objects n03320046 fan.n.01 misc 40 +434 swiffer swiffer 11 40 7 otherprop Objects objects 39 +126 foosball table foosball table 11 39 6 foosball table otherfurniture Furniture table table table 4379243 n04379243 table.n.02 table 5 +919 jar jar 11 40 7 jar otherprop Objects jar 3593526 n03593526 jar.n.01 objects 39 +85 footstool footstool 11 39 6 ottoman otherfurniture Furniture stool n03380724 footstool.n.01 stool 19 +1193 folded table folded table 10 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +108 round table round table 10 7 10 table table Table table table table 4379243 n04114554 round_table.n.02 table 5 +135 hamper hamper 10 40 7 basket otherprop Objects basket 2801938 n03482405 hamper.n.02 objects 39 +1194 poster tube poster tube 10 40 7 otherprop Objects objects 39 +432 case case 10 40 7 case otherprop Objects objects 39 +53 carpet carpet 10 40 7 rug otherprop Objects n04118021 rug.n.01 floor 2 +1195 thermostat thermostat 10 40 7 otherprop Objects n04422875 thermostat.n.01 misc 40 +111 coat coat 10 40 7 jacket otherprop Objects n03057021 coat.n.01 clothes 38 +305 water fountain water fountain 10 38 7 water fountain otherstructure Objects n03241335 drinking_fountain.n.01 misc 40 +1125 smoke detector smoke detector 10 40 7 otherprop Objects misc 40 +13 pillows pillow 9 18 7 pillow pillow Objects pillow 3938244 n03938244 pillow.n.01 cushion 8 +1196 flip flops flip flops 9 40 7 shoe otherprop Objects n04199027 shoe.n.01 clothes 38 +1197 cloth cloth 9 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +1198 banner banner 9 40 7 otherprop Objects n02788021 banner.n.01 misc 40 +1199 clothes hanger clothes hanger 9 40 7 otherprop Objects n03057920 coat_hanger.n.01 objects 39 +1200 whiteboard eraser whiteboard eraser 9 40 7 otherprop Objects objects 39 +378 iron iron 9 40 7 otherprop Objects n03584829 iron.n.04 objects 39 +591 instrument case instrument case 9 40 7 case otherprop Objects objects 39 +49 toilet paper rolls toilet paper 9 40 7 toilet paper otherprop Objects n15075141 toilet_tissue.n.01 objects 39 +92 soap soap 9 40 7 soap otherprop Objects n04253437 soap.n.01 objects 39 +1098 block block 9 40 7 otherprop Objects misc 40 +291 wall hanging wall hanging 8 40 7 otherprop Objects n03491178 hanging.n.01 picture 6 +1063 kitchen island kitchen island 8 38 7 kitchen island otherstructure Objects n03620600 kitchen_island.n.01 counter 26 +107 pipes pipe 8 38 7 otherstructure Objects misc 40 +1135 toothbrush toothbrush 8 40 7 toothbrush otherprop Objects n04453156 toothbrush.n.01 objects 39 +189 shirt shirt 8 40 7 otherprop Objects n04197391 shirt.n.01 clothes 38 +245 cutting board cutting board 8 40 7 cutting board otherprop Objects n03025513 chopping_board.n.01 objects 39 +194 vase vase 8 40 7 vase otherprop Objects vase jar 3593526 n04522168 vase.n.01 objects 39 +1201 shower control valve shower control valve 8 38 7 otherstructure Objects n04208936 shower.n.01 shower 23 +386 exercise machine exercise machine 8 40 7 machine otherprop Objects gym_equipment 33 +1202 compost bin compost bin 8 39 6 garbage bin otherfurniture Furniture trash_bin 2747177 n02747177 ashcan.n.01 objects 39 +857 shorts shorts 8 40 7 shorts otherprop Objects clothes 38 +452 tire tire 8 40 7 otherprop Objects n04440749 tire.n.01 objects 39 +1203 teddy bear teddy bear 7 40 7 stuffed animal otherprop Objects n04399382 teddy.n.01 objects 39 +346 bathrobe bathrobe 7 40 7 otherprop Objects n02807616 bathrobe.n.01 clothes 38 +152 handrail handrail 7 38 7 railing otherstructure Objects n02788148 bannister.n.02 railing 30 +83 faucet faucet 7 40 7 faucet otherprop Objects faucet 3325088 n03325088 faucet.n.01 misc 40 +1204 pantry wall pantry wall 7 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +726 thermos thermos 7 40 7 flask otherprop Objects bottle bottle 2876657 n04422727 thermos.n.01 objects 39 +61 rug rug 7 40 7 rug otherprop Objects n04118021 rug.n.01 floor 2 +39 couch cushions cushion 7 18 7 pillow pillow Objects n03151500 cushion.n.03 cushion 8 +1117 tripod tripod 7 39 6 stand otherfurniture Furniture n04485082 tripod.n.01 objects 39 +540 mailbox mailbox 7 29 7 box box Objects mailbox 3710193 n03710193 mailbox.n.01 misc 40 +1205 tupperware tupperware 7 40 7 otherprop Objects objects 39 +415 shoe rack shoe rack 7 40 7 shoe rack otherprop Objects shelving 31 +31 towels towel 6 27 7 towel towel Objects n04459362 towel.n.01 towel 20 +1206 beer bottles beer bottle 6 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +153 treadmill treadmill 6 39 6 treadmill otherfurniture Furniture n04477387 treadmill.n.01 gym_equipment 33 +1207 salt salt 6 40 7 otherprop Objects objects 39 +129 chest chest 6 39 6 chest otherfurniture Furniture dresser dresser chest_of_drawers 13 +220 dispenser dispenser 6 40 7 otherprop Objects n03210683 dispenser.n.01 objects 39 +1208 mirror doors mirror door 6 8 12 door door Wall door n03221720 door.n.01 door 4 +231 remote remote 6 40 7 otherprop Objects remote_control 4074963 n04074963 remote_control.n.01 objects 39 +1209 folded ladder folded ladder 6 39 6 ladder otherfurniture Furniture stairs n03632277 ladder.n.01 misc 40 +39 cushion cushion 6 18 7 pillow pillow Objects n03151500 cushion.n.03 cushion 8 +1210 carton carton 6 40 7 otherprop Objects objects 39 +117 step step 6 38 7 otherstructure Objects n04314914 step.n.04 misc 40 +822 drying rack drying rack 6 39 6 drying rack otherfurniture Furniture shelving 31 +238 slippers slipper 6 40 7 shoe otherprop Objects n04241394 slipper.n.01 clothes 38 +143 pool table pool table 6 39 6 pool table otherfurniture Furniture table table table 4379243 n03982430 pool_table.n.01 table 5 +1211 soda stream soda stream 6 40 7 otherprop Objects objects 39 +228 toilet brush toilet brush 6 40 7 toilet brush otherprop Objects objects 39 +494 loft bed loft bed 6 4 1 bed bed Bed bed bed bed 2818832 n02818832 bed.n.01 bed 11 +226 cooking pot cooking pot 6 40 7 pot otherprop Objects objects 39 +91 heater heater 6 39 6 heater otherfurniture Furniture n03508101 heater.n.01 misc 40 +1072 messenger bag messenger bag 6 37 7 bag bag Objects objects 39 +435 stapler stapler 6 40 7 stapler otherprop Objects n04303497 stapler.n.01 objects 39 +1165 closet walls closet wall 5 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +345 scanner scanner 5 40 7 otherprop Objects appliances 37 +893 elliptical machine elliptical machine 5 40 7 machine otherprop Objects gym_equipment 33 +621 kettle kettle 5 40 7 pot otherprop Objects n03612814 kettle.n.01 objects 39 +1212 metronome metronome 5 40 7 otherprop Objects n03757604 metronome.n.01 objects 39 +297 dumbell dumbell 5 40 7 otherprop Objects objects 39 +1213 music book music book 5 23 2 book books Books n02870526 book.n.11 objects 39 +1214 rice cooker rice cooker 5 40 7 otherprop Objects objects 39 +1215 dart board dart board 5 38 7 board otherstructure Objects n03162940 dartboard.n.01 objects 39 +529 sewing machine sewing machine 5 40 7 sewing machine otherprop Objects n04179913 sewing_machine.n.01 objects 39 +1216 grab bar grab bar 5 38 7 railing otherstructure Objects railing 30 +1217 flowerpot flowerpot 5 40 7 vase otherprop Objects vase jar 3593526 n04522168 vase.n.01 objects 39 +1218 painting painting 5 11 8 picture picture Picture n03931044 picture.n.01 picture 6 +1219 railing railing 5 38 7 railing otherstructure Objects n04047401 railing.n.01 railing 30 +1220 stair stair 5 38 7 stairs otherstructure Objects stairs n04314914 step.n.04 stairs 16 +525 toolbox toolbox 5 39 6 chest otherfurniture Furniture n04452615 toolbox.n.01 objects 39 +204 nerf gun nerf gun 5 40 7 otherprop Objects objects 39 +693 binders binder 5 40 7 binder otherprop Objects objects 39 +179 desk lamp desk lamp 5 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +1221 quadcopter quadcopter 5 40 7 otherprop Objects objects 39 +1222 pitcher pitcher 5 40 7 pitcher otherprop Objects n03950228 pitcher.n.02 objects 39 +1223 hanging hanging 5 40 7 otherprop Objects misc 40 +1224 mail mail 5 40 7 otherprop Objects misc 40 +1225 closet ceiling closet ceiling 5 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 ceiling 17 +1226 hoverboard hoverboard 5 40 7 otherprop Objects objects 39 +1227 beanbag chair beanbag chair 5 39 6 bean bag otherfurniture Furniture n02816656 beanbag.n.01 chair 3 +571 water heater water heater 5 40 7 water heater otherprop Objects n04560113 water_heater.n.01 misc 40 +1228 spray bottle spray bottle 5 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +556 rope rope 5 40 7 rope otherprop Objects n04108268 rope.n.01 objects 39 +280 plastic container plastic container 5 40 7 container otherprop Objects objects 39 +1229 soap bottle soap bottle 5 40 7 soap otherprop Objects objects 39 +1230 ikea bag ikea bag 4 37 7 bag bag Objects 2773838 n02773838 bag.n.06 objects 39 +1231 sleeping bag sleeping bag 4 40 7 otherprop Objects n04235860 sleeping_bag.n.01 objects 39 +1232 duffel bag duffel bag 4 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +746 frying pan frying pan 4 40 7 frying pan otherprop Objects n03400231 frying_pan.n.01 objects 39 +1233 oven mitt oven mitt 4 40 7 otherprop Objects objects 39 +1234 pot pot 4 40 7 pot otherprop Objects n04235860 sleeping_bag.n.01 objects 39 +144 hand dryer hand dryer 4 40 7 otherprop Objects objects 39 +282 dollhouse dollhouse 4 39 6 doll house otherfurniture Furniture n03219483 dollhouse.n.01 objects 39 +167 shampoo bottle shampoo bottle 4 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +1235 hair brush hair brush 4 40 7 otherprop Objects n02908217 brush.n.02 objects 39 +1236 tennis racket tennis racket 4 40 7 otherprop Objects n04409806 tennis_racket.n.01 objects 39 +1237 display case display case 4 40 7 case otherprop Objects objects 39 +234 ping pong table ping pong table 4 39 6 ping pong table otherfurniture Furniture table table table 4379243 n04379243 table.n.02 table 5 +563 boiler boiler 4 40 7 otherprop Objects misc 40 +1238 bag of coffee beans bag of coffee beans 4 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +1239 bananas banana 4 40 7 otherprop Objects n00021265 food.n.01 objects 39 +1240 carseat carseat 4 40 7 otherprop Objects misc 40 +366 helmet helmet 4 40 7 otherprop Objects helmet 3513137 n03513137 helmet.n.02 clothes 38 +816 umbrella umbrella 4 40 7 umbrella otherprop Objects n04507155 umbrella.n.01 objects 39 +1241 coffee box coffee box 4 40 7 otherprop Objects objects 39 +719 envelope envelope 4 40 7 envelope otherprop Objects n03291819 envelope.n.01 objects 39 +284 wet floor sign wet floor sign 4 40 7 sign otherprop Objects misc 40 +1242 clothing rack clothing rack 4 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +247 controller controller 4 40 7 otherprop Objects n03096960 control.n.09 objects 39 +1243 bath walls bathroom wall 4 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +1244 podium podium 4 39 6 otherfurniture Furniture n03159640 dais.n.01 furniture 36 +1245 storage box storage box 4 29 7 box box Objects n02883344 box.n.01 objects 39 +1246 dolly dolly 4 40 7 otherprop Objects misc 40 +1247 shampoo shampoo 3 40 7 otherprop Objects n04183516 shampoo.n.01 objects 39 +592 paper tray paper tray 3 40 7 paper tray otherprop Objects objects 39 +385 cabinet door cabinet door 3 8 12 door door Wall door door 4 +1248 changing station changing station 3 40 7 otherprop Objects misc 40 +1249 poster printer poster printer 3 40 7 printer otherprop Objects printer 4004475 n04004475 printer.n.03 appliances 37 +133 screen screen 3 40 7 otherprop Objects n03151077 curtain.n.01 curtain 12 +301 soap bar soap bar 3 38 7 bar otherstructure Objects objects 39 +1250 crutches crutches 3 40 7 otherprop Objects n03141823 crutch.n.01 objects 39 +379 studio light studio light 3 38 7 light otherstructure Objects lighting 28 +130 stack of cups cup 3 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 +1251 toilet flush button toilet flush button 3 40 7 otherprop Objects objects 39 +450 trunk trunk 3 40 7 otherprop Objects misc 40 +1252 grocery bag grocery bag 3 37 7 bag bag Objects suitcase 2773838 n03461288 grocery_bag.n.01 objects 39 +316 plastic bin plastic bin 3 40 7 bin otherprop Objects objects 39 +1253 pizza box pizza box 3 29 7 box box Objects objects 39 +385 cabinet doors cabinet door 3 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 door 4 +1254 legs legs 3 31 7 person person Objects person n05217688 person.n.02 misc 40 +461 car car 3 40 7 car otherprop Objects car car 2958343 n02958343 car.n.01 misc 40 +1255 shaving cream shaving cream 3 40 7 otherprop Objects n04186051 shaving_cream.n.01 objects 39 +1256 luggage stand luggage stand 3 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +599 shredder shredder 3 40 7 otherprop Objects n04210120 shredder.n.01 objects 39 +281 statue statue 3 40 7 sculpture otherprop Objects n04306847 statue.n.01 misc 40 +1257 urinal urinal 3 33 7 toilet toilet Objects toilet toilet n04515991 urinal.n.01 toilet 18 +1258 hose hose 3 40 7 otherprop Objects n03539875 hose.n.03 misc 40 +1259 bike pump bike pump 3 40 7 otherprop Objects objects 39 +319 coatrack coatrack 3 40 7 otherprop Objects n03059103 coatrack.n.01 shelving 31 +1260 bear bear 3 40 7 otherprop Objects objects 39 +28 wall lamp lamp 3 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +1261 humidifier humidifier 3 40 7 otherprop Objects objects 39 +546 toothpaste toothpaste 3 40 7 toothpaste otherprop Objects objects 39 +1262 mouthwash bottle mouthwash bottle 3 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +1263 poster cutter poster cutter 3 40 7 otherprop Objects objects 39 +1264 golf bag golf bag 3 37 7 bag bag Objects suitcase 2773838 n03445617 golf_bag.n.01 objects 39 +1265 food container food container 3 40 7 container otherprop Objects n03094503 container.n.01 objects 39 +1266 camera camera 3 40 7 otherprop Objects objects 39 +28 table lamp lamp 3 35 7 lamp lamp Objects lamp lamp 3636649 n04380533 table_lamp.n.01 lighting 28 +1267 yoga mat yoga mat 3 20 5 floor mat floor mat Floor n03727837 mat.n.01 floor 2 +1268 card card 3 40 7 otherprop Objects objects 39 +1269 mug mug 3 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 +188 shower doors shower door 3 38 7 otherstructure Objects n04208936 shower.n.01 door 4 +689 cardboard cardboard 3 40 7 otherprop Objects objects 39 +1270 rack stand rack stand 3 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +1271 boxes of paper boxes of paper 3 29 7 box box Objects n02883344 box.n.01 objects 39 +1272 flag flag 3 40 7 otherprop Objects misc 40 +354 futon futon 3 39 6 mattress otherfurniture Furniture n03408444 futon.n.01 sofa 10 +339 magazine magazine 3 40 7 magazine otherprop Objects n06595351 magazine.n.01 objects 39 +1009 exit sign exit sign 3 40 7 exit sign otherprop Objects misc 40 +1273 rolled poster rolled poster 3 40 7 otherprop Objects objects 39 +1274 wheel wheel 3 40 7 otherprop Objects objects 39 +15 pictures picture 3 11 8 picture picture Picture n03931044 picture.n.01 picture 6 +1275 blackboard eraser blackboard eraser 3 40 7 eraser otherprop Objects n03294833 eraser.n.01 objects 39 +361 organizer organizer 3 40 7 otherprop Objects n03918737 personal_digital_assistant.n.01 objects 39 +1276 doll doll 3 40 7 toy otherprop Objects n03219135 doll.n.01 objects 39 +326 book rack book rack 3 39 6 bookrack otherfurniture Furniture objects 39 +1277 laundry bag laundry bag 3 40 7 laundry basket otherprop Objects basket 2801938 n03050864 clothes_hamper.n.01 objects 39 +1278 sponge sponge 3 40 7 otherprop Objects n01906749 sponge.n.04 objects 39 +116 seating seat 3 39 6 furniture otherfurniture Furniture n04161981 seat.n.03 furniture 36 +1184 folded chairs folded chair 2 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +1279 lotion bottle lotion bottle 2 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +212 can can 2 40 7 can otherprop Objects can 2946921 n02946921 can.n.01 objects 39 +1280 lunch box lunch box 2 40 7 otherprop Objects objects 39 +1281 food display food display 2 40 7 otherprop Objects misc 40 +794 storage shelf storage shelf 2 40 7 otherprop Objects shelving 31 +1282 sliding wood door sliding wood door 2 40 7 otherprop Objects door 4 +955 pants pants 2 40 7 otherprop Objects n04489008 trouser.n.01 clothes 38 +387 wood wood 2 40 7 otherprop Objects misc 40 +69 boards board 2 38 7 board otherstructure Objects board_panel 35 +65 bottles bottle 2 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +523 washcloth washcloth 2 40 7 otherprop Objects n04554523 washcloth.n.01 towel 20 +389 workbench workbench 2 39 6 bench otherfurniture Furniture bench table 4379243 n04600486 workbench.n.01 table 5 +29 open kitchen cabinet kitchen cabinet 2 3 6 cabinet cabinet Furniture n02933112 cabinet.n.01 cabinet 7 +1283 organizer shelf organizer shelf 2 15 6 shelves shelves Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +146 frame frame 2 38 7 otherstructure Objects misc 40 +130 cups cup 2 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 +372 exercise ball exercise ball 2 40 7 ball otherprop Objects n04285146 sports_equipment.n.01 gym_equipment 33 +289 easel easel 2 39 6 stand otherfurniture Furniture n03262809 easel.n.01 furniture 36 +440 garbage bag garbage bag 2 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +321 roomba roomba 2 40 7 otherprop Objects objects 39 +976 garage door garage door 2 38 7 garage door otherstructure Objects door door 4 +1256 luggage rack luggage stand 2 39 6 stand otherfurniture Furniture n04038440 shelving 31 +1284 bike lock bike lock 2 40 7 otherprop Objects objects 39 +1285 briefcase briefcase 2 40 7 otherprop Objects n02900705 briefcase.n.01 objects 39 +357 hand towel hand towel 2 27 7 towel towel Objects n03490006 hand_towel.n.01 towel 20 +1286 bath products bath product 2 40 7 otherprop Objects objects 39 +1287 star star 2 40 7 otherprop Objects n09444783 star.n.03 misc 40 +365 map map 2 40 7 map otherprop Objects n03720163 map.n.01 misc 40 +1288 coffee bean bag coffee bean bag 2 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +81 headboard headboard 2 39 6 headboard otherfurniture Furniture n03502200 headboard.n.01 bed 11 +1289 ipad ipad 2 40 7 otherprop Objects objects 39 +1290 display rack display rack 2 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +948 traffic cone traffic cone 2 40 7 cone otherprop Objects cone objects 39 +174 toiletry toiletry 2 40 7 otherprop Objects n04447443 toiletry.n.01 objects 39 +1028 canopy canopy 2 40 7 otherprop Objects misc 40 +1291 massage chair massage chair 2 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +1292 paper organizer paper organizer 2 40 7 otherprop Objects objects 39 +1005 barricade barricade 2 40 7 otherprop Objects misc 40 +235 platform platform 2 38 7 otherstructure Objects misc 40 +1293 cap cap 2 40 7 hat otherprop Objects n03497657 hat.n.01 clothes 38 +1294 dumbbell plates dumbbell plates 2 40 7 otherprop Objects objects 39 +1295 elevator elevator 2 38 7 otherstructure Objects misc 40 +1296 cooking pan cooking pan 2 40 7 pan otherprop Objects n03880531 pan.n.01 objects 39 +1297 trash bag trash bag 2 37 7 bag bag Objects objects 39 +1298 santa santa 2 40 7 otherprop Objects misc 40 +1299 jewelry box jewelry box 2 29 7 box box Objects n02883344 box.n.01 objects 39 +1300 boat boat 2 40 7 otherprop Objects misc 40 +1301 sock sock 2 21 7 clothes clothes Objects n04254777 sock.n.01 clothes 38 +1051 kinect kinect 2 40 7 kinect otherprop Objects objects 39 +566 crib crib 2 39 6 crib otherfurniture Furniture furniture 36 +1302 plastic storage bin plastic storage bin 2 40 7 container otherprop Objects n03094503 container.n.01 objects 39 +1062 cooler cooler 2 24 6 refridgerator refridgerator Furniture n03102654 cooler.n.01 appliances 37 +1303 kitchen apron kitchen apron 2 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +1304 dishwashing soap bottle dishwashing soap bottle 2 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +1305 xbox controller xbox controller 2 40 7 otherprop Objects objects 39 +1306 banana holder banana holder 2 40 7 otherprop Objects objects 39 +298 ping pong paddle ping pong paddle 2 40 7 otherprop Objects table 5 +1307 airplane airplane 2 40 7 otherprop Objects misc 40 +1308 conditioner bottle conditioner bottle 2 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +1309 tea kettle tea kettle 2 40 7 tea kettle otherprop Objects n04397768 teakettle.n.01 objects 39 +43 bedframe bedframe 2 39 6 otherfurniture Furniture n02822579 bedstead.n.01 bed 11 +1310 wood beam wood beam 2 38 7 otherstructure Objects beam 29 +593 toilet paper package toilet paper package 2 40 7 otherprop Objects objects 39 +1311 wall mounted coat rack wall mounted coat rack 2 40 7 otherprop Objects n03059103 coatrack.n.01 shelving 31 +1312 film light film light 2 40 7 otherprop Objects lighting 28 +749 ceiling lamp ceiling lamp 1 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +623 chain chain 1 40 7 otherprop Objects chair 3 +1313 sofa sofa 1 6 9 sofa sofa Sofa sofa sofa sofa 4256520 n04256520 sofa.n.01 sofa 10 +99 closet wardrobe wardrobe 1 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 furniture 36 +265 sweater sweater 1 40 7 otherprop Objects n04370048 sweater.n.01 clothes 38 +1314 kitchen mixer kitchen mixer 1 40 7 otherprop Objects appliances 37 +99 wardrobe wardrobe 1 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 furniture 36 +1315 water softener water softener 1 40 7 otherprop Objects misc 40 +448 banister banister 1 38 7 banister otherstructure Objects n02788148 bannister.n.02 railing 30 +257 trolley trolley 1 40 7 trolley otherprop Objects n04335435 streetcar.n.01 misc 40 +1316 pantry shelf pantry shelf 1 15 6 shelves shelves Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +786 sofa bed sofa bed 1 4 1 bed bed Bed bed bed bed 2818832 n02818832 bed.n.01 bed 11 +801 loofa loofa 1 40 7 otherprop Objects objects 39 +972 shower faucet handle shower faucet handle 1 40 7 handle otherprop Objects shower 23 +1317 toy piano toy piano 1 40 7 toy otherprop Objects n03964744 plaything.n.01 objects 39 +1318 fish fish 1 40 7 otherprop Objects n02512053 fish.n.01 objects 39 +75 file cabinets file cabinet 1 3 6 cabinet cabinet Furniture cabinet 2933112 n03337140 file.n.03 cabinet 7 +657 cat litter box cat litter box 1 29 7 box box Objects objects 39 +561 electric panel electric panel 1 40 7 otherprop Objects misc 40 +93 suitcases suitcase 1 40 7 luggage otherprop Objects n02774630 baggage.n.01 objects 39 +513 curtain rod curtain rod 1 38 7 curtain rod otherstructure Objects curtain 12 +411 bunk bed bunk bed 1 39 6 bunk bed otherfurniture Furniture bed bed bed 2818832 n02920259 bunk_bed.n.01 bed 11 +1122 chandelier chandelier 1 38 7 chandelier otherstructure Objects n03005285 chandelier.n.01 lighting 28 +922 tape tape 1 40 7 tape otherprop Objects objects 39 +88 plates plate 1 40 7 otherprop Objects n03959485 plate.n.04 objects 39 +518 alarm alarm 1 40 7 alarm otherprop Objects clock 3046257 n02694662 alarm_clock.n.01 objects 39 +814 fire hose fire hose 1 40 7 otherprop Objects n03346004 fire_hose.n.01 misc 40 +1319 toy dinosaur toy dinosaur 1 40 7 toy otherprop Objects n03964744 plaything.n.01 objects 39 +1320 cone cone 1 40 7 otherprop Objects objects 39 +649 glass doors glass door 1 8 12 door door Wall door n03221720 door.n.01 door 4 +607 hatrack hatrack 1 40 7 otherprop Objects n03059103 coatrack.n.01 shelving 31 +819 subwoofer subwoofer 1 40 7 speaker otherprop Objects speaker 3691459 n04349401 subwoofer.n.01 objects 39 +1321 fire sprinkler fire sprinkler 1 40 7 otherprop Objects misc 40 +1322 trash cabinet trash cabinet 1 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +1204 pantry walls pantry wall 1 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +227 photo photo 1 40 7 photo otherprop Objects n03925226 photograph.n.01 picture 6 +817 barrier barrier 1 40 7 otherprop Objects n02796623 barrier.n.01 misc 40 +130 stacks of cups cup 1 40 7 otherprop Objects n03147509 cup.n.01 objects 39 +712 beachball beachball 1 40 7 ball otherprop Objects n02814224 beach_ball.n.01 objects 39 +1323 folded boxes folded boxes 1 40 7 otherprop Objects objects 39 +1324 contact lens solution bottle contact lens solution bottle 1 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +673 covered box covered box 1 29 7 box box Objects objects 39 +459 folder folder 1 40 7 folder otherprop Objects n03376279 folder.n.02 objects 39 +643 mail trays mail tray 1 40 7 mail tray otherprop Objects objects 39 +238 slipper slipper 1 40 7 otherprop Objects n04241394 slipper.n.01 clothes 38 +765 magazine rack magazine rack 1 39 6 stand otherfurniture Furniture n03704549 magazine_rack.n.01 shelving 31 +1008 sticker sticker 1 40 7 sticker otherprop Objects n07272545 gummed_label.n.01 objects 39 +225 lotion lotion 1 40 7 otherprop Objects n03690938 lotion.n.01 objects 39 +1083 buddha buddha 1 40 7 otherprop Objects objects 39 +813 file organizer file organizer 1 40 7 otherprop Objects objects 39 +138 paper towel rolls paper towel roll 1 40 7 paper towel otherprop Objects n03887697 paper_towel.n.01 towel 20 +1145 night lamp night lamp 1 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +796 fuse box fuse box 1 40 7 otherprop Objects misc 40 +1325 knife block knife block 1 40 7 otherprop Objects objects 39 +363 furnace furnace 1 39 6 furnace otherfurniture Furniture n03404449 furnace.n.01 +1174 cd cases cd case 1 40 7 otherprop Objects objects 39 +38 stools stool 1 40 7 stool otherprop Objects stool n04326896 stool.n.01 stool 19 +1326 hand sanitzer dispenser hand sanitzer dispenser 1 40 7 otherprop Objects n04254120 soap_dispenser.n.01 objects 39 +997 teapot teapot 1 40 7 tea pot otherprop Objects n04398044 teapot.n.01 objects 39 +1327 pen holder pen holder 1 40 7 otherprop Objects objects 39 +1328 tray rack tray rack 1 40 7 otherprop Objects objects 39 +1329 wig wig 1 40 7 otherprop Objects n04584207 wig.n.01 objects 39 +182 switch switch 1 40 7 otherprop Objects n04372370 switch.n.01 misc 40 +280 plastic containers plastic container 1 40 7 container otherprop Objects n03094503 container.n.01 objects 39 +1330 night light night light 1 40 7 otherprop Objects lighting 28 +1331 notepad notepad 1 40 7 otherprop Objects objects 39 +1332 mail bin mail bin 1 40 7 otherprop Objects misc 40 +1333 elevator button elevator button 1 40 7 otherprop Objects misc 40 +939 gaming wheel gaming wheel 1 40 7 otherprop Objects objects 39 +1334 drum set drum set 1 40 7 otherprop Objects objects 39 +480 cosmetic bag cosmetic bag 1 37 7 bag bag Objects objects 39 +907 coffee mug coffee mug 1 40 7 vessel otherprop Objects cup or mug 3797390 n03063599 coffee_mug.n.01 objects 39 +1335 closet shelf closet shelf 1 15 6 shelves shelves Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +1336 baby mobile baby mobile 1 40 7 otherprop Objects objects 39 +829 diaper bin diaper bin 1 40 7 bin otherprop Objects objects 39 +947 door wall door wall 1 1 12 wall wall Wall wall 1 +1116 stepstool stepstool 1 40 7 step stool otherprop Objects objects 39 +599 paper shredder shredder 1 40 7 otherprop Objects n04210120 shredder.n.01 objects 39 +733 dress rack dress rack 1 40 7 otherprop Objects n03238762 dress_rack.n.01 misc 40 +123 cover cover 1 40 7 blanket otherprop Objects objects 39 +506 shopping bag shopping bag 1 37 7 bag bag Objects n04204081 shopping_bag.n.01 objects 39 +569 sliding door sliding door 1 8 12 door door Wall door n04239074 sliding_door.n.01 door 4 +1337 exercise bike exercise bike 1 40 7 machine otherprop Objects n04210120 shredder.n.01 gym_equipment 33 +1338 recliner chair recliner chair 1 5 4 chair chair Chair chair chair chair 3001627 n03238762 dress_rack.n.01 chair 3 +1314 kitchenaid mixer kitchen mixer 1 40 7 otherprop Objects appliances 37 +1339 soda can soda can 1 40 7 can otherprop Objects can 2946921 n02946921 can.n.01 objects 39 +1340 stovetop stovetop 1 38 7 stove otherstructure Objects stove 4330267 n04330267 stove.n.02 appliances 37 +851 stepladder stepladder 1 39 6 ladder otherfurniture Furniture stairs n04315599 step_ladder.n.01 stairs 16 +142 tap tap 1 40 7 faucet otherprop Objects faucet 3325088 n04559451 water_faucet.n.01 objects 39 +436 cable cable 1 40 7 cables otherprop Objects objects 39 +1341 baby changing station baby changing station 1 39 6 otherfurniture Furniture furniture 36 +1342 costume costume 1 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +885 rocking chair rocking chair 1 5 4 chair chair Chair chair chair chair 3001627 n04099969 rocking_chair.n.01 chair 3 +693 binder binder 1 40 7 binder otherprop Objects objects 39 +815 media center media center 1 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +401 towel rack towel rack 1 40 7 otherprop Objects n04459773 towel_rack.n.01 misc 40 +1343 medal medal 1 40 7 otherprop Objects objects 39 +1184 stack of folded chairs folded chair 1 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +1344 telescope telescope 1 40 7 otherprop Objects n04403638 telescope.n.01 objects 39 +1345 closet doorframe closet doorframe 1 8 12 door door Wall door door 4 +160 glass glass 1 38 7 glass otherstructure Objects n03438257 glass.n.02 misc 40 +1126 baseball cap baseball cap 1 40 7 otherprop Objects cap 2954340 n02799323 baseball_cap.n.01 clothes 38 +1346 battery disposal jar battery disposal jar 1 40 7 jar otherprop Objects jar 3593526 n03593526 jar.n.01 objects 39 +332 mop mop 1 40 7 otherprop Objects n04367480 swab.n.02 objects 39 +397 tank tank 1 40 7 otherprop Objects objects 39 +643 mail tray mail tray 1 40 7 mail tray otherprop Objects objects 39 +551 centerpiece centerpiece 1 40 7 centerpiece otherprop Objects n02994419 centerpiece.n.02 objects 39 +1163 stick stick 1 40 7 stick otherprop Objects objects 39 +1347 closet floor closet floor 1 2 5 floor floor Floor n03365592 floor.n.01 floor 2 +1348 dryer sheets dryer sheets 1 40 7 otherprop Objects objects 39 +803 bycicle bycicle 1 40 7 otherprop Objects misc 40 +484 flower stand flower stand 1 39 6 stand otherfurniture Furniture furniture 36 +1349 air mattress air mattress 1 4 1 bed bed Bed bed bed bed 2818832 n02690809 air_mattress.n.01 bed 11 +1350 clip clip 1 40 7 otherprop Objects objects 39 +222 side table side table 1 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +1253 pizza boxes pizza box 1 29 7 box box Objects n02883344 box.n.01 objects 39 +1351 display display 1 39 7 otherfurniture Furniture n03211117 display.n.06 misc 40 +1352 postcard postcard 1 40 7 otherprop Objects objects 39 +828 display sign display sign 1 40 7 sign otherprop Objects misc 40 +1353 paper towel paper towel 1 40 7 paper towel otherprop Objects n03887697 paper_towel.n.01 towel 20 +612 boots boot 1 40 7 shoe otherprop Objects n04199027 shoe.n.01 clothes 38 +1354 tennis racket bag tennis racket bag 1 40 7 otherprop Objects objects 39 +1355 air hockey table air hockey table 1 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +1301 socks sock 1 21 7 clothes clothes Objects n04254777 sock.n.01 clothes 38 +1356 food bag food bag 1 37 7 bag bag Objects objects 39 +1199 clothes hangers clothes hanger 1 40 7 otherprop Objects n03057920 coat_hanger.n.01 misc 40 +1357 starbucks cup starbucks cup 1 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2-labels.combined.tsv b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2-labels.combined.tsv new file mode 100644 index 0000000000000000000000000000000000000000..cff61b132f3ebf4edd513445b76fd39db54462d2 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2-labels.combined.tsv @@ -0,0 +1,608 @@ +id raw_category category count nyu40id eigen13id nyuClass nyu40class eigen13class ModelNet40 ModelNet10 ShapeNetCore55 synsetoffset wnsynsetid wnsynsetkey mpcat40 mpcat40index +1 wall wall 8277 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +2 chair chair 4646 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +22 books book 1678 23 2 book books Books n02870526 book.n.11 objects 39 +3 floor floor 1553 2 5 floor floor Floor n03365592 floor.n.01 floor 2 +5 door door 1483 8 12 door door Wall door n03221720 door.n.01 door 4 +1163 object object 1313 40 7 otherprop Objects objects 39 +16 window window 1209 9 13 window window Window n04587648 window.n.01 window 9 +4 table table 1170 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +56 trash can trash can 1090 39 6 garbage bin otherfurniture Furniture trash_bin 2747177 n02747177 ashcan.n.01 objects 39 +13 pillow pillow 937 18 7 pillow pillow Objects pillow 3938244 n03938244 pillow.n.01 cushion 8 +15 picture picture 862 11 8 picture picture Picture n03931044 picture.n.01 picture 6 +41 ceiling ceiling 806 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 ceiling 17 +26 box box 775 29 7 box box Objects n02883344 box.n.01 objects 39 +161 doorframe doorframe 768 8 12 door door Wall door doorframe.n.01 door 4 +19 monitor monitor 765 40 7 monitor otherprop Objects monitor monitor tv or monitor 3211117 n03782190 monitor.n.04 objects 39 +7 cabinet cabinet 731 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +9 desk desk 680 14 10 desk desk Table desk desk table 4379243 n03179701 desk.n.01 table 5 +8 shelf shelf 641 15 6 shelves shelves Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +10 office chair office chair 595 5 4 chair chair Chair chair chair chair 3001627 n04373704 swivel_chair.n.01 chair 3 +31 towel towel 570 27 7 towel towel Objects n04459362 towel.n.01 towel 20 +6 couch couch 502 6 9 sofa sofa Sofa sofa sofa sofa 4256520 n04256520 sofa.n.01 sofa 10 +14 sink sink 488 34 7 sink sink Objects sink n04223580 sink.n.01 sink 15 +48 backpack backpack 479 40 7 backpack otherprop Objects n02769748 backpack.n.01 objects 39 +28 lamp lamp 419 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +11 bed bed 370 4 1 bed bed Bed bed bed bed 2818832 n02818832 bed.n.01 bed 11 +18 bookshelf bookshelf 360 10 6 bookshelf bookshelf Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +71 mirror mirror 349 19 7 mirror mirror Objects n03773035 mirror.n.01 mirror 21 +21 curtain curtain 347 16 13 curtain curtain Window curtain n03151077 curtain.n.01 curtain 12 +40 plant plant 331 40 7 plant otherprop Objects plant n00017222 plant.n.02 plant 14 +52 whiteboard whiteboard 327 30 7 whiteboard whiteboard Objects n03211616 display_panel.n.01 board_panel 35 +96 radiator radiator 322 39 6 radiator otherfurniture Furniture n04041069 radiator.n.02 misc 40 +22 book book 318 23 2 book books Books n02870526 book.n.11 objects 39 +29 kitchen cabinet kitchen cabinet 310 3 6 cabinet cabinet Furniture n02933112 cabinet.n.01 cabinet 7 +49 toilet paper toilet paper 291 40 7 toilet paper otherprop Objects n15075141 toilet_tissue.n.01 objects 39 +29 kitchen cabinets kitchen cabinet 289 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +23 armchair armchair 281 5 4 chair chair Chair chair chair chair 3001627 n02738535 armchair.n.01 chair 3 +63 shoes shoe 272 40 7 shoe otherprop Objects n04199027 shoe.n.01 clothes 38 +24 coffee table coffee table 258 7 10 coffee table table Table table table table 4379243 n03063968 coffee_table.n.01 table 5 +17 toilet toilet 256 33 7 toilet toilet Objects toilet toilet n04446276 toilet.n.01 toilet 18 +47 bag bag 252 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +32 clothes clothes 248 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +46 keyboard keyboard 246 40 7 keyboard otherprop Objects keyboard computer keyboard 3085013 n03085013 computer_keyboard.n.01 objects 39 +65 bottle bottle 226 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +97 recycling bin recycling bin 225 39 6 garbage bin otherfurniture Furniture trash_bin 2747177 n02747177 ashcan.n.01 objects 39 +34 nightstand nightstand 224 32 6 night stand night stand Furniture night_stand night_stand n03015254 chest_of_drawers.n.01 chest_of_drawers 13 +38 stool stool 221 40 7 stool otherprop Objects stool n04326896 stool.n.01 stool 19 +33 tv tv 219 25 11 television television TV tv or monitor 3211117 n03211117 display.n.06 tv_monitor 22 +75 file cabinet file cabinet 217 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +36 dresser dresser 213 17 6 dresser dresser Furniture dresser dresser n03015254 chest_of_drawers.n.01 chest_of_drawers 13 +64 computer tower computer tower 203 40 7 computer otherprop Objects n03082979 computer.n.01 objects 39 +32 clothing clothes 165 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +101 telephone telephone 164 40 7 telephone otherprop Objects telephone 4401088 n04401088 telephone.n.01 objects 39 +130 cup cup 157 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 +27 refrigerator refrigerator 154 24 6 refridgerator refridgerator Furniture n04070727 refrigerator.n.01 appliances 37 +44 end table end table 147 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +131 jacket jacket 146 40 7 jacket otherprop Objects n03589791 jacket.n.01 clothes 38 +55 shower curtain shower curtain 144 28 7 shower curtain shower curtain Objects curtain n04209239 shower_curtain.n.01 curtain 12 +42 bathtub bathtub 144 36 7 bathtub bathtub Objects bathtub bathtub tub 2808440 n02808440 bathtub.n.01 bathtub 25 +59 microwave microwave 141 40 7 microwave otherprop Objects microwave 3761084 n03761084 microwave.n.02 appliances 37 +159 kitchen counter kitchen counter 140 12 6 counter counter Furniture table table table 4379243 n03116530 counter.n.01 counter 26 +74 sofa chair sofa chair 129 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +82 paper towel dispenser paper towel dispenser 129 40 7 paper towel dispenser otherprop Objects objects 39 +1164 bathroom vanity bathroom vanity 126 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 table 5 +93 suitcase suitcase 118 40 7 luggage otherprop Objects n02773838 bag.n.06 objects 39 +77 laptop laptop 111 40 7 laptop otherprop Objects laptop laptop 3642806 n03642806 laptop.n.01 objects 39 +67 ottoman ottoman 111 39 6 ottoman otherfurniture Furniture stool n03380724 footstool.n.01 stool 19 +128 shower walls shower wall 109 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +50 printer printer 106 40 7 printer otherprop Objects printer 4004475 n04004475 printer.n.03 appliances 37 +35 counter counter 104 12 6 counter counter Furniture table table table 4379243 n03116530 counter.n.01 counter 26 +69 board board 100 38 7 board otherstructure Objects board_panel 35 +100 soap dispenser soap dispenser 99 40 7 otherprop Objects n04254120 soap_dispenser.n.01 objects 39 +62 stove stove 95 38 7 stove otherstructure Objects stove 4330267 n04330267 stove.n.02 appliances 37 +105 light light 93 38 7 light otherstructure Objects n03665366 light.n.02 lighting 28 +1165 closet wall closet wall 90 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +165 mini fridge mini fridge 87 24 6 refridgerator refridgerator Furniture n03273913 electric_refrigerator.n.01 appliances 37 +7 cabinets cabinet 79 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +5 doors door 76 8 12 door door Wall door n03221720 door.n.01 door 4 +76 fan fan 75 40 7 fan otherprop Objects n03320046 fan.n.01 misc 40 +230 tissue box tissue box 73 40 7 tissue box otherprop Objects n02883344 box.n.01 objects 39 +54 blanket blanket 72 40 7 blanket otherprop Objects n02849154 blanket.n.01 objects 39 +125 bathroom stall bathroom stall 71 38 7 otherstructure Objects n02873839 booth.n.02 misc 40 +72 copier copier 70 40 7 otherprop Objects n03257586 duplicator.n.01 appliances 37 +68 bench bench 66 39 6 bench otherfurniture Furniture bench bench 2828884 n02828884 bench.n.01 seating 34 +145 bar bar 66 38 7 bar otherstructure Objects n02788689 bar.n.03 misc 40 +157 soap dish soap dish 65 40 7 soap dish otherprop Objects n04254009 soap_dish.n.01 objects 39 +1166 laundry hamper laundry hamper 65 40 7 laundry basket otherprop Objects objects 39 +132 storage bin storage bin 63 40 7 storage bin otherprop Objects objects 39 +1167 bathroom stall door bathroom stall door 62 8 12 door door Wall door n03221720 door.n.01 door 4 +232 light switch light switch 61 38 7 light switch otherstructure Objects n04372370 switch.n.01 misc 40 +134 coffee maker coffee maker 61 40 7 otherprop Objects n03063338 coffee_maker.n.01 appliances 37 +51 tv stand tv stand 61 39 6 tv stand otherfurniture Furniture tv_stand n03290653 entertainment_center.n.01 furniture 36 +250 decoration decoration 60 40 7 otherprop Objects n03169390 decoration.n.01 misc 40 +1168 ceiling light ceiling light 59 38 7 light otherstructure Objects n03665366 light.n.02 lighting 28 +342 range hood range hood 59 38 7 range hood otherstructure Objects range_hood n04053677 range_hood.n.01 misc 40 +89 blackboard blackboard 58 38 7 blackboard otherstructure Objects n02846511 blackboard.n.01 board_panel 35 +103 clock clock 58 40 7 clock otherprop Objects clock 3046257 n03046257 clock.n.01 objects 39 +99 wardrobe closet wardrobe 54 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 furniture 36 +95 rail rail 53 38 7 railing otherstructure Objects n04047401 railing.n.01 railing 30 +154 bulletin board bulletin board 53 38 7 board otherstructure Objects n03211616 display_panel.n.01 board_panel 35 +140 mat mat 52 20 5 floor mat floor mat Floor n03727837 mat.n.01 floor 2 +1169 trash bin trash bin 52 39 6 garbage bin otherfurniture Furniture trash_bin 2747177 n02747177 ashcan.n.01 objects 39 +193 ledge ledge 51 38 7 otherstructure Objects n09337253 ledge.n.01 misc 40 +116 seat seat 49 39 6 furniture otherfurniture Furniture n04161981 seat.n.03 furniture 36 +202 mouse mouse 49 40 7 mouse otherprop Objects n03793489 mouse.n.04 objects 39 +73 basket basket 48 40 7 basket otherprop Objects basket 2801938 n02801938 basket.n.01 objects 39 +78 shower shower 48 38 7 otherstructure Objects n04208936 shower.n.01 shower 23 +1170 dumbbell dumbbell 48 40 7 otherprop Objects n03255030 dumbbell.n.01 objects 39 +79 paper paper 46 26 7 paper paper Objects n14974264 paper.n.01 objects 39 +80 person person 46 31 7 person person Objects person n05217688 person.n.02 misc 40 +141 windowsill windowsill 45 38 7 otherstructure Objects n04590263 windowsill.n.01 window 9 +57 closet closet 45 39 6 wardrobe otherfurniture Furniture wardrobe misc 40 +102 bucket bucket 45 40 7 bucket otherprop Objects n02909870 bucket.n.01 misc 40 +261 sign sign 44 40 7 sign otherprop Objects n04217882 signboard.n.01 objects 39 +118 speaker speaker 43 40 7 speaker otherprop Objects speaker 3691459 n03691459 loudspeaker.n.01 objects 39 +136 dishwasher dishwasher 43 38 7 dishwasher otherstructure Objects dishwasher 3207941 n03207941 dishwasher.n.01 appliances 37 +98 container container 43 40 7 container otherprop Objects n03094503 container.n.01 objects 39 +1171 stair rail stair rail 42 38 7 banister otherstructure Objects n02788148 bannister.n.02 railing 30 +170 shower curtain rod shower curtain rod 42 40 7 otherprop Objects curtain 12 +1172 tube tube 41 40 7 otherprop Objects misc 40 +1173 bathroom cabinet bathroom cabinet 39 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +79 papers paper 39 26 7 paper paper Objects n14974264 paper.n.01 objects 39 +221 storage container storage container 39 40 7 container otherprop Objects objects 39 +570 paper bag paper bag 39 37 7 bag bag Objects n04122825 sack.n.01 objects 39 +138 paper towel roll paper towel roll 39 40 7 paper towel otherprop Objects n03887697 paper_towel.n.01 towel 20 +168 ball ball 39 40 7 ball otherprop Objects objects 39 +276 closet doors closet door 38 8 12 door door Wall door n03221720 door.n.01 door 4 +106 laundry basket laundry basket 37 40 7 laundry basket otherprop Objects basket 2801938 n03050864 clothes_hamper.n.01 objects 39 +214 cart cart 37 40 7 cart otherprop Objects n03484083 handcart.n.01 shelving 31 +276 closet door closet door 35 8 12 door door Wall door n03221720 door.n.01 door 4 +323 dish rack dish rack 35 40 7 dish rack otherprop Objects n03207630 dish_rack.n.01 objects 39 +58 stairs stairs 35 38 7 stairs otherstructure Objects n04298308 stairway.n.01 stairs 16 +86 blinds blinds 35 13 13 blinds blinds Window n02851099 blind.n.03 blinds 32 +2 stack of chairs chair 35 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +399 purse purse 34 40 7 purse otherprop Objects n02774152 bag.n.04 objects 39 +121 bicycle bicycle 33 40 7 bicycle otherprop Objects bicycle 2834778 n02834778 bicycle.n.01 objects 39 +185 tray tray 32 40 7 tray otherprop Objects n04476259 tray.n.01 objects 39 +300 plunger plunger 30 40 7 otherprop Objects n03970156 plunger.n.03 objects 39 +180 paper cutter paper cutter 30 40 7 paper cutter otherprop Objects n03886940 paper_cutter.n.01 objects 39 +163 toilet paper dispenser toilet paper dispenser 29 40 7 otherprop Objects objects 39 +26 boxes box 29 29 7 box box Objects n02883344 box.n.01 objects 39 +66 bin bin 28 40 7 bin otherprop Objects n02839910 bin.n.01 objects 39 +208 toilet seat cover dispenser toilet seat cover dispenser 28 40 7 otherprop Objects objects 39 +112 guitar guitar 28 40 7 guitar otherprop Objects guitar guitar 3467517 n03467517 guitar.n.01 objects 39 +540 mailboxes mailbox 28 29 7 box box Objects mailbox 3710193 n03710193 mailbox.n.01 misc 40 +395 handicap bar handicap bar 27 38 7 bar otherstructure Objects misc 40 +166 fire extinguisher fire extinguisher 27 40 7 fire extinguisher otherprop Objects n03345837 fire_extinguisher.n.01 misc 40 +122 ladder ladder 27 39 6 ladder otherfurniture Furniture stairs n03632277 ladder.n.01 stairs 16 +120 column column 26 38 7 column otherstructure Objects n03074380 column.n.06 column 24 +107 pipe pipe 25 40 7 pipe otherprop Objects n03944672 pipe.n.02 misc 40 +283 vacuum cleaner vacuum cleaner 25 40 7 otherprop Objects n04517823 vacuum.n.04 objects 39 +88 plate plate 24 40 7 plate otherprop Objects n03959485 plate.n.04 objects 39 +90 piano piano 24 39 6 piano otherfurniture Furniture piano piano 3928116 n03928116 piano.n.01 furniture 36 +177 water cooler water cooler 24 39 6 water cooler otherfurniture Furniture n04559166 water_cooler.n.01 misc 40 +1174 cd case cd case 24 40 7 otherprop Objects objects 39 +562 bowl bowl 24 40 7 bowl otherprop Objects bowl bowl 2880940 n02880940 bowl.n.03 objects 39 +1175 closet rod closet rod 24 40 7 otherprop Objects n04100174 rod.n.01 misc 40 +1156 bathroom counter bathroom counter 24 12 6 counter counter Furniture table table table 4379243 n03116530 counter.n.01 counter 26 +84 oven oven 23 38 7 oven otherstructure Objects n03862676 oven.n.01 appliances 37 +104 stand stand 23 39 6 stand otherfurniture Furniture table table table 4379243 n04301000 stand.n.04 table 5 +229 scale scale 23 40 7 scale otherprop Objects n04141975 scale.n.07 objects 39 +70 washing machine washing machine 23 39 6 washing machine otherfurniture Furniture washing_machine 4554684 n04554684 washer.n.03 appliances 37 +325 broom broom 22 40 7 broom otherprop Objects n02906734 broom.n.01 objects 39 +169 hat hat 22 40 7 hat otherprop Objects n03497657 hat.n.01 clothes 38 +128 shower wall shower wall 22 1 12 wall wall Wall n04208936 shower.n.01 wall 1 +331 guitar case guitar case 21 40 7 guitar case otherprop Objects objects 39 +87 rack rack 21 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +488 water pitcher water pitcher 21 40 7 pitcher otherprop Objects n03950228 pitcher.n.02 objects 39 +776 laundry detergent laundry detergent 21 40 7 otherprop Objects objects 39 +370 hair dryer hair dryer 21 40 7 hair dryer otherprop Objects n03483316 hand_blower.n.01 objects 39 +191 pillar pillar 21 38 7 column otherstructure Objects n03073977 column.n.07 column 24 +748 divider divider 20 40 7 otherprop Objects wall 1 +242 power outlet power outlet 19 40 7 otherprop Objects misc 40 +45 dining table dining table 19 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +417 shower floor shower floor 19 2 5 floor floor Floor n04208936 shower.n.01 floor 2 +70 washing machines washing machine 19 39 6 washing machine otherfurniture Furniture washing_machine 4554684 n04554684 washer.n.03 appliances 37 +188 shower door shower door 19 8 12 door door Wall door n04208936 shower.n.01 door 4 +1176 coffee kettle coffee kettle 18 40 7 pot otherprop Objects n03612814 kettle.n.01 objects 39 +1177 wardrobe cabinet wardrobe 18 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 furniture 36 +1178 structure structure 18 38 7 otherstructure Objects misc 40 +18 bookshelves bookshelf 17 10 6 bookshelf bookshelf Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +110 clothes dryer clothes dryer 17 39 6 otherfurniture Furniture n03251766 dryer.n.01 appliances 37 +148 toaster toaster 17 40 7 toaster otherprop Objects n04442312 toaster.n.02 appliances 37 +63 shoe shoe 17 40 7 shoe otherprop Objects n04199027 shoe.n.01 clothes 38 +155 ironing board ironing board 16 39 6 ironing board otherfurniture Furniture n03586090 ironing_board.n.01 objects 39 +572 alarm clock alarm clock 16 40 7 alarm clock otherprop Objects clock 3046257 n02694662 alarm_clock.n.01 objects 39 +1179 shower head shower head 15 38 7 otherstructure Objects shower 23 +28 lamp base lamp 15 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +392 water bottle water bottle 15 40 7 bottle otherprop Objects bottle bottle 2876657 n04557648 water_bottle.n.01 objects 39 +1180 keyboard piano keyboard piano 15 39 6 piano otherfurniture Furniture piano piano 3928116 n03928116 piano.n.01 furniture 36 +609 projector screen projector screen 15 38 7 projector screen otherstructure Objects misc 40 +1181 case of water bottles case of water bottles 15 40 7 otherprop Objects objects 39 +195 toaster oven toaster oven 14 40 7 toaster oven otherprop Objects n04442441 toaster_oven.n.01 appliances 37 +581 music stand music stand 14 39 6 music stand otherfurniture Furniture n03801760 music_stand.n.01 furniture 36 +58 staircase stairs 14 38 7 stairs otherstructure Objects n04298308 stairway.n.01 stairs 16 +1182 coat rack coat rack 14 40 7 otherprop Objects n03059103 coatrack.n.01 shelving 3 +1183 storage organizer storage organizer 14 40 7 otherprop Objects shelving 3 +139 machine machine 14 40 7 machine otherprop Objects n03699975 machine.n.01 appliances 37 +1184 folded chair folded chair 14 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +1185 fire alarm fire alarm 14 40 7 otherprop Objects n03343737 fire_alarm.n.02 misc 40 +156 fireplace fireplace 13 38 7 fireplace otherstructure Objects n03346455 fireplace.n.01 fireplace 27 +408 vent vent 13 40 7 otherprop Objects n04526241 vent.n.01 misc 40 +213 furniture furniture 13 39 6 furniture otherfurniture Furniture n03405725 furniture.n.01 furniture 36 +1186 power strip power strip 13 40 7 otherprop Objects objects 39 +1187 calendar calendar 13 40 7 otherprop Objects objects 39 +1188 poster poster 13 11 8 picture picture Picture n03931044 picture.n.01 picture 6 +115 toilet paper holder toilet paper holder 13 40 7 toilet paper holder otherprop Objects objects 39 +1189 potted plant potted plant 12 40 7 plant otherprop Objects plant n00017222 plant.n.02 plant 14 +304 stuffed animal stuffed animal 12 40 7 stuffed animal otherprop Objects n04399382 teddy.n.01 objects 39 +1190 luggage luggage 12 40 7 luggage otherprop Objects n02774630 baggage.n.01 objects 39 +21 curtains curtain 12 16 13 curtain curtain Window curtain n03151077 curtain.n.01 curtain 12 +312 headphones headphones 12 40 7 otherprop Objects n03261776 earphone.n.01 objects 39 +233 crate crate 12 39 6 crate otherfurniture Furniture n03127925 crate.n.01 objects 39 +286 candle candle 12 40 7 candle otherprop Objects lamp n02948072 candle.n.01 objects 39 +264 projector projector 12 40 7 projector otherprop Objects n04009552 projector.n.02 objects 39 +110 clothes dryers clothes dryer 12 39 6 otherfurniture Furniture n03251766 dryer.n.01 appliances 37 +1191 mattress mattress 12 4 1 bed bed Bed bed bed bed 2818832 n02818832 bed.n.01 bed 11 +356 dustpan dustpan 12 40 7 otherprop Objects n03259009 dustpan.n.02 objects 39 +25 drawer drawer 11 39 6 drawer otherfurniture Furniture n03233905 drawer.n.01 furniture 36 +750 rod rod 11 40 7 otherprop Objects pistol 3948459 n03427202 gat.n.01 misc 40 +269 globe globe 11 40 7 globe otherprop Objects objects 39 +307 footrest footrest 11 39 6 foot rest otherfurniture Furniture stool n03380724 footstool.n.01 stool 19 +410 piano bench piano bench 11 39 6 piano bench otherfurniture Furniture bench bench 2828884 n02828884 bench.n.01 seating 34 +730 breakfast bar breakfast bar 11 38 7 bar otherstructure Objects counter 26 +216 step stool step stool 11 40 7 step stool otherprop Objects stool n04315713 step_stool.n.01 stool 19 +1192 hand rail hand rail 11 38 7 railing otherstructure Objects railing 30 +119 vending machine vending machine 11 40 7 machine otherprop Objects n04525305 vending_machine.n.01 appliances 37 +682 ceiling fan ceiling fan 11 40 7 fan otherprop Objects n03320046 fan.n.01 misc 40 +434 swiffer swiffer 11 40 7 otherprop Objects objects 39 +126 foosball table foosball table 11 39 6 foosball table otherfurniture Furniture table table table 4379243 n04379243 table.n.02 table 5 +919 jar jar 11 40 7 jar otherprop Objects jar 3593526 n03593526 jar.n.01 objects 39 +85 footstool footstool 11 39 6 ottoman otherfurniture Furniture stool n03380724 footstool.n.01 stool 19 +1193 folded table folded table 10 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +108 round table round table 10 7 10 table table Table table table table 4379243 n04114554 round_table.n.02 table 5 +135 hamper hamper 10 40 7 basket otherprop Objects basket 2801938 n03482405 hamper.n.02 objects 39 +1194 poster tube poster tube 10 40 7 otherprop Objects objects 39 +432 case case 10 40 7 case otherprop Objects objects 39 +53 carpet carpet 10 40 7 rug otherprop Objects n04118021 rug.n.01 floor 2 +1195 thermostat thermostat 10 40 7 otherprop Objects n04422875 thermostat.n.01 misc 40 +111 coat coat 10 40 7 jacket otherprop Objects n03057021 coat.n.01 clothes 38 +305 water fountain water fountain 10 38 7 water fountain otherstructure Objects n03241335 drinking_fountain.n.01 misc 40 +1125 smoke detector smoke detector 10 40 7 otherprop Objects misc 40 +13 pillows pillow 9 18 7 pillow pillow Objects pillow 3938244 n03938244 pillow.n.01 cushion 8 +1196 flip flops flip flops 9 40 7 shoe otherprop Objects n04199027 shoe.n.01 clothes 38 +1197 cloth cloth 9 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +1198 banner banner 9 40 7 otherprop Objects n02788021 banner.n.01 misc 40 +1199 clothes hanger clothes hanger 9 40 7 otherprop Objects n03057920 coat_hanger.n.01 objects 39 +1200 whiteboard eraser whiteboard eraser 9 40 7 otherprop Objects objects 39 +378 iron iron 9 40 7 otherprop Objects n03584829 iron.n.04 objects 39 +591 instrument case instrument case 9 40 7 case otherprop Objects objects 39 +49 toilet paper rolls toilet paper 9 40 7 toilet paper otherprop Objects n15075141 toilet_tissue.n.01 objects 39 +92 soap soap 9 40 7 soap otherprop Objects n04253437 soap.n.01 objects 39 +1098 block block 9 40 7 otherprop Objects misc 40 +291 wall hanging wall hanging 8 40 7 otherprop Objects n03491178 hanging.n.01 picture 6 +1063 kitchen island kitchen island 8 38 7 kitchen island otherstructure Objects n03620600 kitchen_island.n.01 counter 26 +107 pipes pipe 8 38 7 otherstructure Objects misc 40 +1135 toothbrush toothbrush 8 40 7 toothbrush otherprop Objects n04453156 toothbrush.n.01 objects 39 +189 shirt shirt 8 40 7 otherprop Objects n04197391 shirt.n.01 clothes 38 +245 cutting board cutting board 8 40 7 cutting board otherprop Objects n03025513 chopping_board.n.01 objects 39 +194 vase vase 8 40 7 vase otherprop Objects vase jar 3593526 n04522168 vase.n.01 objects 39 +1201 shower control valve shower control valve 8 38 7 otherstructure Objects n04208936 shower.n.01 shower 23 +386 exercise machine exercise machine 8 40 7 machine otherprop Objects gym_equipment 33 +1202 compost bin compost bin 8 39 6 garbage bin otherfurniture Furniture trash_bin 2747177 n02747177 ashcan.n.01 objects 39 +857 shorts shorts 8 40 7 shorts otherprop Objects clothes 38 +452 tire tire 8 40 7 otherprop Objects n04440749 tire.n.01 objects 39 +1203 teddy bear teddy bear 7 40 7 stuffed animal otherprop Objects n04399382 teddy.n.01 objects 39 +346 bathrobe bathrobe 7 40 7 otherprop Objects n02807616 bathrobe.n.01 clothes 38 +152 handrail handrail 7 38 7 railing otherstructure Objects n02788148 bannister.n.02 railing 30 +83 faucet faucet 7 40 7 faucet otherprop Objects faucet 3325088 n03325088 faucet.n.01 misc 40 +1204 pantry wall pantry wall 7 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +726 thermos thermos 7 40 7 flask otherprop Objects bottle bottle 2876657 n04422727 thermos.n.01 objects 39 +61 rug rug 7 40 7 rug otherprop Objects n04118021 rug.n.01 floor 2 +39 couch cushions cushion 7 18 7 pillow pillow Objects n03151500 cushion.n.03 cushion 8 +1117 tripod tripod 7 39 6 stand otherfurniture Furniture n04485082 tripod.n.01 objects 39 +540 mailbox mailbox 7 29 7 box box Objects mailbox 3710193 n03710193 mailbox.n.01 misc 40 +1205 tupperware tupperware 7 40 7 otherprop Objects objects 39 +415 shoe rack shoe rack 7 40 7 shoe rack otherprop Objects shelving 31 +31 towels towel 6 27 7 towel towel Objects n04459362 towel.n.01 towel 20 +1206 beer bottles beer bottle 6 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +153 treadmill treadmill 6 39 6 treadmill otherfurniture Furniture n04477387 treadmill.n.01 gym_equipment 33 +1207 salt salt 6 40 7 otherprop Objects objects 39 +129 chest chest 6 39 6 chest otherfurniture Furniture dresser dresser chest_of_drawers 13 +220 dispenser dispenser 6 40 7 otherprop Objects n03210683 dispenser.n.01 objects 39 +1208 mirror doors mirror door 6 8 12 door door Wall door n03221720 door.n.01 door 4 +231 remote remote 6 40 7 otherprop Objects remote_control 4074963 n04074963 remote_control.n.01 objects 39 +1209 folded ladder folded ladder 6 39 6 ladder otherfurniture Furniture stairs n03632277 ladder.n.01 misc 40 +39 cushion cushion 6 18 7 pillow pillow Objects n03151500 cushion.n.03 cushion 8 +1210 carton carton 6 40 7 otherprop Objects objects 39 +117 step step 6 38 7 otherstructure Objects n04314914 step.n.04 misc 40 +822 drying rack drying rack 6 39 6 drying rack otherfurniture Furniture shelving 31 +238 slippers slipper 6 40 7 shoe otherprop Objects n04241394 slipper.n.01 clothes 38 +143 pool table pool table 6 39 6 pool table otherfurniture Furniture table table table 4379243 n03982430 pool_table.n.01 table 5 +1211 soda stream soda stream 6 40 7 otherprop Objects objects 39 +228 toilet brush toilet brush 6 40 7 toilet brush otherprop Objects objects 39 +494 loft bed loft bed 6 4 1 bed bed Bed bed bed bed 2818832 n02818832 bed.n.01 bed 11 +226 cooking pot cooking pot 6 40 7 pot otherprop Objects objects 39 +91 heater heater 6 39 6 heater otherfurniture Furniture n03508101 heater.n.01 misc 40 +1072 messenger bag messenger bag 6 37 7 bag bag Objects objects 39 +435 stapler stapler 6 40 7 stapler otherprop Objects n04303497 stapler.n.01 objects 39 +1165 closet walls closet wall 5 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +345 scanner scanner 5 40 7 otherprop Objects appliances 37 +893 elliptical machine elliptical machine 5 40 7 machine otherprop Objects gym_equipment 33 +621 kettle kettle 5 40 7 pot otherprop Objects n03612814 kettle.n.01 objects 39 +1212 metronome metronome 5 40 7 otherprop Objects n03757604 metronome.n.01 objects 39 +297 dumbell dumbell 5 40 7 otherprop Objects objects 39 +1213 music book music book 5 23 2 book books Books n02870526 book.n.11 objects 39 +1214 rice cooker rice cooker 5 40 7 otherprop Objects objects 39 +1215 dart board dart board 5 38 7 board otherstructure Objects n03162940 dartboard.n.01 objects 39 +529 sewing machine sewing machine 5 40 7 sewing machine otherprop Objects n04179913 sewing_machine.n.01 objects 39 +1216 grab bar grab bar 5 38 7 railing otherstructure Objects railing 30 +1217 flowerpot flowerpot 5 40 7 vase otherprop Objects vase jar 3593526 n04522168 vase.n.01 objects 39 +1218 painting painting 5 11 8 picture picture Picture n03931044 picture.n.01 picture 6 +1219 railing railing 5 38 7 railing otherstructure Objects n04047401 railing.n.01 railing 30 +1220 stair stair 5 38 7 stairs otherstructure Objects stairs n04314914 step.n.04 stairs 16 +525 toolbox toolbox 5 39 6 chest otherfurniture Furniture n04452615 toolbox.n.01 objects 39 +204 nerf gun nerf gun 5 40 7 otherprop Objects objects 39 +693 binders binder 5 40 7 binder otherprop Objects objects 39 +179 desk lamp desk lamp 5 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +1221 quadcopter quadcopter 5 40 7 otherprop Objects objects 39 +1222 pitcher pitcher 5 40 7 pitcher otherprop Objects n03950228 pitcher.n.02 objects 39 +1223 hanging hanging 5 40 7 otherprop Objects misc 40 +1224 mail mail 5 40 7 otherprop Objects misc 40 +1225 closet ceiling closet ceiling 5 22 3 ceiling ceiling Ceiling n02990373 ceiling.n.01 ceiling 17 +1226 hoverboard hoverboard 5 40 7 otherprop Objects objects 39 +1227 beanbag chair beanbag chair 5 39 6 bean bag otherfurniture Furniture n02816656 beanbag.n.01 chair 3 +571 water heater water heater 5 40 7 water heater otherprop Objects n04560113 water_heater.n.01 misc 40 +1228 spray bottle spray bottle 5 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +556 rope rope 5 40 7 rope otherprop Objects n04108268 rope.n.01 objects 39 +280 plastic container plastic container 5 40 7 container otherprop Objects objects 39 +1229 soap bottle soap bottle 5 40 7 soap otherprop Objects objects 39 +1230 ikea bag ikea bag 4 37 7 bag bag Objects 2773838 n02773838 bag.n.06 objects 39 +1231 sleeping bag sleeping bag 4 40 7 otherprop Objects n04235860 sleeping_bag.n.01 objects 39 +1232 duffel bag duffel bag 4 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +746 frying pan frying pan 4 40 7 frying pan otherprop Objects n03400231 frying_pan.n.01 objects 39 +1233 oven mitt oven mitt 4 40 7 otherprop Objects objects 39 +1234 pot pot 4 40 7 pot otherprop Objects n04235860 sleeping_bag.n.01 objects 39 +144 hand dryer hand dryer 4 40 7 otherprop Objects objects 39 +282 dollhouse dollhouse 4 39 6 doll house otherfurniture Furniture n03219483 dollhouse.n.01 objects 39 +167 shampoo bottle shampoo bottle 4 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +1235 hair brush hair brush 4 40 7 otherprop Objects n02908217 brush.n.02 objects 39 +1236 tennis racket tennis racket 4 40 7 otherprop Objects n04409806 tennis_racket.n.01 objects 39 +1237 display case display case 4 40 7 case otherprop Objects objects 39 +234 ping pong table ping pong table 4 39 6 ping pong table otherfurniture Furniture table table table 4379243 n04379243 table.n.02 table 5 +563 boiler boiler 4 40 7 otherprop Objects misc 40 +1238 bag of coffee beans bag of coffee beans 4 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +1239 bananas banana 4 40 7 otherprop Objects n00021265 food.n.01 objects 39 +1240 carseat carseat 4 40 7 otherprop Objects misc 40 +366 helmet helmet 4 40 7 otherprop Objects helmet 3513137 n03513137 helmet.n.02 clothes 38 +816 umbrella umbrella 4 40 7 umbrella otherprop Objects n04507155 umbrella.n.01 objects 39 +1241 coffee box coffee box 4 40 7 otherprop Objects objects 39 +719 envelope envelope 4 40 7 envelope otherprop Objects n03291819 envelope.n.01 objects 39 +284 wet floor sign wet floor sign 4 40 7 sign otherprop Objects misc 40 +1242 clothing rack clothing rack 4 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +247 controller controller 4 40 7 otherprop Objects n03096960 control.n.09 objects 39 +1243 bath walls bathroom wall 4 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +1244 podium podium 4 39 6 otherfurniture Furniture n03159640 dais.n.01 furniture 36 +1245 storage box storage box 4 29 7 box box Objects n02883344 box.n.01 objects 39 +1246 dolly dolly 4 40 7 otherprop Objects misc 40 +1247 shampoo shampoo 3 40 7 otherprop Objects n04183516 shampoo.n.01 objects 39 +592 paper tray paper tray 3 40 7 paper tray otherprop Objects objects 39 +385 cabinet door cabinet door 3 8 12 door door Wall door door 4 +1248 changing station changing station 3 40 7 otherprop Objects misc 40 +1249 poster printer poster printer 3 40 7 printer otherprop Objects printer 4004475 n04004475 printer.n.03 appliances 37 +133 screen screen 3 40 7 otherprop Objects n03151077 curtain.n.01 curtain 12 +301 soap bar soap bar 3 38 7 bar otherstructure Objects objects 39 +1250 crutches crutches 3 40 7 otherprop Objects n03141823 crutch.n.01 objects 39 +379 studio light studio light 3 38 7 light otherstructure Objects lighting 28 +130 stack of cups cup 3 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 +1251 toilet flush button toilet flush button 3 40 7 otherprop Objects objects 39 +450 trunk trunk 3 40 7 otherprop Objects misc 40 +1252 grocery bag grocery bag 3 37 7 bag bag Objects suitcase 2773838 n03461288 grocery_bag.n.01 objects 39 +316 plastic bin plastic bin 3 40 7 bin otherprop Objects objects 39 +1253 pizza box pizza box 3 29 7 box box Objects objects 39 +385 cabinet doors cabinet door 3 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 door 4 +1254 legs legs 3 31 7 person person Objects person n05217688 person.n.02 misc 40 +461 car car 3 40 7 car otherprop Objects car car 2958343 n02958343 car.n.01 misc 40 +1255 shaving cream shaving cream 3 40 7 otherprop Objects n04186051 shaving_cream.n.01 objects 39 +1256 luggage stand luggage stand 3 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +599 shredder shredder 3 40 7 otherprop Objects n04210120 shredder.n.01 objects 39 +281 statue statue 3 40 7 sculpture otherprop Objects n04306847 statue.n.01 misc 40 +1257 urinal urinal 3 33 7 toilet toilet Objects toilet toilet n04515991 urinal.n.01 toilet 18 +1258 hose hose 3 40 7 otherprop Objects n03539875 hose.n.03 misc 40 +1259 bike pump bike pump 3 40 7 otherprop Objects objects 39 +319 coatrack coatrack 3 40 7 otherprop Objects n03059103 coatrack.n.01 shelving 31 +1260 bear bear 3 40 7 otherprop Objects objects 39 +28 wall lamp lamp 3 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +1261 humidifier humidifier 3 40 7 otherprop Objects objects 39 +546 toothpaste toothpaste 3 40 7 toothpaste otherprop Objects objects 39 +1262 mouthwash bottle mouthwash bottle 3 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +1263 poster cutter poster cutter 3 40 7 otherprop Objects objects 39 +1264 golf bag golf bag 3 37 7 bag bag Objects suitcase 2773838 n03445617 golf_bag.n.01 objects 39 +1265 food container food container 3 40 7 container otherprop Objects n03094503 container.n.01 objects 39 +1266 camera camera 3 40 7 otherprop Objects objects 39 +28 table lamp lamp 3 35 7 lamp lamp Objects lamp lamp 3636649 n04380533 table_lamp.n.01 lighting 28 +1267 yoga mat yoga mat 3 20 5 floor mat floor mat Floor n03727837 mat.n.01 floor 2 +1268 card card 3 40 7 otherprop Objects objects 39 +1269 mug mug 3 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 +188 shower doors shower door 3 38 7 otherstructure Objects n04208936 shower.n.01 door 4 +689 cardboard cardboard 3 40 7 otherprop Objects objects 39 +1270 rack stand rack stand 3 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +1271 boxes of paper boxes of paper 3 29 7 box box Objects n02883344 box.n.01 objects 39 +1272 flag flag 3 40 7 otherprop Objects misc 40 +354 futon futon 3 39 6 mattress otherfurniture Furniture n03408444 futon.n.01 sofa 10 +339 magazine magazine 3 40 7 magazine otherprop Objects n06595351 magazine.n.01 objects 39 +1009 exit sign exit sign 3 40 7 exit sign otherprop Objects misc 40 +1273 rolled poster rolled poster 3 40 7 otherprop Objects objects 39 +1274 wheel wheel 3 40 7 otherprop Objects objects 39 +15 pictures picture 3 11 8 picture picture Picture n03931044 picture.n.01 picture 6 +1275 blackboard eraser blackboard eraser 3 40 7 eraser otherprop Objects n03294833 eraser.n.01 objects 39 +361 organizer organizer 3 40 7 otherprop Objects n03918737 personal_digital_assistant.n.01 objects 39 +1276 doll doll 3 40 7 toy otherprop Objects n03219135 doll.n.01 objects 39 +326 book rack book rack 3 39 6 bookrack otherfurniture Furniture objects 39 +1277 laundry bag laundry bag 3 40 7 laundry basket otherprop Objects basket 2801938 n03050864 clothes_hamper.n.01 objects 39 +1278 sponge sponge 3 40 7 otherprop Objects n01906749 sponge.n.04 objects 39 +116 seating seat 3 39 6 furniture otherfurniture Furniture n04161981 seat.n.03 furniture 36 +1184 folded chairs folded chair 2 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +1279 lotion bottle lotion bottle 2 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +212 can can 2 40 7 can otherprop Objects can 2946921 n02946921 can.n.01 objects 39 +1280 lunch box lunch box 2 40 7 otherprop Objects objects 39 +1281 food display food display 2 40 7 otherprop Objects misc 40 +794 storage shelf storage shelf 2 40 7 otherprop Objects shelving 31 +1282 sliding wood door sliding wood door 2 40 7 otherprop Objects door 4 +955 pants pants 2 40 7 otherprop Objects n04489008 trouser.n.01 clothes 38 +387 wood wood 2 40 7 otherprop Objects misc 40 +69 boards board 2 38 7 board otherstructure Objects board_panel 35 +65 bottles bottle 2 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +523 washcloth washcloth 2 40 7 otherprop Objects n04554523 washcloth.n.01 towel 20 +389 workbench workbench 2 39 6 bench otherfurniture Furniture bench table 4379243 n04600486 workbench.n.01 table 5 +29 open kitchen cabinet kitchen cabinet 2 3 6 cabinet cabinet Furniture n02933112 cabinet.n.01 cabinet 7 +1283 organizer shelf organizer shelf 2 15 6 shelves shelves Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +146 frame frame 2 38 7 otherstructure Objects misc 40 +130 cups cup 2 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 +372 exercise ball exercise ball 2 40 7 ball otherprop Objects n04285146 sports_equipment.n.01 gym_equipment 33 +289 easel easel 2 39 6 stand otherfurniture Furniture n03262809 easel.n.01 furniture 36 +440 garbage bag garbage bag 2 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +321 roomba roomba 2 40 7 otherprop Objects objects 39 +976 garage door garage door 2 38 7 garage door otherstructure Objects door door 4 +1256 luggage rack luggage stand 2 39 6 stand otherfurniture Furniture n04038440 shelving 31 +1284 bike lock bike lock 2 40 7 otherprop Objects objects 39 +1285 briefcase briefcase 2 40 7 otherprop Objects n02900705 briefcase.n.01 objects 39 +357 hand towel hand towel 2 27 7 towel towel Objects n03490006 hand_towel.n.01 towel 20 +1286 bath products bath product 2 40 7 otherprop Objects objects 39 +1287 star star 2 40 7 otherprop Objects n09444783 star.n.03 misc 40 +365 map map 2 40 7 map otherprop Objects n03720163 map.n.01 misc 40 +1288 coffee bean bag coffee bean bag 2 37 7 bag bag Objects suitcase 2773838 n02773838 bag.n.06 objects 39 +81 headboard headboard 2 39 6 headboard otherfurniture Furniture n03502200 headboard.n.01 bed 11 +1289 ipad ipad 2 40 7 otherprop Objects objects 39 +1290 display rack display rack 2 39 6 stand otherfurniture Furniture n04038440 rack.n.05 shelving 31 +948 traffic cone traffic cone 2 40 7 cone otherprop Objects cone objects 39 +174 toiletry toiletry 2 40 7 otherprop Objects n04447443 toiletry.n.01 objects 39 +1028 canopy canopy 2 40 7 otherprop Objects misc 40 +1291 massage chair massage chair 2 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +1292 paper organizer paper organizer 2 40 7 otherprop Objects objects 39 +1005 barricade barricade 2 40 7 otherprop Objects misc 40 +235 platform platform 2 38 7 otherstructure Objects misc 40 +1293 cap cap 2 40 7 hat otherprop Objects n03497657 hat.n.01 clothes 38 +1294 dumbbell plates dumbbell plates 2 40 7 otherprop Objects objects 39 +1295 elevator elevator 2 38 7 otherstructure Objects misc 40 +1296 cooking pan cooking pan 2 40 7 pan otherprop Objects n03880531 pan.n.01 objects 39 +1297 trash bag trash bag 2 37 7 bag bag Objects objects 39 +1298 santa santa 2 40 7 otherprop Objects misc 40 +1299 jewelry box jewelry box 2 29 7 box box Objects n02883344 box.n.01 objects 39 +1300 boat boat 2 40 7 otherprop Objects misc 40 +1301 sock sock 2 21 7 clothes clothes Objects n04254777 sock.n.01 clothes 38 +1051 kinect kinect 2 40 7 kinect otherprop Objects objects 39 +566 crib crib 2 39 6 crib otherfurniture Furniture furniture 36 +1302 plastic storage bin plastic storage bin 2 40 7 container otherprop Objects n03094503 container.n.01 objects 39 +1062 cooler cooler 2 24 6 refridgerator refridgerator Furniture n03102654 cooler.n.01 appliances 37 +1303 kitchen apron kitchen apron 2 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +1304 dishwashing soap bottle dishwashing soap bottle 2 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +1305 xbox controller xbox controller 2 40 7 otherprop Objects objects 39 +1306 banana holder banana holder 2 40 7 otherprop Objects objects 39 +298 ping pong paddle ping pong paddle 2 40 7 otherprop Objects table 5 +1307 airplane airplane 2 40 7 otherprop Objects misc 40 +1308 conditioner bottle conditioner bottle 2 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +1309 tea kettle tea kettle 2 40 7 tea kettle otherprop Objects n04397768 teakettle.n.01 objects 39 +43 bedframe bedframe 2 39 6 otherfurniture Furniture n02822579 bedstead.n.01 bed 11 +1310 wood beam wood beam 2 38 7 otherstructure Objects beam 29 +593 toilet paper package toilet paper package 2 40 7 otherprop Objects objects 39 +1311 wall mounted coat rack wall mounted coat rack 2 40 7 otherprop Objects n03059103 coatrack.n.01 shelving 31 +1312 film light film light 2 40 7 otherprop Objects lighting 28 +749 ceiling lamp ceiling lamp 1 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +623 chain chain 1 40 7 otherprop Objects chair 3 +1313 sofa sofa 1 6 9 sofa sofa Sofa sofa sofa sofa 4256520 n04256520 sofa.n.01 sofa 10 +99 closet wardrobe wardrobe 1 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 furniture 36 +265 sweater sweater 1 40 7 otherprop Objects n04370048 sweater.n.01 clothes 38 +1314 kitchen mixer kitchen mixer 1 40 7 otherprop Objects appliances 37 +99 wardrobe wardrobe 1 39 6 wardrobe otherfurniture Furniture wardrobe n04550184 wardrobe.n.01 furniture 36 +1315 water softener water softener 1 40 7 otherprop Objects misc 40 +448 banister banister 1 38 7 banister otherstructure Objects n02788148 bannister.n.02 railing 30 +257 trolley trolley 1 40 7 trolley otherprop Objects n04335435 streetcar.n.01 misc 40 +1316 pantry shelf pantry shelf 1 15 6 shelves shelves Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +786 sofa bed sofa bed 1 4 1 bed bed Bed bed bed bed 2818832 n02818832 bed.n.01 bed 11 +801 loofa loofa 1 40 7 otherprop Objects objects 39 +972 shower faucet handle shower faucet handle 1 40 7 handle otherprop Objects shower 23 +1317 toy piano toy piano 1 40 7 toy otherprop Objects n03964744 plaything.n.01 objects 39 +1318 fish fish 1 40 7 otherprop Objects n02512053 fish.n.01 objects 39 +75 file cabinets file cabinet 1 3 6 cabinet cabinet Furniture cabinet 2933112 n03337140 file.n.03 cabinet 7 +657 cat litter box cat litter box 1 29 7 box box Objects objects 39 +561 electric panel electric panel 1 40 7 otherprop Objects misc 40 +93 suitcases suitcase 1 40 7 luggage otherprop Objects n02774630 baggage.n.01 objects 39 +513 curtain rod curtain rod 1 38 7 curtain rod otherstructure Objects curtain 12 +411 bunk bed bunk bed 1 39 6 bunk bed otherfurniture Furniture bed bed bed 2818832 n02920259 bunk_bed.n.01 bed 11 +1122 chandelier chandelier 1 38 7 chandelier otherstructure Objects n03005285 chandelier.n.01 lighting 28 +922 tape tape 1 40 7 tape otherprop Objects objects 39 +88 plates plate 1 40 7 otherprop Objects n03959485 plate.n.04 objects 39 +518 alarm alarm 1 40 7 alarm otherprop Objects clock 3046257 n02694662 alarm_clock.n.01 objects 39 +814 fire hose fire hose 1 40 7 otherprop Objects n03346004 fire_hose.n.01 misc 40 +1319 toy dinosaur toy dinosaur 1 40 7 toy otherprop Objects n03964744 plaything.n.01 objects 39 +1320 cone cone 1 40 7 otherprop Objects objects 39 +649 glass doors glass door 1 8 12 door door Wall door n03221720 door.n.01 door 4 +607 hatrack hatrack 1 40 7 otherprop Objects n03059103 coatrack.n.01 shelving 31 +819 subwoofer subwoofer 1 40 7 speaker otherprop Objects speaker 3691459 n04349401 subwoofer.n.01 objects 39 +1321 fire sprinkler fire sprinkler 1 40 7 otherprop Objects misc 40 +1322 trash cabinet trash cabinet 1 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +1204 pantry walls pantry wall 1 1 12 wall wall Wall n04546855 wall.n.01 wall 1 +227 photo photo 1 40 7 photo otherprop Objects n03925226 photograph.n.01 picture 6 +817 barrier barrier 1 40 7 otherprop Objects n02796623 barrier.n.01 misc 40 +130 stacks of cups cup 1 40 7 otherprop Objects n03147509 cup.n.01 objects 39 +712 beachball beachball 1 40 7 ball otherprop Objects n02814224 beach_ball.n.01 objects 39 +1323 folded boxes folded boxes 1 40 7 otherprop Objects objects 39 +1324 contact lens solution bottle contact lens solution bottle 1 40 7 bottle otherprop Objects bottle bottle 2876657 n02876657 bottle.n.01 objects 39 +673 covered box covered box 1 29 7 box box Objects objects 39 +459 folder folder 1 40 7 folder otherprop Objects n03376279 folder.n.02 objects 39 +643 mail trays mail tray 1 40 7 mail tray otherprop Objects objects 39 +238 slipper slipper 1 40 7 otherprop Objects n04241394 slipper.n.01 clothes 38 +765 magazine rack magazine rack 1 39 6 stand otherfurniture Furniture n03704549 magazine_rack.n.01 shelving 31 +1008 sticker sticker 1 40 7 sticker otherprop Objects n07272545 gummed_label.n.01 objects 39 +225 lotion lotion 1 40 7 otherprop Objects n03690938 lotion.n.01 objects 39 +1083 buddha buddha 1 40 7 otherprop Objects objects 39 +813 file organizer file organizer 1 40 7 otherprop Objects objects 39 +138 paper towel rolls paper towel roll 1 40 7 paper towel otherprop Objects n03887697 paper_towel.n.01 towel 20 +1145 night lamp night lamp 1 35 7 lamp lamp Objects lamp lamp 3636649 n03636649 lamp.n.02 lighting 28 +796 fuse box fuse box 1 40 7 otherprop Objects misc 40 +1325 knife block knife block 1 40 7 otherprop Objects objects 39 +363 furnace furnace 1 39 6 furnace otherfurniture Furniture n03404449 furnace.n.01 +1174 cd cases cd case 1 40 7 otherprop Objects objects 39 +38 stools stool 1 40 7 stool otherprop Objects stool n04326896 stool.n.01 stool 19 +1326 hand sanitzer dispenser hand sanitzer dispenser 1 40 7 otherprop Objects n04254120 soap_dispenser.n.01 objects 39 +997 teapot teapot 1 40 7 tea pot otherprop Objects n04398044 teapot.n.01 objects 39 +1327 pen holder pen holder 1 40 7 otherprop Objects objects 39 +1328 tray rack tray rack 1 40 7 otherprop Objects objects 39 +1329 wig wig 1 40 7 otherprop Objects n04584207 wig.n.01 objects 39 +182 switch switch 1 40 7 otherprop Objects n04372370 switch.n.01 misc 40 +280 plastic containers plastic container 1 40 7 container otherprop Objects n03094503 container.n.01 objects 39 +1330 night light night light 1 40 7 otherprop Objects lighting 28 +1331 notepad notepad 1 40 7 otherprop Objects objects 39 +1332 mail bin mail bin 1 40 7 otherprop Objects misc 40 +1333 elevator button elevator button 1 40 7 otherprop Objects misc 40 +939 gaming wheel gaming wheel 1 40 7 otherprop Objects objects 39 +1334 drum set drum set 1 40 7 otherprop Objects objects 39 +480 cosmetic bag cosmetic bag 1 37 7 bag bag Objects objects 39 +907 coffee mug coffee mug 1 40 7 vessel otherprop Objects cup or mug 3797390 n03063599 coffee_mug.n.01 objects 39 +1335 closet shelf closet shelf 1 15 6 shelves shelves Furniture bookshelf bookshelf 2871439 n02871439 bookshelf.n.01 shelving 31 +1336 baby mobile baby mobile 1 40 7 otherprop Objects objects 39 +829 diaper bin diaper bin 1 40 7 bin otherprop Objects objects 39 +947 door wall door wall 1 1 12 wall wall Wall wall 1 +1116 stepstool stepstool 1 40 7 step stool otherprop Objects objects 39 +599 paper shredder shredder 1 40 7 otherprop Objects n04210120 shredder.n.01 objects 39 +733 dress rack dress rack 1 40 7 otherprop Objects n03238762 dress_rack.n.01 misc 40 +123 cover cover 1 40 7 blanket otherprop Objects objects 39 +506 shopping bag shopping bag 1 37 7 bag bag Objects n04204081 shopping_bag.n.01 objects 39 +569 sliding door sliding door 1 8 12 door door Wall door n04239074 sliding_door.n.01 door 4 +1337 exercise bike exercise bike 1 40 7 machine otherprop Objects n04210120 shredder.n.01 gym_equipment 33 +1338 recliner chair recliner chair 1 5 4 chair chair Chair chair chair chair 3001627 n03238762 dress_rack.n.01 chair 3 +1314 kitchenaid mixer kitchen mixer 1 40 7 otherprop Objects appliances 37 +1339 soda can soda can 1 40 7 can otherprop Objects can 2946921 n02946921 can.n.01 objects 39 +1340 stovetop stovetop 1 38 7 stove otherstructure Objects stove 4330267 n04330267 stove.n.02 appliances 37 +851 stepladder stepladder 1 39 6 ladder otherfurniture Furniture stairs n04315599 step_ladder.n.01 stairs 16 +142 tap tap 1 40 7 faucet otherprop Objects faucet 3325088 n04559451 water_faucet.n.01 objects 39 +436 cable cable 1 40 7 cables otherprop Objects objects 39 +1341 baby changing station baby changing station 1 39 6 otherfurniture Furniture furniture 36 +1342 costume costume 1 21 7 clothes clothes Objects n02728440 apparel.n.01 clothes 38 +885 rocking chair rocking chair 1 5 4 chair chair Chair chair chair chair 3001627 n04099969 rocking_chair.n.01 chair 3 +693 binder binder 1 40 7 binder otherprop Objects objects 39 +815 media center media center 1 3 6 cabinet cabinet Furniture cabinet 2933112 n02933112 cabinet.n.01 cabinet 7 +401 towel rack towel rack 1 40 7 otherprop Objects n04459773 towel_rack.n.01 misc 40 +1343 medal medal 1 40 7 otherprop Objects objects 39 +1184 stack of folded chairs folded chair 1 5 4 chair chair Chair chair chair chair 3001627 n03001627 chair.n.01 chair 3 +1344 telescope telescope 1 40 7 otherprop Objects n04403638 telescope.n.01 objects 39 +1345 closet doorframe closet doorframe 1 8 12 door door Wall door door 4 +160 glass glass 1 38 7 glass otherstructure Objects n03438257 glass.n.02 misc 40 +1126 baseball cap baseball cap 1 40 7 otherprop Objects cap 2954340 n02799323 baseball_cap.n.01 clothes 38 +1346 battery disposal jar battery disposal jar 1 40 7 jar otherprop Objects jar 3593526 n03593526 jar.n.01 objects 39 +332 mop mop 1 40 7 otherprop Objects n04367480 swab.n.02 objects 39 +397 tank tank 1 40 7 otherprop Objects objects 39 +643 mail tray mail tray 1 40 7 mail tray otherprop Objects objects 39 +551 centerpiece centerpiece 1 40 7 centerpiece otherprop Objects n02994419 centerpiece.n.02 objects 39 +1163 object stick 1 40 7 stick otherprop Objects objects 39 +1347 closet floor closet floor 1 2 5 floor floor Floor n03365592 floor.n.01 floor 2 +1348 dryer sheets dryer sheets 1 40 7 otherprop Objects objects 39 +803 bycicle bycicle 1 40 7 otherprop Objects misc 40 +484 flower stand flower stand 1 39 6 stand otherfurniture Furniture furniture 36 +1349 air mattress air mattress 1 4 1 bed bed Bed bed bed bed 2818832 n02690809 air_mattress.n.01 bed 11 +1350 clip clip 1 40 7 otherprop Objects objects 39 +222 side table side table 1 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +1253 pizza boxes pizza box 1 29 7 box box Objects n02883344 box.n.01 objects 39 +1351 display display 1 39 7 otherfurniture Furniture n03211117 display.n.06 misc 40 +1352 postcard postcard 1 40 7 otherprop Objects objects 39 +828 display sign display sign 1 40 7 sign otherprop Objects misc 40 +1353 paper towel paper towel 1 40 7 paper towel otherprop Objects n03887697 paper_towel.n.01 towel 20 +612 boots boot 1 40 7 shoe otherprop Objects n04199027 shoe.n.01 clothes 38 +1354 tennis racket bag tennis racket bag 1 40 7 otherprop Objects objects 39 +1355 air hockey table air hockey table 1 7 10 table table Table table table table 4379243 n04379243 table.n.02 table 5 +1301 socks sock 1 21 7 clothes clothes Objects n04254777 sock.n.01 clothes 38 +1356 food bag food bag 1 37 7 bag bag Objects objects 39 +1199 clothes hangers clothes hanger 1 40 7 otherprop Objects n03057920 coat_hanger.n.01 misc 40 +1357 starbucks cup starbucks cup 1 40 7 cup otherprop Objects cup cup or mug 3797390 n03797390 mug.n.04 objects 39 \ No newline at end of file diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_test.txt b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_test.txt new file mode 100644 index 0000000000000000000000000000000000000000..79d15b0ee4afa889883562a722b837b78ee8ce4b --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_test.txt @@ -0,0 +1,100 @@ +scene0707_00 +scene0708_00 +scene0709_00 +scene0710_00 +scene0711_00 +scene0712_00 +scene0713_00 +scene0714_00 +scene0715_00 +scene0716_00 +scene0717_00 +scene0718_00 +scene0719_00 +scene0720_00 +scene0721_00 +scene0722_00 +scene0723_00 +scene0724_00 +scene0725_00 +scene0726_00 +scene0727_00 +scene0728_00 +scene0729_00 +scene0730_00 +scene0731_00 +scene0732_00 +scene0733_00 +scene0734_00 +scene0735_00 +scene0736_00 +scene0737_00 +scene0738_00 +scene0739_00 +scene0740_00 +scene0741_00 +scene0742_00 +scene0743_00 +scene0744_00 +scene0745_00 +scene0746_00 +scene0747_00 +scene0748_00 +scene0749_00 +scene0750_00 +scene0751_00 +scene0752_00 +scene0753_00 +scene0754_00 +scene0755_00 +scene0756_00 +scene0757_00 +scene0758_00 +scene0759_00 +scene0760_00 +scene0761_00 +scene0762_00 +scene0763_00 +scene0764_00 +scene0765_00 +scene0766_00 +scene0767_00 +scene0768_00 +scene0769_00 +scene0770_00 +scene0771_00 +scene0772_00 +scene0773_00 +scene0774_00 +scene0775_00 +scene0776_00 +scene0777_00 +scene0778_00 +scene0779_00 +scene0780_00 +scene0781_00 +scene0782_00 +scene0783_00 +scene0784_00 +scene0785_00 +scene0786_00 +scene0787_00 +scene0788_00 +scene0789_00 +scene0790_00 +scene0791_00 +scene0792_00 +scene0793_00 +scene0794_00 +scene0795_00 +scene0796_00 +scene0797_00 +scene0798_00 +scene0799_00 +scene0800_00 +scene0801_00 +scene0802_00 +scene0803_00 +scene0804_00 +scene0805_00 +scene0806_00 diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_train.txt b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_train.txt new file mode 100644 index 0000000000000000000000000000000000000000..ef625f120b812fea5ac507d3b7049fc7ebd2e7e4 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_train.txt @@ -0,0 +1,1201 @@ +scene0191_00 +scene0191_01 +scene0191_02 +scene0119_00 +scene0230_00 +scene0528_00 +scene0528_01 +scene0705_00 +scene0705_01 +scene0705_02 +scene0415_00 +scene0415_01 +scene0415_02 +scene0007_00 +scene0141_00 +scene0141_01 +scene0141_02 +scene0515_00 +scene0515_01 +scene0515_02 +scene0447_00 +scene0447_01 +scene0447_02 +scene0531_00 +scene0503_00 +scene0285_00 +scene0069_00 +scene0584_00 +scene0584_01 +scene0584_02 +scene0581_00 +scene0581_01 +scene0581_02 +scene0620_00 +scene0620_01 +scene0263_00 +scene0263_01 +scene0481_00 +scene0481_01 +scene0020_00 +scene0020_01 +scene0291_00 +scene0291_01 +scene0291_02 +scene0469_00 +scene0469_01 +scene0469_02 +scene0659_00 +scene0659_01 +scene0024_00 +scene0024_01 +scene0024_02 +scene0564_00 +scene0117_00 +scene0027_00 +scene0027_01 +scene0027_02 +scene0028_00 +scene0330_00 +scene0418_00 +scene0418_01 +scene0418_02 +scene0233_00 +scene0233_01 +scene0673_00 +scene0673_01 +scene0673_02 +scene0673_03 +scene0673_04 +scene0673_05 +scene0585_00 +scene0585_01 +scene0362_00 +scene0362_01 +scene0362_02 +scene0362_03 +scene0035_00 +scene0035_01 +scene0358_00 +scene0358_01 +scene0358_02 +scene0037_00 +scene0194_00 +scene0321_00 +scene0293_00 +scene0293_01 +scene0623_00 +scene0623_01 +scene0592_00 +scene0592_01 +scene0569_00 +scene0569_01 +scene0413_00 +scene0313_00 +scene0313_01 +scene0313_02 +scene0480_00 +scene0480_01 +scene0401_00 +scene0517_00 +scene0517_01 +scene0517_02 +scene0032_00 +scene0032_01 +scene0613_00 +scene0613_01 +scene0613_02 +scene0306_00 +scene0306_01 +scene0052_00 +scene0052_01 +scene0052_02 +scene0053_00 +scene0444_00 +scene0444_01 +scene0055_00 +scene0055_01 +scene0055_02 +scene0560_00 +scene0589_00 +scene0589_01 +scene0589_02 +scene0610_00 +scene0610_01 +scene0610_02 +scene0364_00 +scene0364_01 +scene0383_00 +scene0383_01 +scene0383_02 +scene0006_00 +scene0006_01 +scene0006_02 +scene0275_00 +scene0451_00 +scene0451_01 +scene0451_02 +scene0451_03 +scene0451_04 +scene0451_05 +scene0135_00 +scene0065_00 +scene0065_01 +scene0065_02 +scene0104_00 +scene0674_00 +scene0674_01 +scene0448_00 +scene0448_01 +scene0448_02 +scene0502_00 +scene0502_01 +scene0502_02 +scene0440_00 +scene0440_01 +scene0440_02 +scene0071_00 +scene0072_00 +scene0072_01 +scene0072_02 +scene0509_00 +scene0509_01 +scene0509_02 +scene0649_00 +scene0649_01 +scene0602_00 +scene0694_00 +scene0694_01 +scene0101_00 +scene0101_01 +scene0101_02 +scene0101_03 +scene0101_04 +scene0101_05 +scene0218_00 +scene0218_01 +scene0579_00 +scene0579_01 +scene0579_02 +scene0039_00 +scene0039_01 +scene0493_00 +scene0493_01 +scene0242_00 +scene0242_01 +scene0242_02 +scene0083_00 +scene0083_01 +scene0127_00 +scene0127_01 +scene0662_00 +scene0662_01 +scene0662_02 +scene0018_00 +scene0087_00 +scene0087_01 +scene0087_02 +scene0332_00 +scene0332_01 +scene0332_02 +scene0628_00 +scene0628_01 +scene0628_02 +scene0134_00 +scene0134_01 +scene0134_02 +scene0238_00 +scene0238_01 +scene0092_00 +scene0092_01 +scene0092_02 +scene0092_03 +scene0092_04 +scene0022_00 +scene0022_01 +scene0467_00 +scene0392_00 +scene0392_01 +scene0392_02 +scene0424_00 +scene0424_01 +scene0424_02 +scene0646_00 +scene0646_01 +scene0646_02 +scene0098_00 +scene0098_01 +scene0044_00 +scene0044_01 +scene0044_02 +scene0510_00 +scene0510_01 +scene0510_02 +scene0571_00 +scene0571_01 +scene0166_00 +scene0166_01 +scene0166_02 +scene0563_00 +scene0172_00 +scene0172_01 +scene0388_00 +scene0388_01 +scene0215_00 +scene0215_01 +scene0252_00 +scene0287_00 +scene0668_00 +scene0572_00 +scene0572_01 +scene0572_02 +scene0026_00 +scene0224_00 +scene0113_00 +scene0113_01 +scene0551_00 +scene0381_00 +scene0381_01 +scene0381_02 +scene0371_00 +scene0371_01 +scene0460_00 +scene0118_00 +scene0118_01 +scene0118_02 +scene0417_00 +scene0008_00 +scene0634_00 +scene0521_00 +scene0123_00 +scene0123_01 +scene0123_02 +scene0045_00 +scene0045_01 +scene0511_00 +scene0511_01 +scene0114_00 +scene0114_01 +scene0114_02 +scene0070_00 +scene0029_00 +scene0029_01 +scene0029_02 +scene0129_00 +scene0103_00 +scene0103_01 +scene0002_00 +scene0002_01 +scene0132_00 +scene0132_01 +scene0132_02 +scene0124_00 +scene0124_01 +scene0143_00 +scene0143_01 +scene0143_02 +scene0604_00 +scene0604_01 +scene0604_02 +scene0507_00 +scene0105_00 +scene0105_01 +scene0105_02 +scene0428_00 +scene0428_01 +scene0311_00 +scene0140_00 +scene0140_01 +scene0182_00 +scene0182_01 +scene0182_02 +scene0142_00 +scene0142_01 +scene0399_00 +scene0399_01 +scene0012_00 +scene0012_01 +scene0012_02 +scene0060_00 +scene0060_01 +scene0370_00 +scene0370_01 +scene0370_02 +scene0310_00 +scene0310_01 +scene0310_02 +scene0661_00 +scene0650_00 +scene0152_00 +scene0152_01 +scene0152_02 +scene0158_00 +scene0158_01 +scene0158_02 +scene0482_00 +scene0482_01 +scene0600_00 +scene0600_01 +scene0600_02 +scene0393_00 +scene0393_01 +scene0393_02 +scene0562_00 +scene0174_00 +scene0174_01 +scene0157_00 +scene0157_01 +scene0161_00 +scene0161_01 +scene0161_02 +scene0159_00 +scene0254_00 +scene0254_01 +scene0115_00 +scene0115_01 +scene0115_02 +scene0162_00 +scene0163_00 +scene0163_01 +scene0523_00 +scene0523_01 +scene0523_02 +scene0459_00 +scene0459_01 +scene0175_00 +scene0085_00 +scene0085_01 +scene0279_00 +scene0279_01 +scene0279_02 +scene0201_00 +scene0201_01 +scene0201_02 +scene0283_00 +scene0456_00 +scene0456_01 +scene0429_00 +scene0043_00 +scene0043_01 +scene0419_00 +scene0419_01 +scene0419_02 +scene0368_00 +scene0368_01 +scene0348_00 +scene0348_01 +scene0348_02 +scene0442_00 +scene0178_00 +scene0380_00 +scene0380_01 +scene0380_02 +scene0165_00 +scene0165_01 +scene0165_02 +scene0181_00 +scene0181_01 +scene0181_02 +scene0181_03 +scene0333_00 +scene0614_00 +scene0614_01 +scene0614_02 +scene0404_00 +scene0404_01 +scene0404_02 +scene0185_00 +scene0126_00 +scene0126_01 +scene0126_02 +scene0519_00 +scene0236_00 +scene0236_01 +scene0189_00 +scene0075_00 +scene0267_00 +scene0192_00 +scene0192_01 +scene0192_02 +scene0281_00 +scene0420_00 +scene0420_01 +scene0420_02 +scene0195_00 +scene0195_01 +scene0195_02 +scene0597_00 +scene0597_01 +scene0597_02 +scene0041_00 +scene0041_01 +scene0111_00 +scene0111_01 +scene0111_02 +scene0666_00 +scene0666_01 +scene0666_02 +scene0200_00 +scene0200_01 +scene0200_02 +scene0536_00 +scene0536_01 +scene0536_02 +scene0390_00 +scene0280_00 +scene0280_01 +scene0280_02 +scene0344_00 +scene0344_01 +scene0205_00 +scene0205_01 +scene0205_02 +scene0484_00 +scene0484_01 +scene0009_00 +scene0009_01 +scene0009_02 +scene0302_00 +scene0302_01 +scene0209_00 +scene0209_01 +scene0209_02 +scene0210_00 +scene0210_01 +scene0395_00 +scene0395_01 +scene0395_02 +scene0683_00 +scene0601_00 +scene0601_01 +scene0214_00 +scene0214_01 +scene0214_02 +scene0477_00 +scene0477_01 +scene0439_00 +scene0439_01 +scene0468_00 +scene0468_01 +scene0468_02 +scene0546_00 +scene0466_00 +scene0466_01 +scene0220_00 +scene0220_01 +scene0220_02 +scene0122_00 +scene0122_01 +scene0130_00 +scene0110_00 +scene0110_01 +scene0110_02 +scene0327_00 +scene0156_00 +scene0266_00 +scene0266_01 +scene0001_00 +scene0001_01 +scene0228_00 +scene0199_00 +scene0219_00 +scene0464_00 +scene0232_00 +scene0232_01 +scene0232_02 +scene0299_00 +scene0299_01 +scene0530_00 +scene0363_00 +scene0453_00 +scene0453_01 +scene0570_00 +scene0570_01 +scene0570_02 +scene0183_00 +scene0239_00 +scene0239_01 +scene0239_02 +scene0373_00 +scene0373_01 +scene0241_00 +scene0241_01 +scene0241_02 +scene0188_00 +scene0622_00 +scene0622_01 +scene0244_00 +scene0244_01 +scene0691_00 +scene0691_01 +scene0206_00 +scene0206_01 +scene0206_02 +scene0247_00 +scene0247_01 +scene0061_00 +scene0061_01 +scene0082_00 +scene0250_00 +scene0250_01 +scene0250_02 +scene0501_00 +scene0501_01 +scene0501_02 +scene0320_00 +scene0320_01 +scene0320_02 +scene0320_03 +scene0631_00 +scene0631_01 +scene0631_02 +scene0255_00 +scene0255_01 +scene0255_02 +scene0047_00 +scene0265_00 +scene0265_01 +scene0265_02 +scene0004_00 +scene0336_00 +scene0336_01 +scene0058_00 +scene0058_01 +scene0260_00 +scene0260_01 +scene0260_02 +scene0243_00 +scene0603_00 +scene0603_01 +scene0093_00 +scene0093_01 +scene0093_02 +scene0109_00 +scene0109_01 +scene0434_00 +scene0434_01 +scene0434_02 +scene0290_00 +scene0627_00 +scene0627_01 +scene0470_00 +scene0470_01 +scene0137_00 +scene0137_01 +scene0137_02 +scene0270_00 +scene0270_01 +scene0270_02 +scene0271_00 +scene0271_01 +scene0504_00 +scene0274_00 +scene0274_01 +scene0274_02 +scene0036_00 +scene0036_01 +scene0276_00 +scene0276_01 +scene0272_00 +scene0272_01 +scene0499_00 +scene0698_00 +scene0698_01 +scene0051_00 +scene0051_01 +scene0051_02 +scene0051_03 +scene0108_00 +scene0245_00 +scene0369_00 +scene0369_01 +scene0369_02 +scene0284_00 +scene0289_00 +scene0289_01 +scene0286_00 +scene0286_01 +scene0286_02 +scene0286_03 +scene0031_00 +scene0031_01 +scene0031_02 +scene0545_00 +scene0545_01 +scene0545_02 +scene0557_00 +scene0557_01 +scene0557_02 +scene0533_00 +scene0533_01 +scene0116_00 +scene0116_01 +scene0116_02 +scene0611_00 +scene0611_01 +scene0688_00 +scene0294_00 +scene0294_01 +scene0294_02 +scene0295_00 +scene0295_01 +scene0296_00 +scene0296_01 +scene0596_00 +scene0596_01 +scene0596_02 +scene0532_00 +scene0532_01 +scene0637_00 +scene0638_00 +scene0121_00 +scene0121_01 +scene0121_02 +scene0040_00 +scene0040_01 +scene0197_00 +scene0197_01 +scene0197_02 +scene0410_00 +scene0410_01 +scene0305_00 +scene0305_01 +scene0615_00 +scene0615_01 +scene0703_00 +scene0703_01 +scene0555_00 +scene0297_00 +scene0297_01 +scene0297_02 +scene0582_00 +scene0582_01 +scene0582_02 +scene0023_00 +scene0094_00 +scene0013_00 +scene0013_01 +scene0013_02 +scene0136_00 +scene0136_01 +scene0136_02 +scene0407_00 +scene0407_01 +scene0062_00 +scene0062_01 +scene0062_02 +scene0386_00 +scene0318_00 +scene0554_00 +scene0554_01 +scene0497_00 +scene0213_00 +scene0258_00 +scene0323_00 +scene0323_01 +scene0324_00 +scene0324_01 +scene0016_00 +scene0016_01 +scene0016_02 +scene0681_00 +scene0398_00 +scene0398_01 +scene0227_00 +scene0090_00 +scene0066_00 +scene0262_00 +scene0262_01 +scene0155_00 +scene0155_01 +scene0155_02 +scene0352_00 +scene0352_01 +scene0352_02 +scene0038_00 +scene0038_01 +scene0038_02 +scene0335_00 +scene0335_01 +scene0335_02 +scene0261_00 +scene0261_01 +scene0261_02 +scene0261_03 +scene0640_00 +scene0640_01 +scene0640_02 +scene0080_00 +scene0080_01 +scene0080_02 +scene0403_00 +scene0403_01 +scene0282_00 +scene0282_01 +scene0282_02 +scene0682_00 +scene0173_00 +scene0173_01 +scene0173_02 +scene0522_00 +scene0687_00 +scene0345_00 +scene0345_01 +scene0612_00 +scene0612_01 +scene0411_00 +scene0411_01 +scene0411_02 +scene0625_00 +scene0625_01 +scene0211_00 +scene0211_01 +scene0211_02 +scene0211_03 +scene0676_00 +scene0676_01 +scene0179_00 +scene0498_00 +scene0498_01 +scene0498_02 +scene0547_00 +scene0547_01 +scene0547_02 +scene0269_00 +scene0269_01 +scene0269_02 +scene0366_00 +scene0680_00 +scene0680_01 +scene0588_00 +scene0588_01 +scene0588_02 +scene0588_03 +scene0346_00 +scene0346_01 +scene0359_00 +scene0359_01 +scene0014_00 +scene0120_00 +scene0120_01 +scene0212_00 +scene0212_01 +scene0212_02 +scene0176_00 +scene0049_00 +scene0259_00 +scene0259_01 +scene0586_00 +scene0586_01 +scene0586_02 +scene0309_00 +scene0309_01 +scene0125_00 +scene0455_00 +scene0177_00 +scene0177_01 +scene0177_02 +scene0326_00 +scene0372_00 +scene0171_00 +scene0171_01 +scene0374_00 +scene0654_00 +scene0654_01 +scene0445_00 +scene0445_01 +scene0475_00 +scene0475_01 +scene0475_02 +scene0349_00 +scene0349_01 +scene0234_00 +scene0669_00 +scene0669_01 +scene0375_00 +scene0375_01 +scene0375_02 +scene0387_00 +scene0387_01 +scene0387_02 +scene0312_00 +scene0312_01 +scene0312_02 +scene0384_00 +scene0385_00 +scene0385_01 +scene0385_02 +scene0000_00 +scene0000_01 +scene0000_02 +scene0376_00 +scene0376_01 +scene0376_02 +scene0301_00 +scene0301_01 +scene0301_02 +scene0322_00 +scene0542_00 +scene0079_00 +scene0079_01 +scene0099_00 +scene0099_01 +scene0476_00 +scene0476_01 +scene0476_02 +scene0394_00 +scene0394_01 +scene0147_00 +scene0147_01 +scene0067_00 +scene0067_01 +scene0067_02 +scene0397_00 +scene0397_01 +scene0337_00 +scene0337_01 +scene0337_02 +scene0431_00 +scene0223_00 +scene0223_01 +scene0223_02 +scene0010_00 +scene0010_01 +scene0402_00 +scene0268_00 +scene0268_01 +scene0268_02 +scene0679_00 +scene0679_01 +scene0405_00 +scene0128_00 +scene0408_00 +scene0408_01 +scene0190_00 +scene0107_00 +scene0076_00 +scene0167_00 +scene0361_00 +scene0361_01 +scene0361_02 +scene0216_00 +scene0202_00 +scene0303_00 +scene0303_01 +scene0303_02 +scene0446_00 +scene0446_01 +scene0089_00 +scene0089_01 +scene0089_02 +scene0360_00 +scene0150_00 +scene0150_01 +scene0150_02 +scene0421_00 +scene0421_01 +scene0421_02 +scene0454_00 +scene0626_00 +scene0626_01 +scene0626_02 +scene0186_00 +scene0186_01 +scene0538_00 +scene0479_00 +scene0479_01 +scene0479_02 +scene0656_00 +scene0656_01 +scene0656_02 +scene0656_03 +scene0525_00 +scene0525_01 +scene0525_02 +scene0308_00 +scene0396_00 +scene0396_01 +scene0396_02 +scene0624_00 +scene0292_00 +scene0292_01 +scene0632_00 +scene0253_00 +scene0021_00 +scene0325_00 +scene0325_01 +scene0437_00 +scene0437_01 +scene0438_00 +scene0590_00 +scene0590_01 +scene0400_00 +scene0400_01 +scene0541_00 +scene0541_01 +scene0541_02 +scene0677_00 +scene0677_01 +scene0677_02 +scene0443_00 +scene0315_00 +scene0288_00 +scene0288_01 +scene0288_02 +scene0422_00 +scene0672_00 +scene0672_01 +scene0184_00 +scene0449_00 +scene0449_01 +scene0449_02 +scene0048_00 +scene0048_01 +scene0138_00 +scene0452_00 +scene0452_01 +scene0452_02 +scene0667_00 +scene0667_01 +scene0667_02 +scene0463_00 +scene0463_01 +scene0078_00 +scene0078_01 +scene0078_02 +scene0636_00 +scene0457_00 +scene0457_01 +scene0457_02 +scene0465_00 +scene0465_01 +scene0577_00 +scene0151_00 +scene0151_01 +scene0339_00 +scene0573_00 +scene0573_01 +scene0154_00 +scene0096_00 +scene0096_01 +scene0096_02 +scene0235_00 +scene0168_00 +scene0168_01 +scene0168_02 +scene0594_00 +scene0587_00 +scene0587_01 +scene0587_02 +scene0587_03 +scene0229_00 +scene0229_01 +scene0229_02 +scene0512_00 +scene0106_00 +scene0106_01 +scene0106_02 +scene0472_00 +scene0472_01 +scene0472_02 +scene0489_00 +scene0489_01 +scene0489_02 +scene0425_00 +scene0425_01 +scene0641_00 +scene0526_00 +scene0526_01 +scene0317_00 +scene0317_01 +scene0544_00 +scene0017_00 +scene0017_01 +scene0017_02 +scene0042_00 +scene0042_01 +scene0042_02 +scene0576_00 +scene0576_01 +scene0576_02 +scene0347_00 +scene0347_01 +scene0347_02 +scene0436_00 +scene0226_00 +scene0226_01 +scene0485_00 +scene0486_00 +scene0487_00 +scene0487_01 +scene0619_00 +scene0097_00 +scene0367_00 +scene0367_01 +scene0491_00 +scene0492_00 +scene0492_01 +scene0005_00 +scene0005_01 +scene0543_00 +scene0543_01 +scene0543_02 +scene0657_00 +scene0341_00 +scene0341_01 +scene0534_00 +scene0534_01 +scene0319_00 +scene0273_00 +scene0273_01 +scene0225_00 +scene0198_00 +scene0003_00 +scene0003_01 +scene0003_02 +scene0409_00 +scene0409_01 +scene0331_00 +scene0331_01 +scene0505_00 +scene0505_01 +scene0505_02 +scene0505_03 +scene0505_04 +scene0506_00 +scene0057_00 +scene0057_01 +scene0074_00 +scene0074_01 +scene0074_02 +scene0091_00 +scene0112_00 +scene0112_01 +scene0112_02 +scene0240_00 +scene0102_00 +scene0102_01 +scene0513_00 +scene0514_00 +scene0514_01 +scene0537_00 +scene0516_00 +scene0516_01 +scene0495_00 +scene0617_00 +scene0133_00 +scene0520_00 +scene0520_01 +scene0635_00 +scene0635_01 +scene0054_00 +scene0473_00 +scene0473_01 +scene0524_00 +scene0524_01 +scene0379_00 +scene0471_00 +scene0471_01 +scene0471_02 +scene0566_00 +scene0248_00 +scene0248_01 +scene0248_02 +scene0529_00 +scene0529_01 +scene0529_02 +scene0391_00 +scene0264_00 +scene0264_01 +scene0264_02 +scene0675_00 +scene0675_01 +scene0350_00 +scene0350_01 +scene0350_02 +scene0450_00 +scene0068_00 +scene0068_01 +scene0237_00 +scene0237_01 +scene0365_00 +scene0365_01 +scene0365_02 +scene0605_00 +scene0605_01 +scene0539_00 +scene0539_01 +scene0539_02 +scene0540_00 +scene0540_01 +scene0540_02 +scene0170_00 +scene0170_01 +scene0170_02 +scene0433_00 +scene0340_00 +scene0340_01 +scene0340_02 +scene0160_00 +scene0160_01 +scene0160_02 +scene0160_03 +scene0160_04 +scene0059_00 +scene0059_01 +scene0059_02 +scene0056_00 +scene0056_01 +scene0478_00 +scene0478_01 +scene0548_00 +scene0548_01 +scene0548_02 +scene0204_00 +scene0204_01 +scene0204_02 +scene0033_00 +scene0145_00 +scene0483_00 +scene0508_00 +scene0508_01 +scene0508_02 +scene0180_00 +scene0148_00 +scene0556_00 +scene0556_01 +scene0416_00 +scene0416_01 +scene0416_02 +scene0416_03 +scene0416_04 +scene0073_00 +scene0073_01 +scene0073_02 +scene0073_03 +scene0034_00 +scene0034_01 +scene0034_02 +scene0639_00 +scene0561_00 +scene0561_01 +scene0298_00 +scene0692_00 +scene0692_01 +scene0692_02 +scene0692_03 +scene0692_04 +scene0642_00 +scene0642_01 +scene0642_02 +scene0642_03 +scene0630_00 +scene0630_01 +scene0630_02 +scene0630_03 +scene0630_04 +scene0630_05 +scene0630_06 +scene0706_00 +scene0567_00 +scene0567_01 diff --git a/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_val.txt b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_val.txt new file mode 100644 index 0000000000000000000000000000000000000000..b9e7d9205321e8ca047a527466f4b7100c9c9d2c --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/meta_data/scannetv2_val.txt @@ -0,0 +1,312 @@ +scene0568_00 +scene0568_01 +scene0568_02 +scene0304_00 +scene0488_00 +scene0488_01 +scene0412_00 +scene0412_01 +scene0217_00 +scene0019_00 +scene0019_01 +scene0414_00 +scene0575_00 +scene0575_01 +scene0575_02 +scene0426_00 +scene0426_01 +scene0426_02 +scene0426_03 +scene0549_00 +scene0549_01 +scene0578_00 +scene0578_01 +scene0578_02 +scene0665_00 +scene0665_01 +scene0050_00 +scene0050_01 +scene0050_02 +scene0257_00 +scene0025_00 +scene0025_01 +scene0025_02 +scene0583_00 +scene0583_01 +scene0583_02 +scene0701_00 +scene0701_01 +scene0701_02 +scene0580_00 +scene0580_01 +scene0565_00 +scene0169_00 +scene0169_01 +scene0655_00 +scene0655_01 +scene0655_02 +scene0063_00 +scene0221_00 +scene0221_01 +scene0591_00 +scene0591_01 +scene0591_02 +scene0678_00 +scene0678_01 +scene0678_02 +scene0462_00 +scene0427_00 +scene0595_00 +scene0193_00 +scene0193_01 +scene0164_00 +scene0164_01 +scene0164_02 +scene0164_03 +scene0598_00 +scene0598_01 +scene0598_02 +scene0599_00 +scene0599_01 +scene0599_02 +scene0328_00 +scene0300_00 +scene0300_01 +scene0354_00 +scene0458_00 +scene0458_01 +scene0423_00 +scene0423_01 +scene0423_02 +scene0307_00 +scene0307_01 +scene0307_02 +scene0606_00 +scene0606_01 +scene0606_02 +scene0432_00 +scene0432_01 +scene0608_00 +scene0608_01 +scene0608_02 +scene0651_00 +scene0651_01 +scene0651_02 +scene0430_00 +scene0430_01 +scene0689_00 +scene0357_00 +scene0357_01 +scene0574_00 +scene0574_01 +scene0574_02 +scene0329_00 +scene0329_01 +scene0329_02 +scene0153_00 +scene0153_01 +scene0616_00 +scene0616_01 +scene0671_00 +scene0671_01 +scene0618_00 +scene0382_00 +scene0382_01 +scene0490_00 +scene0621_00 +scene0607_00 +scene0607_01 +scene0149_00 +scene0695_00 +scene0695_01 +scene0695_02 +scene0695_03 +scene0389_00 +scene0377_00 +scene0377_01 +scene0377_02 +scene0342_00 +scene0139_00 +scene0629_00 +scene0629_01 +scene0629_02 +scene0496_00 +scene0633_00 +scene0633_01 +scene0518_00 +scene0652_00 +scene0406_00 +scene0406_01 +scene0406_02 +scene0144_00 +scene0144_01 +scene0494_00 +scene0278_00 +scene0278_01 +scene0316_00 +scene0609_00 +scene0609_01 +scene0609_02 +scene0609_03 +scene0084_00 +scene0084_01 +scene0084_02 +scene0696_00 +scene0696_01 +scene0696_02 +scene0351_00 +scene0351_01 +scene0643_00 +scene0644_00 +scene0645_00 +scene0645_01 +scene0645_02 +scene0081_00 +scene0081_01 +scene0081_02 +scene0647_00 +scene0647_01 +scene0535_00 +scene0353_00 +scene0353_01 +scene0353_02 +scene0559_00 +scene0559_01 +scene0559_02 +scene0593_00 +scene0593_01 +scene0246_00 +scene0653_00 +scene0653_01 +scene0064_00 +scene0064_01 +scene0356_00 +scene0356_01 +scene0356_02 +scene0030_00 +scene0030_01 +scene0030_02 +scene0222_00 +scene0222_01 +scene0338_00 +scene0338_01 +scene0338_02 +scene0378_00 +scene0378_01 +scene0378_02 +scene0660_00 +scene0553_00 +scene0553_01 +scene0553_02 +scene0527_00 +scene0663_00 +scene0663_01 +scene0663_02 +scene0664_00 +scene0664_01 +scene0664_02 +scene0334_00 +scene0334_01 +scene0334_02 +scene0046_00 +scene0046_01 +scene0046_02 +scene0203_00 +scene0203_01 +scene0203_02 +scene0088_00 +scene0088_01 +scene0088_02 +scene0088_03 +scene0086_00 +scene0086_01 +scene0086_02 +scene0670_00 +scene0670_01 +scene0256_00 +scene0256_01 +scene0256_02 +scene0249_00 +scene0441_00 +scene0658_00 +scene0704_00 +scene0704_01 +scene0187_00 +scene0187_01 +scene0131_00 +scene0131_01 +scene0131_02 +scene0207_00 +scene0207_01 +scene0207_02 +scene0461_00 +scene0011_00 +scene0011_01 +scene0343_00 +scene0251_00 +scene0077_00 +scene0077_01 +scene0684_00 +scene0684_01 +scene0550_00 +scene0686_00 +scene0686_01 +scene0686_02 +scene0208_00 +scene0500_00 +scene0500_01 +scene0552_00 +scene0552_01 +scene0648_00 +scene0648_01 +scene0435_00 +scene0435_01 +scene0435_02 +scene0435_03 +scene0690_00 +scene0690_01 +scene0693_00 +scene0693_01 +scene0693_02 +scene0700_00 +scene0700_01 +scene0700_02 +scene0699_00 +scene0231_00 +scene0231_01 +scene0231_02 +scene0697_00 +scene0697_01 +scene0697_02 +scene0697_03 +scene0474_00 +scene0474_01 +scene0474_02 +scene0474_03 +scene0474_04 +scene0474_05 +scene0355_00 +scene0355_01 +scene0146_00 +scene0146_01 +scene0146_02 +scene0196_00 +scene0702_00 +scene0702_01 +scene0702_02 +scene0314_00 +scene0277_00 +scene0277_01 +scene0277_02 +scene0095_00 +scene0095_01 +scene0015_00 +scene0100_00 +scene0100_01 +scene0100_02 +scene0558_00 +scene0558_01 +scene0558_02 +scene0685_00 +scene0685_01 +scene0685_02 diff --git a/pointcept/datasets/preprocessing/scannet/preprocess_scannet.py b/pointcept/datasets/preprocessing/scannet/preprocess_scannet.py new file mode 100644 index 0000000000000000000000000000000000000000..549a4261080b0dfb47f31ed390f821446d35322e --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/preprocess_scannet.py @@ -0,0 +1,253 @@ +""" +Preprocessing Script for ScanNet 20/200 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import warnings + +warnings.filterwarnings("ignore", category=DeprecationWarning) + +import os +import argparse +import glob +import json +import plyfile +import numpy as np +import pandas as pd +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat +from pathlib import Path + +# Load external constants +from meta_data.scannet200_constants import VALID_CLASS_IDS_200, VALID_CLASS_IDS_20 + +CLOUD_FILE_PFIX = "_vh_clean_2" +SEGMENTS_FILE_PFIX = ".0.010000.segs.json" +AGGREGATIONS_FILE_PFIX = ".aggregation.json" +CLASS_IDS200 = VALID_CLASS_IDS_200 +CLASS_IDS20 = VALID_CLASS_IDS_20 +IGNORE_INDEX = -1 + + +def read_plymesh(filepath): + """Read ply file and return it as numpy array. Returns None if emtpy.""" + with open(filepath, "rb") as f: + plydata = plyfile.PlyData.read(f) + if plydata.elements: + vertices = pd.DataFrame(plydata["vertex"].data).values + faces = np.stack(plydata["face"].data["vertex_indices"], axis=0) + return vertices, faces + + +# Map the raw category id to the point cloud +def point_indices_from_group(seg_indices, group, labels_pd): + group_segments = np.array(group["segments"]) + label = group["label"] + + # Map the category name to id + label_id20 = labels_pd[labels_pd["raw_category"] == label]["nyu40id"] + label_id20 = int(label_id20.iloc[0]) if len(label_id20) > 0 else 0 + label_id200 = labels_pd[labels_pd["raw_category"] == label]["id"] + label_id200 = int(label_id200.iloc[0]) if len(label_id200) > 0 else 0 + + # Only store for the valid categories + if label_id20 in CLASS_IDS20: + label_id20 = CLASS_IDS20.index(label_id20) + else: + label_id20 = IGNORE_INDEX + + if label_id200 in CLASS_IDS200: + label_id200 = CLASS_IDS200.index(label_id200) + else: + label_id200 = IGNORE_INDEX + + # get points, where segment indices (points labelled with segment ids) are in the group segment list + point_idx = np.where(np.isin(seg_indices, group_segments))[0] + return point_idx, label_id20, label_id200 + + +def face_normal(vertex, face): + v01 = vertex[face[:, 1]] - vertex[face[:, 0]] + v02 = vertex[face[:, 2]] - vertex[face[:, 0]] + vec = np.cross(v01, v02) + length = np.sqrt(np.sum(vec**2, axis=1, keepdims=True)) + 1.0e-8 + nf = vec / length + area = length * 0.5 + return nf, area + + +def vertex_normal(vertex, face): + nf, area = face_normal(vertex, face) + nf = nf * area + + nv = np.zeros_like(vertex) + for i in range(face.shape[0]): + nv[face[i]] += nf[i] + + length = np.sqrt(np.sum(nv**2, axis=1, keepdims=True)) + 1.0e-8 + nv = nv / length + return nv + + +def handle_process( + scene_path, output_path, labels_pd, train_scenes, val_scenes, parse_normals=True +): + scene_id = os.path.basename(scene_path) + mesh_path = os.path.join(scene_path, f"{scene_id}{CLOUD_FILE_PFIX}.ply") + segments_file = os.path.join( + scene_path, f"{scene_id}{CLOUD_FILE_PFIX}{SEGMENTS_FILE_PFIX}" + ) + aggregations_file = os.path.join(scene_path, f"{scene_id}{AGGREGATIONS_FILE_PFIX}") + info_file = os.path.join(scene_path, f"{scene_id}.txt") + + if scene_id in train_scenes: + output_path = os.path.join(output_path, "train", f"{scene_id}") + split_name = "train" + elif scene_id in val_scenes: + output_path = os.path.join(output_path, "val", f"{scene_id}") + split_name = "val" + else: + output_path = os.path.join(output_path, "test", f"{scene_id}") + split_name = "test" + + print(f"Processing: {scene_id} in {split_name}") + + vertices, faces = read_plymesh(mesh_path) + coords = vertices[:, :3] + colors = vertices[:, 3:6] + save_dict = dict( + coord=coords.astype(np.float32), + color=colors.astype(np.uint8), + ) + + # # Rotating the mesh to axis aligned + # info_dict = {} + # with open(info_file) as f: + # for line in f: + # (key, val) = line.split(" = ") + # info_dict[key] = np.fromstring(val, sep=' ') + # + # if 'axisAlignment' not in info_dict: + # rot_matrix = np.identity(4) + # else: + # rot_matrix = info_dict['axisAlignment'].reshape(4, 4) + # r_coords = coords.transpose() + # r_coords = np.append(r_coords, np.ones((1, r_coords.shape[1])), axis=0) + # r_coords = np.dot(rot_matrix, r_coords) + # coords = r_coords + + # Parse Normals + if parse_normals: + save_dict["normal"] = vertex_normal(coords, faces).astype(np.float32) + + # Load segments file + if split_name != "test": + with open(segments_file) as f: + segments = json.load(f) + seg_indices = np.array(segments["segIndices"]) + + # Load Aggregations file + with open(aggregations_file) as f: + aggregation = json.load(f) + seg_groups = np.array(aggregation["segGroups"]) + + # Generate new labels + semantic_gt20 = np.ones((vertices.shape[0]), dtype=np.int16) * IGNORE_INDEX + semantic_gt200 = np.ones((vertices.shape[0]), dtype=np.int16) * IGNORE_INDEX + instance_ids = np.ones((vertices.shape[0]), dtype=np.int16) * IGNORE_INDEX + for group in seg_groups: + point_idx, label_id20, label_id200 = point_indices_from_group( + seg_indices, group, labels_pd + ) + + semantic_gt20[point_idx] = label_id20 + semantic_gt200[point_idx] = label_id200 + instance_ids[point_idx] = group["id"] + + semantic_gt20 = semantic_gt20.astype(int) + semantic_gt200 = semantic_gt200.astype(int) + instance_ids = instance_ids.astype(int) + + save_dict["segment20"] = semantic_gt20 + save_dict["segment200"] = semantic_gt200 + save_dict["instance"] = instance_ids + + # Concatenate with original cloud + processed_vertices = np.hstack((semantic_gt200, instance_ids)) + + if np.any(np.isnan(processed_vertices)) or not np.all( + np.isfinite(processed_vertices) + ): + raise ValueError(f"Find NaN in Scene: {scene_id}") + + # Save processed data + os.makedirs(output_path, exist_ok=True) + for key in save_dict.keys(): + np.save(os.path.join(output_path, f"{key}.npy"), save_dict[key]) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--parse_normals", default=True, type=bool, help="Whether parse point normals" + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + config = parser.parse_args() + meta_root = Path(os.path.dirname(__file__)) / "meta_data" + + # Load label map + labels_pd = pd.read_csv( + meta_root / "scannetv2-labels.combined.tsv", + sep="\t", + header=0, + ) + + # Load train/val splits + with open(meta_root / "scannetv2_train.txt") as train_file: + train_scenes = train_file.read().splitlines() + with open(meta_root / "scannetv2_val.txt") as val_file: + val_scenes = val_file.read().splitlines() + + # Create output directories + train_output_dir = os.path.join(config.output_root, "train") + os.makedirs(train_output_dir, exist_ok=True) + val_output_dir = os.path.join(config.output_root, "val") + os.makedirs(val_output_dir, exist_ok=True) + test_output_dir = os.path.join(config.output_root, "test") + os.makedirs(test_output_dir, exist_ok=True) + + # Load scene paths + scene_paths = sorted(glob.glob(config.dataset_root + "/scans*/scene*")) + + # Preprocess data. + print("Processing scenes...") + pool = ProcessPoolExecutor(max_workers=config.num_workers) + _ = list( + pool.map( + handle_process, + scene_paths, + repeat(config.output_root), + repeat(labels_pd), + repeat(train_scenes), + repeat(val_scenes), + repeat(config.parse_normals), + ) + ) diff --git a/pointcept/datasets/preprocessing/scannet/scannet_pair/SensorData.py b/pointcept/datasets/preprocessing/scannet/scannet_pair/SensorData.py new file mode 100644 index 0000000000000000000000000000000000000000..d90c8770e812f782e4735cc7095c100cd6258bf6 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/scannet_pair/SensorData.py @@ -0,0 +1,183 @@ +import os, struct +import numpy as np +import zlib +import imageio +import cv2 + +COMPRESSION_TYPE_COLOR = {-1: "unknown", 0: "raw", 1: "png", 2: "jpeg"} +COMPRESSION_TYPE_DEPTH = { + -1: "unknown", + 0: "raw_ushort", + 1: "zlib_ushort", + 2: "occi_ushort", +} + + +class RGBDFrame: + def load(self, file_handle): + self.camera_to_world = np.asarray( + struct.unpack("f" * 16, file_handle.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.timestamp_color = struct.unpack("Q", file_handle.read(8))[0] + self.timestamp_depth = struct.unpack("Q", file_handle.read(8))[0] + self.color_size_bytes = struct.unpack("Q", file_handle.read(8))[0] + self.depth_size_bytes = struct.unpack("Q", file_handle.read(8))[0] + self.color_data = b"".join( + struct.unpack( + "c" * self.color_size_bytes, file_handle.read(self.color_size_bytes) + ) + ) + self.depth_data = b"".join( + struct.unpack( + "c" * self.depth_size_bytes, file_handle.read(self.depth_size_bytes) + ) + ) + + def decompress_depth(self, compression_type): + if compression_type == "zlib_ushort": + return self.decompress_depth_zlib() + else: + raise + + def decompress_depth_zlib(self): + return zlib.decompress(self.depth_data) + + def decompress_color(self, compression_type): + if compression_type == "jpeg": + return self.decompress_color_jpeg() + else: + raise + + def decompress_color_jpeg(self): + return imageio.imread(self.color_data) + + +class SensorData: + def __init__(self, filename): + self.version = 4 + self.load(filename) + + def load(self, filename): + with open(filename, "rb") as f: + version = struct.unpack("I", f.read(4))[0] + assert self.version == version + strlen = struct.unpack("Q", f.read(8))[0] + self.sensor_name = b"".join(struct.unpack("c" * strlen, f.read(strlen))) + self.intrinsic_color = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.extrinsic_color = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.intrinsic_depth = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.extrinsic_depth = np.asarray( + struct.unpack("f" * 16, f.read(16 * 4)), dtype=np.float32 + ).reshape(4, 4) + self.color_compression_type = COMPRESSION_TYPE_COLOR[ + struct.unpack("i", f.read(4))[0] + ] + self.depth_compression_type = COMPRESSION_TYPE_DEPTH[ + struct.unpack("i", f.read(4))[0] + ] + self.color_width = struct.unpack("I", f.read(4))[0] + self.color_height = struct.unpack("I", f.read(4))[0] + self.depth_width = struct.unpack("I", f.read(4))[0] + self.depth_height = struct.unpack("I", f.read(4))[0] + self.depth_shift = struct.unpack("f", f.read(4))[0] + num_frames = struct.unpack("Q", f.read(8))[0] + self.frames = [] + for i in range(num_frames): + frame = RGBDFrame() + frame.load(f) + self.frames.append(frame) + + def export_depth_images(self, output_path, image_size=None, frame_skip=1): + if not os.path.exists(output_path): + os.makedirs(output_path) + print( + "exporting", len(self.frames) // frame_skip, " depth frames to", output_path + ) + for f in range(0, len(self.frames), frame_skip): + if os.path.exists((os.path.join(output_path, str(f) + ".png"))): + continue + if f % 100 == 0: + print( + "exporting", + f, + "th depth frames to", + os.path.join(output_path, str(f) + ".png"), + ) + + depth_data = self.frames[f].decompress_depth(self.depth_compression_type) + depth = np.fromstring(depth_data, dtype=np.uint16).reshape( + self.depth_height, self.depth_width + ) + if image_size is not None: + depth = cv2.resize( + depth, + (image_size[1], image_size[0]), + interpolation=cv2.INTER_NEAREST, + ) + imageio.imwrite(os.path.join(output_path, str(f) + ".png"), depth) + + def export_color_images(self, output_path, image_size=None, frame_skip=1): + if not os.path.exists(output_path): + os.makedirs(output_path) + print( + "exporting", len(self.frames) // frame_skip, "color frames to", output_path + ) + for f in range(0, len(self.frames), frame_skip): + if os.path.exists((os.path.join(output_path, str(f) + ".png"))): + continue + if f % 100 == 0: + print( + "exporting", + f, + "th color frames to", + os.path.join(output_path, str(f) + ".png"), + ) + color = self.frames[f].decompress_color(self.color_compression_type) + if image_size is not None: + color = cv2.resize( + color, + (image_size[1], image_size[0]), + interpolation=cv2.INTER_NEAREST, + ) + # imageio.imwrite(os.path.join(output_path, str(f) + '.jpg'), color) + imageio.imwrite(os.path.join(output_path, str(f) + ".png"), color) + + def save_mat_to_file(self, matrix, filename): + with open(filename, "w") as f: + for line in matrix: + np.savetxt(f, line[np.newaxis], fmt="%f") + + def export_poses(self, output_path, frame_skip=1): + if not os.path.exists(output_path): + os.makedirs(output_path) + print( + "exporting", len(self.frames) // frame_skip, "camera poses to", output_path + ) + for f in range(0, len(self.frames), frame_skip): + self.save_mat_to_file( + self.frames[f].camera_to_world, + os.path.join(output_path, str(f) + ".txt"), + ) + + def export_intrinsics(self, output_path): + if not os.path.exists(output_path): + os.makedirs(output_path) + print("exporting camera intrinsics to", output_path) + self.save_mat_to_file( + self.intrinsic_color, os.path.join(output_path, "intrinsic_color.txt") + ) + self.save_mat_to_file( + self.extrinsic_color, os.path.join(output_path, "extrinsic_color.txt") + ) + self.save_mat_to_file( + self.intrinsic_depth, os.path.join(output_path, "intrinsic_depth.txt") + ) + self.save_mat_to_file( + self.extrinsic_depth, os.path.join(output_path, "extrinsic_depth.txt") + ) diff --git a/pointcept/datasets/preprocessing/scannet/scannet_pair/compute_full_overlapping.py b/pointcept/datasets/preprocessing/scannet/scannet_pair/compute_full_overlapping.py new file mode 100644 index 0000000000000000000000000000000000000000..a6b407eebad280f2817805d15ec43b9f7f6afbf4 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/scannet_pair/compute_full_overlapping.py @@ -0,0 +1,91 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import copy +import torch +import numpy as np +import math +import glob, os +import argparse +import open3d as o3d + + +def make_open3d_point_cloud(xyz, color=None, voxel_size=None): + if np.isnan(xyz).any(): + return None + + xyz = xyz[:, :3] + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(xyz) + if color is not None: + pcd.colors = o3d.utility.Vector3dVector(color) + if voxel_size is not None: + pcd = pcd.voxel_down_sample(voxel_size) + + return pcd + + +def compute_overlap_ratio(pcd0, pcd1, voxel_size): + pcd0_down = pcd0.voxel_down_sample(voxel_size) + pcd1_down = pcd1.voxel_down_sample(voxel_size) + matching01 = get_matching_indices(pcd0_down, pcd1_down, voxel_size * 1.5, 1) + matching10 = get_matching_indices(pcd1_down, pcd0_down, voxel_size * 1.5, 1) + overlap0 = float(len(matching01)) / float(len(pcd0_down.points)) + overlap1 = float(len(matching10)) / float(len(pcd1_down.points)) + return max(overlap0, overlap1) + + +def get_matching_indices(source, pcd_tree, search_voxel_size, K=None): + match_inds = [] + for i, point in enumerate(source.points): + [_, idx, _] = pcd_tree.search_radius_vector_3d(point, search_voxel_size) + if K is not None: + idx = idx[:K] + for j in idx: + match_inds.append((i, j)) + return match_inds + + +def compute_full_overlapping(data_root, scene_id, voxel_size=0.05): + _points = [ + ( + pcd_name, + make_open3d_point_cloud( + torch.load(pcd_name)["coord"], voxel_size=voxel_size + ), + ) + for pcd_name in glob.glob(os.path.join(data_root, scene_id, "pcd", "*.pth")) + ] + points = [(pcd_name, pcd) for (pcd_name, pcd) in _points if pcd is not None] + print( + "load {} point clouds ({} invalid has been filtered), computing matching/overlapping".format( + len(points), len(_points) - len(points) + ) + ) + + matching_matrix = np.zeros((len(points), len(points))) + for i, (pcd0_name, pcd0) in enumerate(points): + print("matching to...{}".format(pcd0_name)) + pcd0_tree = o3d.geometry.KDTreeFlann(copy.deepcopy(pcd0)) + for j, (pcd1_name, pcd1) in enumerate(points): + if i == j: + continue + matching_matrix[i, j] = float( + len(get_matching_indices(pcd1, pcd0_tree, 1.5 * voxel_size, 1)) + ) / float(len(pcd1.points)) + + # write to file + with open(os.path.join(data_root, scene_id, "pcd", "overlap.txt"), "w") as f: + for i, (pcd0_name, pcd0) in enumerate(points): + for j, (pcd1_name, pcd1) in enumerate(points): + if i < j: + overlap = max(matching_matrix[i, j], matching_matrix[j, i]) + f.write( + "{} {} {}\n".format( + pcd0_name.replace(data_root, ""), + pcd1_name.replace(data_root, ""), + overlap, + ) + ) diff --git a/pointcept/datasets/preprocessing/scannet/scannet_pair/generage_list.py b/pointcept/datasets/preprocessing/scannet/scannet_pair/generage_list.py new file mode 100644 index 0000000000000000000000000000000000000000..a8943ba040cc24fc8d3130bd8784052cb57ce6c9 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/scannet_pair/generage_list.py @@ -0,0 +1,33 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + + +import argparse +import glob, os, sys + +from SensorData import SensorData + +# params +parser = argparse.ArgumentParser() +# data paths +parser.add_argument("--target_dir", required=True, help="path to the target dir") + +opt = parser.parse_args() +print(opt) + + +def main(): + overlaps = glob.glob(os.path.join(opt.target_dir, "*/pcd/overlap.txt")) + with open(os.path.join(opt.target_dir, "overlap30.txt"), "w") as f: + for fo in overlaps: + for line in open(fo): + pcd0, pcd1, op = line.strip().split() + if float(op) >= 0.3: + print("{} {} {}".format(pcd0, pcd1, op), file=f) + print("done") + + +if __name__ == "__main__": + main() diff --git a/pointcept/datasets/preprocessing/scannet/scannet_pair/plyfile.py b/pointcept/datasets/preprocessing/scannet/scannet_pair/plyfile.py new file mode 100644 index 0000000000000000000000000000000000000000..3f0930b05b59069638d4c8315683fbea320d2a2f --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/scannet_pair/plyfile.py @@ -0,0 +1,893 @@ +# Copyright 2014 Darsh Ranjan +# +# This file is part of python-plyfile. +# +# python-plyfile is free software: you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# python-plyfile is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with python-plyfile. If not, see +# . + +from itertools import islice as _islice + +import numpy as _np +from sys import byteorder as _byteorder + +try: + _range = xrange +except NameError: + _range = range + + +# Many-many relation +_data_type_relation = [ + ("int8", "i1"), + ("char", "i1"), + ("uint8", "u1"), + ("uchar", "b1"), + ("uchar", "u1"), + ("int16", "i2"), + ("short", "i2"), + ("uint16", "u2"), + ("ushort", "u2"), + ("int32", "i4"), + ("int", "i4"), + ("uint32", "u4"), + ("uint", "u4"), + ("float32", "f4"), + ("float", "f4"), + ("float64", "f8"), + ("double", "f8"), +] + +_data_types = dict(_data_type_relation) +_data_type_reverse = dict((b, a) for (a, b) in _data_type_relation) + +_types_list = [] +_types_set = set() +for _a, _b in _data_type_relation: + if _a not in _types_set: + _types_list.append(_a) + _types_set.add(_a) + if _b not in _types_set: + _types_list.append(_b) + _types_set.add(_b) + + +_byte_order_map = {"ascii": "=", "binary_little_endian": "<", "binary_big_endian": ">"} + +_byte_order_reverse = {"<": "binary_little_endian", ">": "binary_big_endian"} + +_native_byte_order = {"little": "<", "big": ">"}[_byteorder] + + +def _lookup_type(type_str): + if type_str not in _data_type_reverse: + try: + type_str = _data_types[type_str] + except KeyError: + raise ValueError("field type %r not in %r" % (type_str, _types_list)) + + return _data_type_reverse[type_str] + + +def _split_line(line, n): + fields = line.split(None, n) + if len(fields) == n: + fields.append("") + + assert len(fields) == n + 1 + + return fields + + +def make2d(array, cols=None, dtype=None): + """ + Make a 2D array from an array of arrays. The `cols' and `dtype' + arguments can be omitted if the array is not empty. + + """ + if (cols is None or dtype is None) and not len(array): + raise RuntimeError("cols and dtype must be specified for empty " "array") + + if cols is None: + cols = len(array[0]) + + if dtype is None: + dtype = array[0].dtype + + return _np.fromiter(array, [("_", dtype, (cols,))], count=len(array))["_"] + + +class PlyParseError(Exception): + """ + Raised when a PLY file cannot be parsed. + + The attributes `element', `row', `property', and `message' give + additional information. + + """ + + def __init__(self, message, element=None, row=None, prop=None): + self.message = message + self.element = element + self.row = row + self.prop = prop + + s = "" + if self.element: + s += "element %r: " % self.element.name + if self.row is not None: + s += "row %d: " % self.row + if self.prop: + s += "property %r: " % self.prop.name + s += self.message + + Exception.__init__(self, s) + + def __repr__(self): + return ( + "PlyParseError(%r, element=%r, row=%r, prop=%r)" % self.message, + self.element, + self.row, + self.prop, + ) + + +class PlyData(object): + """ + PLY file header and data. + + A PlyData instance is created in one of two ways: by the static + method PlyData.read (to read a PLY file), or directly from __init__ + given a sequence of elements (which can then be written to a PLY + file). + + """ + + def __init__( + self, elements=[], text=False, byte_order="=", comments=[], obj_info=[] + ): + """ + elements: sequence of PlyElement instances. + + text: whether the resulting PLY file will be text (True) or + binary (False). + + byte_order: '<' for little-endian, '>' for big-endian, or '=' + for native. This is only relevant if `text' is False. + + comments: sequence of strings that will be placed in the header + between the 'ply' and 'format ...' lines. + + obj_info: like comments, but will be placed in the header with + "obj_info ..." instead of "comment ...". + + """ + if byte_order == "=" and not text: + byte_order = _native_byte_order + + self.byte_order = byte_order + self.text = text + + self.comments = list(comments) + self.obj_info = list(obj_info) + self.elements = elements + + def _get_elements(self): + return self._elements + + def _set_elements(self, elements): + self._elements = tuple(elements) + self._index() + + elements = property(_get_elements, _set_elements) + + def _get_byte_order(self): + return self._byte_order + + def _set_byte_order(self, byte_order): + if byte_order not in ["<", ">", "="]: + raise ValueError("byte order must be '<', '>', or '='") + + self._byte_order = byte_order + + byte_order = property(_get_byte_order, _set_byte_order) + + def _index(self): + self._element_lookup = dict((elt.name, elt) for elt in self._elements) + if len(self._element_lookup) != len(self._elements): + raise ValueError("two elements with same name") + + @staticmethod + def _parse_header(stream): + """ + Parse a PLY header from a readable file-like stream. + + """ + lines = [] + comments = {"comment": [], "obj_info": []} + while True: + line = stream.readline().decode("ascii").strip() + fields = _split_line(line, 1) + + if fields[0] == "end_header": + break + + elif fields[0] in comments.keys(): + lines.append(fields) + else: + lines.append(line.split()) + + a = 0 + if lines[a] != ["ply"]: + raise PlyParseError("expected 'ply'") + + a += 1 + while lines[a][0] in comments.keys(): + comments[lines[a][0]].append(lines[a][1]) + a += 1 + + if lines[a][0] != "format": + raise PlyParseError("expected 'format'") + + if lines[a][2] != "1.0": + raise PlyParseError("expected version '1.0'") + + if len(lines[a]) != 3: + raise PlyParseError("too many fields after 'format'") + + fmt = lines[a][1] + + if fmt not in _byte_order_map: + raise PlyParseError("don't understand format %r" % fmt) + + byte_order = _byte_order_map[fmt] + text = fmt == "ascii" + + a += 1 + while a < len(lines) and lines[a][0] in comments.keys(): + comments[lines[a][0]].append(lines[a][1]) + a += 1 + + return PlyData( + PlyElement._parse_multi(lines[a:]), + text, + byte_order, + comments["comment"], + comments["obj_info"], + ) + + @staticmethod + def read(stream): + """ + Read PLY data from a readable file-like object or filename. + + """ + must_close, stream = _open_stream(stream, "read") + try: + data = PlyData._parse_header(stream) + for elt in data: + elt._read(stream, data.text, data.byte_order) + finally: + if must_close: + stream.close() + + return data + + def write(self, stream): + """ + Write PLY data to a writeable file-like object or filename. + + """ + must_close, stream = _open_stream(stream, "write") + try: + stream.write(self.header.encode("ascii")) + stream.write(b"\r\n") + for elt in self: + elt._write(stream, self.text, self.byte_order) + finally: + if must_close: + stream.close() + + @property + def header(self): + """ + Provide PLY-formatted metadata for the instance. + + """ + lines = ["ply"] + + if self.text: + lines.append("format ascii 1.0") + else: + lines.append("format " + _byte_order_reverse[self.byte_order] + " 1.0") + + # Some information is lost here, since all comments are placed + # between the 'format' line and the first element. + for c in self.comments: + lines.append("comment " + c) + + for c in self.obj_info: + lines.append("obj_info " + c) + + lines.extend(elt.header for elt in self.elements) + lines.append("end_header") + return "\r\n".join(lines) + + def __iter__(self): + return iter(self.elements) + + def __len__(self): + return len(self.elements) + + def __contains__(self, name): + return name in self._element_lookup + + def __getitem__(self, name): + return self._element_lookup[name] + + def __str__(self): + return self.header + + def __repr__(self): + return "PlyData(%r, text=%r, byte_order=%r, " "comments=%r, obj_info=%r)" % ( + self.elements, + self.text, + self.byte_order, + self.comments, + self.obj_info, + ) + + +def _open_stream(stream, read_or_write): + if hasattr(stream, read_or_write): + return (False, stream) + try: + return (True, open(stream, read_or_write[0] + "b")) + except TypeError: + raise RuntimeError("expected open file or filename") + + +class PlyElement(object): + """ + PLY file element. + + A client of this library doesn't normally need to instantiate this + directly, so the following is only for the sake of documenting the + internals. + + Creating a PlyElement instance is generally done in one of two ways: + as a byproduct of PlyData.read (when reading a PLY file) and by + PlyElement.describe (before writing a PLY file). + + """ + + def __init__(self, name, properties, count, comments=[]): + """ + This is not part of the public interface. The preferred methods + of obtaining PlyElement instances are PlyData.read (to read from + a file) and PlyElement.describe (to construct from a numpy + array). + + """ + self._name = str(name) + self._check_name() + self._count = count + + self._properties = tuple(properties) + self._index() + + self.comments = list(comments) + + self._have_list = any(isinstance(p, PlyListProperty) for p in self.properties) + + @property + def count(self): + return self._count + + def _get_data(self): + return self._data + + def _set_data(self, data): + self._data = data + self._count = len(data) + self._check_sanity() + + data = property(_get_data, _set_data) + + def _check_sanity(self): + for prop in self.properties: + if prop.name not in self._data.dtype.fields: + raise ValueError("dangling property %r" % prop.name) + + def _get_properties(self): + return self._properties + + def _set_properties(self, properties): + self._properties = tuple(properties) + self._check_sanity() + self._index() + + properties = property(_get_properties, _set_properties) + + def _index(self): + self._property_lookup = dict((prop.name, prop) for prop in self._properties) + if len(self._property_lookup) != len(self._properties): + raise ValueError("two properties with same name") + + def ply_property(self, name): + return self._property_lookup[name] + + @property + def name(self): + return self._name + + def _check_name(self): + if any(c.isspace() for c in self._name): + msg = "element name %r contains spaces" % self._name + raise ValueError(msg) + + def dtype(self, byte_order="="): + """ + Return the numpy dtype of the in-memory representation of the + data. (If there are no list properties, and the PLY format is + binary, then this also accurately describes the on-disk + representation of the element.) + + """ + return [(prop.name, prop.dtype(byte_order)) for prop in self.properties] + + @staticmethod + def _parse_multi(header_lines): + """ + Parse a list of PLY element definitions. + + """ + elements = [] + while header_lines: + elt, header_lines = PlyElement._parse_one(header_lines) + elements.append(elt) + + return elements + + @staticmethod + def _parse_one(lines): + """ + Consume one element definition. The unconsumed input is + returned along with a PlyElement instance. + + """ + a = 0 + line = lines[a] + + if line[0] != "element": + raise PlyParseError("expected 'element'") + if len(line) > 3: + raise PlyParseError("too many fields after 'element'") + if len(line) < 3: + raise PlyParseError("too few fields after 'element'") + + name, count = (line[1], int(line[2])) + + comments = [] + properties = [] + while True: + a += 1 + if a >= len(lines): + break + + if lines[a][0] == "comment": + comments.append(lines[a][1]) + elif lines[a][0] == "property": + properties.append(PlyProperty._parse_one(lines[a])) + else: + break + + return (PlyElement(name, properties, count, comments), lines[a:]) + + @staticmethod + def describe(data, name, len_types={}, val_types={}, comments=[]): + """ + Construct a PlyElement from an array's metadata. + + len_types and val_types can be given as mappings from list + property names to type strings (like 'u1', 'f4', etc., or + 'int8', 'float32', etc.). These can be used to define the length + and value types of list properties. List property lengths + always default to type 'u1' (8-bit unsigned integer), and value + types default to 'i4' (32-bit integer). + + """ + if not isinstance(data, _np.ndarray): + raise TypeError("only numpy arrays are supported") + + if len(data.shape) != 1: + raise ValueError("only one-dimensional arrays are " "supported") + + count = len(data) + + properties = [] + descr = data.dtype.descr + + for t in descr: + if not isinstance(t[1], str): + raise ValueError("nested records not supported") + + if not t[0]: + raise ValueError("field with empty name") + + if len(t) != 2 or t[1][1] == "O": + # non-scalar field, which corresponds to a list + # property in PLY. + + if t[1][1] == "O": + if len(t) != 2: + raise ValueError("non-scalar object fields not " "supported") + + len_str = _data_type_reverse[len_types.get(t[0], "u1")] + if t[1][1] == "O": + val_type = val_types.get(t[0], "i4") + val_str = _lookup_type(val_type) + else: + val_str = _lookup_type(t[1][1:]) + + prop = PlyListProperty(t[0], len_str, val_str) + else: + val_str = _lookup_type(t[1][1:]) + prop = PlyProperty(t[0], val_str) + + properties.append(prop) + + elt = PlyElement(name, properties, count, comments) + elt.data = data + + return elt + + def _read(self, stream, text, byte_order): + """ + Read the actual data from a PLY file. + + """ + if text: + self._read_txt(stream) + else: + if self._have_list: + # There are list properties, so a simple load is + # impossible. + self._read_bin(stream, byte_order) + else: + # There are no list properties, so loading the data is + # much more straightforward. + self._data = _np.fromfile(stream, self.dtype(byte_order), self.count) + + if len(self._data) < self.count: + k = len(self._data) + del self._data + raise PlyParseError("early end-of-file", self, k) + + self._check_sanity() + + def _write(self, stream, text, byte_order): + """ + Write the data to a PLY file. + + """ + if text: + self._write_txt(stream) + else: + if self._have_list: + # There are list properties, so serialization is + # slightly complicated. + self._write_bin(stream, byte_order) + else: + # no list properties, so serialization is + # straightforward. + self.data.astype(self.dtype(byte_order), copy=False).tofile(stream) + + def _read_txt(self, stream): + """ + Load a PLY element from an ASCII-format PLY file. The element + may contain list properties. + + """ + self._data = _np.empty(self.count, dtype=self.dtype()) + + k = 0 + for line in _islice(iter(stream.readline, b""), self.count): + fields = iter(line.strip().split()) + for prop in self.properties: + try: + self._data[prop.name][k] = prop._from_fields(fields) + except StopIteration: + raise PlyParseError("early end-of-line", self, k, prop) + except ValueError: + raise PlyParseError("malformed input", self, k, prop) + try: + next(fields) + except StopIteration: + pass + else: + raise PlyParseError("expected end-of-line", self, k) + k += 1 + + if k < self.count: + del self._data + raise PlyParseError("early end-of-file", self, k) + + def _write_txt(self, stream): + """ + Save a PLY element to an ASCII-format PLY file. The element may + contain list properties. + + """ + for rec in self.data: + fields = [] + for prop in self.properties: + fields.extend(prop._to_fields(rec[prop.name])) + + _np.savetxt(stream, [fields], "%.18g", newline="\r\n") + + def _read_bin(self, stream, byte_order): + """ + Load a PLY element from a binary PLY file. The element may + contain list properties. + + """ + self._data = _np.empty(self.count, dtype=self.dtype(byte_order)) + + for k in _range(self.count): + for prop in self.properties: + try: + self._data[prop.name][k] = prop._read_bin(stream, byte_order) + except StopIteration: + raise PlyParseError("early end-of-file", self, k, prop) + + def _write_bin(self, stream, byte_order): + """ + Save a PLY element to a binary PLY file. The element may + contain list properties. + + """ + for rec in self.data: + for prop in self.properties: + prop._write_bin(rec[prop.name], stream, byte_order) + + @property + def header(self): + """ + Format this element's metadata as it would appear in a PLY + header. + + """ + lines = ["element %s %d" % (self.name, self.count)] + + # Some information is lost here, since all comments are placed + # between the 'element' line and the first property definition. + for c in self.comments: + lines.append("comment " + c) + + lines.extend(list(map(str, self.properties))) + + return "\r\n".join(lines) + + def __getitem__(self, key): + return self.data[key] + + def __setitem__(self, key, value): + self.data[key] = value + + def __str__(self): + return self.header + + def __repr__(self): + return "PlyElement(%r, %r, count=%d, comments=%r)" % ( + self.name, + self.properties, + self.count, + self.comments, + ) + + +class PlyProperty(object): + """ + PLY property description. This class is pure metadata; the data + itself is contained in PlyElement instances. + + """ + + def __init__(self, name, val_dtype): + self._name = str(name) + self._check_name() + self.val_dtype = val_dtype + + def _get_val_dtype(self): + return self._val_dtype + + def _set_val_dtype(self, val_dtype): + self._val_dtype = _data_types[_lookup_type(val_dtype)] + + val_dtype = property(_get_val_dtype, _set_val_dtype) + + @property + def name(self): + return self._name + + def _check_name(self): + if any(c.isspace() for c in self._name): + msg = "Error: property name %r contains spaces" % self._name + raise RuntimeError(msg) + + @staticmethod + def _parse_one(line): + assert line[0] == "property" + + if line[1] == "list": + if len(line) > 5: + raise PlyParseError("too many fields after " "'property list'") + if len(line) < 5: + raise PlyParseError("too few fields after " "'property list'") + + return PlyListProperty(line[4], line[2], line[3]) + + else: + if len(line) > 3: + raise PlyParseError("too many fields after " "'property'") + if len(line) < 3: + raise PlyParseError("too few fields after " "'property'") + + return PlyProperty(line[2], line[1]) + + def dtype(self, byte_order="="): + """ + Return the numpy dtype description for this property (as a tuple + of strings). + + """ + return byte_order + self.val_dtype + + def _from_fields(self, fields): + """ + Parse from generator. Raise StopIteration if the property could + not be read. + + """ + return _np.dtype(self.dtype()).type(next(fields)) + + def _to_fields(self, data): + """ + Return generator over one item. + + """ + yield _np.dtype(self.dtype()).type(data) + + def _read_bin(self, stream, byte_order): + """ + Read data from a binary stream. Raise StopIteration if the + property could not be read. + + """ + try: + return _np.fromfile(stream, self.dtype(byte_order), 1)[0] + except IndexError: + raise StopIteration + + def _write_bin(self, data, stream, byte_order): + """ + Write data to a binary stream. + + """ + _np.dtype(self.dtype(byte_order)).type(data).tofile(stream) + + def __str__(self): + val_str = _data_type_reverse[self.val_dtype] + return "property %s %s" % (val_str, self.name) + + def __repr__(self): + return "PlyProperty(%r, %r)" % (self.name, _lookup_type(self.val_dtype)) + + +class PlyListProperty(PlyProperty): + """ + PLY list property description. + + """ + + def __init__(self, name, len_dtype, val_dtype): + PlyProperty.__init__(self, name, val_dtype) + + self.len_dtype = len_dtype + + def _get_len_dtype(self): + return self._len_dtype + + def _set_len_dtype(self, len_dtype): + self._len_dtype = _data_types[_lookup_type(len_dtype)] + + len_dtype = property(_get_len_dtype, _set_len_dtype) + + def dtype(self, byte_order="="): + """ + List properties always have a numpy dtype of "object". + + """ + return "|O" + + def list_dtype(self, byte_order="="): + """ + Return the pair (len_dtype, val_dtype) (both numpy-friendly + strings). + + """ + return (byte_order + self.len_dtype, byte_order + self.val_dtype) + + def _from_fields(self, fields): + len_t, val_t = self.list_dtype() + + n = int(_np.dtype(len_t).type(next(fields))) + + data = _np.loadtxt(list(_islice(fields, n)), val_t, ndmin=1) + if len(data) < n: + raise StopIteration + + return data + + def _to_fields(self, data): + """ + Return generator over the (numerical) PLY representation of the + list data (length followed by actual data). + + """ + len_t, val_t = self.list_dtype() + + data = _np.asarray(data, dtype=val_t).ravel() + + yield _np.dtype(len_t).type(data.size) + for x in data: + yield x + + def _read_bin(self, stream, byte_order): + len_t, val_t = self.list_dtype(byte_order) + + try: + n = _np.fromfile(stream, len_t, 1)[0] + except IndexError: + raise StopIteration + + data = _np.fromfile(stream, val_t, n) + if len(data) < n: + raise StopIteration + + return data + + def _write_bin(self, data, stream, byte_order): + """ + Write data to a binary stream. + + """ + len_t, val_t = self.list_dtype(byte_order) + + data = _np.asarray(data, dtype=val_t).ravel() + + _np.array(data.size, dtype=len_t).tofile(stream) + data.tofile(stream) + + def __str__(self): + len_str = _data_type_reverse[self.len_dtype] + val_str = _data_type_reverse[self.val_dtype] + return "property list %s %s %s" % (len_str, val_str, self.name) + + def __repr__(self): + return "PlyListProperty(%r, %r, %r)" % ( + self.name, + _lookup_type(self.len_dtype), + _lookup_type(self.val_dtype), + ) diff --git a/pointcept/datasets/preprocessing/scannet/scannet_pair/point_cloud_extractor.py b/pointcept/datasets/preprocessing/scannet/scannet_pair/point_cloud_extractor.py new file mode 100644 index 0000000000000000000000000000000000000000..1cbff78d9453bac7efe8359448dabcd6edb60452 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/scannet_pair/point_cloud_extractor.py @@ -0,0 +1,98 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + + +import glob, os +import numpy as np +import cv2 +import torch + + +def extractor(input_path, output_path): + if not os.path.exists(output_path): + os.mkdir(output_path) + + # Load Depth Camera Intrinsic + depth_intrinsic = np.loadtxt(input_path + "/intrinsic/intrinsic_depth.txt") + print("Depth intrinsic: ") + print(depth_intrinsic) + + # Compute Camrea Distance (just for demo, so you can choose the camera distance in frame sampling) + poses = sorted( + glob.glob(input_path + "/pose/*.txt"), + key=lambda a: int(os.path.basename(a).split(".")[0]), + ) + depths = sorted( + glob.glob(input_path + "/depth/*.png"), + key=lambda a: int(os.path.basename(a).split(".")[0]), + ) + colors = sorted( + glob.glob(input_path + "/color/*.png"), + key=lambda a: int(os.path.basename(a).split(".")[0]), + ) + + # # Get Aligned Point Clouds. + for ind, (pose, depth, color) in enumerate(zip(poses, depths, colors)): + name = os.path.basename(pose).split(".")[0] + + if os.path.exists(output_path + "/{}.npz".format(name)): + continue + + try: + print("=" * 50, ": {}".format(pose)) + depth_img = cv2.imread(depth, -1) # read 16bit grayscale image + mask = depth_img != 0 + color_image = cv2.imread(color) + color_image = cv2.resize(color_image, (640, 480)) + color_image = np.reshape(color_image[mask], [-1, 3]) + colors = np.zeros_like(color_image) + colors[:, 0] = color_image[:, 2] + colors[:, 1] = color_image[:, 1] + colors[:, 2] = color_image[:, 0] + + pose = np.loadtxt(poses[ind]) + print("Camera pose: ") + print(pose) + + depth_shift = 1000.0 + x, y = np.meshgrid( + np.linspace(0, depth_img.shape[1] - 1, depth_img.shape[1]), + np.linspace(0, depth_img.shape[0] - 1, depth_img.shape[0]), + ) + uv_depth = np.zeros((depth_img.shape[0], depth_img.shape[1], 3)) + uv_depth[:, :, 0] = x + uv_depth[:, :, 1] = y + uv_depth[:, :, 2] = depth_img / depth_shift + uv_depth = np.reshape(uv_depth, [-1, 3]) + uv_depth = uv_depth[np.where(uv_depth[:, 2] != 0), :].squeeze() + + intrinsic_inv = np.linalg.inv(depth_intrinsic) + fx = depth_intrinsic[0, 0] + fy = depth_intrinsic[1, 1] + cx = depth_intrinsic[0, 2] + cy = depth_intrinsic[1, 2] + bx = depth_intrinsic[0, 3] + by = depth_intrinsic[1, 3] + point_list = [] + n = uv_depth.shape[0] + points = np.ones((n, 4)) + X = (uv_depth[:, 0] - cx) * uv_depth[:, 2] / fx + bx + Y = (uv_depth[:, 1] - cy) * uv_depth[:, 2] / fy + by + points[:, 0] = X + points[:, 1] = Y + points[:, 2] = uv_depth[:, 2] + points_world = np.dot(points, np.transpose(pose)) + print(points_world.shape) + + pcd = dict(coord=points_world[:, :3], color=colors) + # pcd_save = np.zeros((points_world.shape[0], 7)) + # pcd_save[:, :3] = points_world[:, :3] + # pcd_save[:, 3:6] = colors + + # print('Saving npz file...') + # np.savez(output_path + '/{}.npz'.format(name), pcd=pcd_save) + torch.save(pcd, output_path + "/{}.pth".format(name)) + except: + continue diff --git a/pointcept/datasets/preprocessing/scannet/scannet_pair/preprocess.py b/pointcept/datasets/preprocessing/scannet/scannet_pair/preprocess.py new file mode 100644 index 0000000000000000000000000000000000000000..6e096b35c5d625694aa044453eed7d5dd1f3fecf --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/scannet_pair/preprocess.py @@ -0,0 +1,50 @@ +import os +import argparse +import glob +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat +from reader import reader +from point_cloud_extractor import extractor +from compute_full_overlapping import compute_full_overlapping + +frame_skip = 25 + + +def parse_sens(sens_dir, output_dir): + scene_id = os.path.basename(os.path.dirname(sens_dir)) + print(f"Parsing sens data{sens_dir}") + reader( + sens_dir, + os.path.join(output_dir, scene_id), + frame_skip, + export_color_images=True, + export_depth_images=True, + export_poses=True, + export_intrinsics=True, + ) + extractor( + os.path.join(output_dir, scene_id), os.path.join(output_dir, scene_id, "pcd") + ) + compute_full_overlapping(output_dir, scene_id) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + opt = parser.parse_args() + sens_list = sorted(glob.glob(os.path.join(opt.dataset_root, "scans/scene*/*.sens"))) + # Preprocess data. + pool = ProcessPoolExecutor(max_workers=mp.cpu_count()) + # pool = ProcessPoolExecutor(max_workers=1) + print("Processing scenes...") + _ = list(pool.map(parse_sens, sens_list, repeat(opt.output_root))) diff --git a/pointcept/datasets/preprocessing/scannet/scannet_pair/reader.py b/pointcept/datasets/preprocessing/scannet/scannet_pair/reader.py new file mode 100644 index 0000000000000000000000000000000000000000..d21aa0ce88006f34775edc9d9aaf4e750d523197 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannet/scannet_pair/reader.py @@ -0,0 +1,33 @@ +import argparse +import os, sys + +from SensorData import SensorData + + +def reader( + filename, + output_path, + frame_skip, + export_color_images=False, + export_depth_images=False, + export_poses=False, + export_intrinsics=False, +): + if not os.path.exists(output_path): + os.makedirs(output_path) + + # load the data + print("loading %s..." % filename) + sd = SensorData(filename) + if export_depth_images: + sd.export_depth_images( + os.path.join(output_path, "depth"), frame_skip=frame_skip + ) + if export_color_images: + sd.export_color_images( + os.path.join(output_path, "color"), frame_skip=frame_skip + ) + if export_poses: + sd.export_poses(os.path.join(output_path, "pose"), frame_skip=frame_skip) + if export_intrinsics: + sd.export_intrinsics(os.path.join(output_path, "intrinsic")) diff --git a/pointcept/datasets/preprocessing/scannetpp/preprocess_scannetpp.py b/pointcept/datasets/preprocessing/scannetpp/preprocess_scannetpp.py new file mode 100644 index 0000000000000000000000000000000000000000..12dd024261f53bad114aa5d799b0680be1e5d489 --- /dev/null +++ b/pointcept/datasets/preprocessing/scannetpp/preprocess_scannetpp.py @@ -0,0 +1,277 @@ +""" +Preprocessing Script for ScanNet++ +modified from official preprocess code. + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import warnings +import argparse +import json +import torch +import numpy as np +import pandas as pd +import open3d as o3d +import multiprocessing as mp +from collections import OrderedDict +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat +from pathlib import Path + +try: + import pointseg +except: + # Pointseg is located in libs/pointseg + warnings.warn("Pointseg is not installed, superpoint segmentation will be skipped.") + pointseg = None + + +def parse_scene( + name, + split, + dataset_root, + output_root, + label_mapping, + class2idx, + ignore_index=-1, +): + print(f"Parsing scene {name} in {split} split") + dataset_root = Path(dataset_root) + output_root = Path(output_root) + scene_path = dataset_root / "data" / name / "scans" + mesh_path = scene_path / "mesh_aligned_0.05.ply" + segs_path = scene_path / "segments.json" + anno_path = scene_path / "segments_anno.json" + + # load mesh vertices and colors + mesh = o3d.io.read_triangle_mesh(str(mesh_path)) + + # extract mesh information + mesh.compute_vertex_normals(normalized=True) + coord = np.array(mesh.vertices).astype(np.float32) + color = (np.array(mesh.vertex_colors) * 255).astype(np.uint8) + normal = np.array(mesh.vertex_normals).astype(np.float32) + + # extract superpoint information + if pointseg is not None: + vertices_sp = torch.from_numpy(np.array(mesh.vertices).astype(np.float32)) + faces_sp = torch.from_numpy(np.array(mesh.triangles).astype(np.int64)) + superpoint = pointseg.segment_mesh(vertices_sp, faces_sp).numpy() + else: + superpoint = None + + save_path = output_root / split / name + save_path.mkdir(parents=True, exist_ok=True) + np.save(save_path / "coord.npy", coord) + np.save(save_path / "color.npy", color) + np.save(save_path / "normal.npy", normal) + if superpoint is not None: + np.save(save_path / "superpoint.npy", superpoint) + + if split == "test": + return + + # get label on vertices + # load segments = vertices per segment ID + with open(segs_path) as f: + segments = json.load(f) + # load anno = (instance, groups of segments) + with open(anno_path) as f: + anno = json.load(f) + seg_indices = np.array(segments["segIndices"], dtype=np.uint32) + num_vertices = len(seg_indices) + assert num_vertices == len(coord) + semantic_gt = np.ones((num_vertices, 3), dtype=np.int16) * ignore_index + instance_gt = np.ones((num_vertices, 3), dtype=np.int16) * ignore_index + + # number of labels are used per vertex. initially 0 + # increment each time a new label is added + instance_size = np.ones((num_vertices, 3), dtype=np.int16) * np.inf + + # keep track of the size of the instance (#vertices) assigned to each vertex + # later, keep the label of the smallest instance for major label of vertices + # store inf initially so that we can pick the smallest instance + labels_used = np.zeros(num_vertices, dtype=np.int16) + + for idx, instance in enumerate(anno["segGroups"]): + label = instance["label"] + instance["label_orig"] = label + # remap label + instance["label"] = label_mapping.get(label, None) + instance["label_index"] = class2idx.get(instance["label"], ignore_index) + + if instance["label_index"] == ignore_index: + continue + # get all the vertices with segment index in this instance + # and max number of labels not yet applied + mask = np.isin(seg_indices, instance["segments"]) & (labels_used < 3) + size = mask.sum() + if size == 0: + continue + + # get the position to add the label - 0, 1, 2 + label_position = labels_used[mask] + semantic_gt[mask, label_position] = instance["label_index"] + # store all valid instance (include ignored instance) + instance_gt[mask, label_position] = instance["objectId"] + instance_size[mask, label_position] = size + labels_used[mask] += 1 + + # major label is the label of smallest instance for each vertex + # use major label for single class segmentation + # shift major label to the first column + mask = labels_used > 1 + if mask.sum() > 0: + major_label_position = np.argmin(instance_size[mask], axis=1) + + major_semantic_label = semantic_gt[mask, major_label_position] + semantic_gt[mask, major_label_position] = semantic_gt[:, 0][mask] + semantic_gt[:, 0][mask] = major_semantic_label + + major_instance_label = instance_gt[mask, major_label_position] + instance_gt[mask, major_label_position] = instance_gt[:, 0][mask] + instance_gt[:, 0][mask] = major_instance_label + + np.save(save_path / "segment.npy", semantic_gt) + np.save(save_path / "instance.npy", instance_gt) + + +def filter_map_classes(mapping, mapping_type): + if mapping_type == "semantic": + map_key = "semantic_map_to" + elif mapping_type == "instance": + map_key = "instance_map_to" + else: + raise NotImplementedError + # create a dict with classes to be mapped + # classes that don't have mapping are entered as x->x + # otherwise x->y + map_dict = OrderedDict() + + for i in range(mapping.shape[0]): + row = mapping.iloc[i] + class_name = row["class"] + map_target = row[map_key] + + # map to None or some other label -> don't add this class to the label list + try: + if len(map_target) > 0: + # map to None -> don't use this class + if map_target == "None": + pass + else: + # map to something else -> use this class + map_dict[class_name] = map_target + except TypeError: + # nan values -> no mapping, keep label as is + if class_name not in map_dict: + map_dict[class_name] = class_name + + return map_dict + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet++ dataset containing data/metadata/splits.", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val/test folders will be located.", + ) + parser.add_argument( + "--ignore_index", + default=-1, + type=int, + help="Default ignore index.", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + config = parser.parse_args() + + print("Loading meta data...") + config.dataset_root = Path(config.dataset_root) + config.output_root = Path(config.output_root) + + train_list = np.loadtxt( + config.dataset_root / "splits" / "nvs_sem_train.txt", + dtype=str, + ) + print("Num samples in training split:", len(train_list)) + + val_list = np.loadtxt( + config.dataset_root / "splits" / "nvs_sem_val.txt", + dtype=str, + ) + print("Num samples in validation split:", len(val_list)) + + test_list = np.loadtxt( + config.dataset_root / "splits" / "sem_test.txt", + dtype=str, + ) + print("Num samples in testing split:", len(test_list)) + + data_list = np.concatenate([train_list, val_list, test_list]) + split_list = np.concatenate( + [ + np.full_like(train_list, "train"), + np.full_like(val_list, "val"), + np.full_like(test_list, "test"), + ] + ) + + # Parsing label information and mapping + segment_class_names = np.loadtxt( + config.dataset_root / "metadata" / "semantic_benchmark" / "top100.txt", + dtype=str, + delimiter=".", # dummy delimiter to replace " " + ) + print("Num classes in segment class list:", len(segment_class_names)) + + instance_class_names = np.loadtxt( + config.dataset_root / "metadata" / "semantic_benchmark" / "top100_instance.txt", + dtype=str, + delimiter=".", # dummy delimiter to replace " " + ) + print("Num classes in instance class list:", len(instance_class_names)) + + label_mapping = pd.read_csv( + config.dataset_root / "metadata" / "semantic_benchmark" / "map_benchmark.csv" + ) + label_mapping = filter_map_classes(label_mapping, mapping_type="semantic") + class2idx = { + class_name: idx for (idx, class_name) in enumerate(segment_class_names) + } + + print("Processing scenes...") + pool = ProcessPoolExecutor(max_workers=config.num_workers) + _ = list( + pool.map( + parse_scene, + data_list, + split_list, + repeat(config.dataset_root), + repeat(config.output_root), + repeat(label_mapping), + repeat(class2idx), + repeat(config.ignore_index), + ) + ) + pool.shutdown() + # parse_scene( + # data_list[0], + # split_list[0], + # config.dataset_root, + # config.output_root, + # label_mapping, + # class2idx, + # config.ignore_index, + # ) diff --git a/pointcept/datasets/preprocessing/structured3d/download_structured3d.sh b/pointcept/datasets/preprocessing/structured3d/download_structured3d.sh new file mode 100644 index 0000000000000000000000000000000000000000..d9fe4c52369a36d0693eb41cea11be7a58cdca8d --- /dev/null +++ b/pointcept/datasets/preprocessing/structured3d/download_structured3d.sh @@ -0,0 +1,10 @@ +DOWNLOAD_PATH=$1 +mkdir $DOWNLOAD_PATH + +for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 wuxiaoyang@SH-IDC1-10-140-24-32 +do tmux new-session -d -s pano$i && tmux send-keys -t pano$i "cd $DOWNLOAD_PATH && get https://zju-kjl-jointlab-azure.kujiale.com/zju-kjl-jointlab/Structured3D/Structured3D_panorama_$i.zip" Enter +done + +for i in 01 02 03 04 05 06 07 08 10 11 12 13 14 15 16 17 wuxiaoyang@SH-IDC1-10-140-24-32 +do tmux new-session -d -s prsp$i && tmux send-keys -t prsp$i "cd $DOWNLOAD_PATH && wget https://zju-kjl-jointlab-azure.kujiale.com/zju-kjl-jointlab/Structured3D/Structured3D_perspective_full_$i.zip" Enter +done \ No newline at end of file diff --git a/pointcept/datasets/preprocessing/structured3d/preprocess_structured3d.py b/pointcept/datasets/preprocessing/structured3d/preprocess_structured3d.py new file mode 100644 index 0000000000000000000000000000000000000000..92281a617592e96f671e78761d11e77ac8446113 --- /dev/null +++ b/pointcept/datasets/preprocessing/structured3d/preprocess_structured3d.py @@ -0,0 +1,419 @@ +""" +Preprocessing Script for Structured3D + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import argparse +import io +import os +import PIL +from PIL import Image +import cv2 +import zipfile +import numpy as np +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat + +VALID_CLASS_IDS_25 = ( + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 11, + 14, + 15, + 16, + 17, + 18, + 19, + 22, + 24, + 25, + 32, + 34, + 35, + 38, + 39, + 40, +) +CLASS_LABELS_25 = ( + "wall", + "floor", + "cabinet", + "bed", + "chair", + "sofa", + "table", + "door", + "window", + "picture", + "desk", + "shelves", + "curtain", + "dresser", + "pillow", + "mirror", + "ceiling", + "refrigerator", + "television", + "nightstand", + "sink", + "lamp", + "otherstructure", + "otherfurniture", + "otherprop", +) + + +def normal_from_cross_product(points_2d: np.ndarray) -> np.ndarray: + xyz_points_pad = np.pad(points_2d, ((0, 1), (0, 1), (0, 0)), mode="symmetric") + xyz_points_ver = (xyz_points_pad[:, :-1, :] - xyz_points_pad[:, 1:, :])[:-1, :, :] + xyz_points_hor = (xyz_points_pad[:-1, :, :] - xyz_points_pad[1:, :, :])[:, :-1, :] + xyz_normal = np.cross(xyz_points_hor, xyz_points_ver) + xyz_dist = np.linalg.norm(xyz_normal, axis=-1, keepdims=True) + xyz_normal = np.divide( + xyz_normal, xyz_dist, out=np.zeros_like(xyz_normal), where=xyz_dist != 0 + ) + return xyz_normal + + +class Structured3DReader: + def __init__(self, files): + super().__init__() + if isinstance(files, str): + files = [files] + self.readers = [zipfile.ZipFile(f, "r") for f in files] + self.names_mapper = dict() + for idx, reader in enumerate(self.readers): + for name in reader.namelist(): + self.names_mapper[name] = idx + + def filelist(self): + return list(self.names_mapper.keys()) + + def listdir(self, dir_name): + dir_name = dir_name.lstrip(os.path.sep).rstrip(os.path.sep) + file_list = list( + np.unique( + [ + f.replace(dir_name + os.path.sep, "", 1).split(os.path.sep)[0] + for f in self.filelist() + if f.startswith(dir_name + os.path.sep) + ] + ) + ) + if "" in file_list: + file_list.remove("") + return file_list + + def read(self, file_name): + split = self.names_mapper[file_name] + return self.readers[split].read(file_name) + + def read_camera(self, camera_path): + z2y_top_m = np.array([[0, 1, 0], [0, 0, 1], [1, 0, 0]], dtype=np.float32) + cam_extr = np.fromstring(self.read(camera_path), dtype=np.float32, sep=" ") + cam_t = np.matmul(z2y_top_m, cam_extr[:3] / 1000) + if cam_extr.shape[0] > 3: + cam_front, cam_up = cam_extr[3:6], cam_extr[6:9] + cam_n = np.cross(cam_front, cam_up) + cam_r = np.stack((cam_front, cam_up, cam_n), axis=1).astype(np.float32) + cam_r = np.matmul(z2y_top_m, cam_r) + cam_f = cam_extr[9:11] + else: + cam_r = np.eye(3, dtype=np.float32) + cam_f = None + return cam_r, cam_t, cam_f + + def read_depth(self, depth_path): + depth = cv2.imdecode( + np.frombuffer(self.read(depth_path), np.uint8), cv2.IMREAD_UNCHANGED + )[..., np.newaxis] + depth[depth == 0] = 65535 + return depth + + def read_color(self, color_path): + color = cv2.imdecode( + np.frombuffer(self.read(color_path), np.uint8), cv2.IMREAD_UNCHANGED + )[..., :3][..., ::-1] + return color + + def read_segment(self, segment_path): + segment = np.array(PIL.Image.open(io.BytesIO(self.read(segment_path))))[ + ..., np.newaxis + ] + return segment + + +def parse_scene( + scene, + dataset_root, + output_root, + ignore_index=-1, + grid_size=None, + fuse_prsp=True, + fuse_pano=True, + vis=False, +): + assert fuse_prsp or fuse_pano + reader = Structured3DReader( + [ + os.path.join(dataset_root, f) + for f in os.listdir(dataset_root) + if f.endswith(".zip") + ] + ) + scene_id = int(os.path.basename(scene).split("_")[-1]) + if scene_id < 3000: + split = "train" + elif 3000 <= scene_id < 3250: + split = "val" + else: + split = "test" + + print(f"Processing: {scene} in {split}") + rooms = reader.listdir(os.path.join("Structured3D", scene, "2D_rendering")) + for room in rooms: + room_path = os.path.join("Structured3D", scene, "2D_rendering", room) + coord_list = list() + color_list = list() + normal_list = list() + segment_list = list() + if fuse_prsp: + prsp_path = os.path.join(room_path, "perspective", "full") + frames = reader.listdir(prsp_path) + + for frame in frames: + try: + cam_r, cam_t, cam_f = reader.read_camera( + os.path.join(prsp_path, frame, "camera_pose.txt") + ) + depth = reader.read_depth( + os.path.join(prsp_path, frame, "depth.png") + ) + color = reader.read_color( + os.path.join(prsp_path, frame, "rgb_rawlight.png") + ) + segment = reader.read_segment( + os.path.join(prsp_path, frame, "semantic.png") + ) + except: + print( + f"Skipping {scene}_room{room}_frame{frame} perspective view due to loading error" + ) + else: + fx, fy = cam_f + height, width = depth.shape[0], depth.shape[1] + pixel = np.transpose(np.indices((width, height)), (2, 1, 0)) + pixel = pixel.reshape((-1, 2)) + pixel = np.hstack((pixel, np.ones((pixel.shape[0], 1)))) + k = np.diag([1.0, 1.0, 1.0]) + + k[0, 2] = width / 2 + k[1, 2] = height / 2 + + k[0, 0] = k[0, 2] / np.tan(fx) + k[1, 1] = k[1, 2] / np.tan(fy) + coord = ( + depth.reshape((-1, 1)) * (np.linalg.inv(k) @ pixel.T).T + ).reshape(height, width, 3) + coord = coord @ np.array([[0, 0, 1], [0, -1, 0], [1, 0, 0]]) + normal = normal_from_cross_product(coord) + + # Filtering invalid points + view_dist = np.maximum( + np.linalg.norm(coord, axis=-1, keepdims=True), float(10e-5) + ) + cosine_dist = np.sum( + (coord * normal / view_dist), axis=-1, keepdims=True + ) + cosine_dist = np.abs(cosine_dist) + mask = ((cosine_dist > 0.15) & (depth < 65535) & (segment > 0))[ + ..., 0 + ].reshape(-1) + + coord = np.matmul(coord / 1000, cam_r.T) + cam_t + normal = normal_from_cross_product(coord) + + if sum(mask) > 0: + coord_list.append(coord.reshape(-1, 3)[mask]) + color_list.append(color.reshape(-1, 3)[mask]) + normal_list.append(normal.reshape(-1, 3)[mask]) + segment_list.append(segment.reshape(-1, 1)[mask]) + else: + print( + f"Skipping {scene}_room{room}_frame{frame} perspective view due to all points are filtered out" + ) + + if fuse_pano: + pano_path = os.path.join(room_path, "panorama") + try: + _, cam_t, _ = reader.read_camera( + os.path.join(pano_path, "camera_xyz.txt") + ) + depth = reader.read_depth(os.path.join(pano_path, "full", "depth.png")) + color = reader.read_color( + os.path.join(pano_path, "full", "rgb_rawlight.png") + ) + segment = reader.read_segment( + os.path.join(pano_path, "full", "semantic.png") + ) + except: + print(f"Skipping {scene}_room{room} panorama view due to loading error") + else: + p_h, p_w = depth.shape[:2] + p_a = np.arange(p_w, dtype=np.float32) / p_w * 2 * np.pi - np.pi + p_b = np.arange(p_h, dtype=np.float32) / p_h * np.pi * -1 + np.pi / 2 + p_a = np.tile(p_a[None], [p_h, 1])[..., np.newaxis] + p_b = np.tile(p_b[:, None], [1, p_w])[..., np.newaxis] + p_a_sin, p_a_cos, p_b_sin, p_b_cos = ( + np.sin(p_a), + np.cos(p_a), + np.sin(p_b), + np.cos(p_b), + ) + x = depth * p_a_cos * p_b_cos + y = depth * p_b_sin + z = depth * p_a_sin * p_b_cos + coord = np.concatenate([x, y, z], axis=-1) / 1000 + normal = normal_from_cross_product(coord) + + # Filtering invalid points + view_dist = np.maximum( + np.linalg.norm(coord, axis=-1, keepdims=True), float(10e-5) + ) + cosine_dist = np.sum( + (coord * normal / view_dist), axis=-1, keepdims=True + ) + cosine_dist = np.abs(cosine_dist) + mask = ((cosine_dist > 0.15) & (depth < 65535) & (segment > 0))[ + ..., 0 + ].reshape(-1) + coord = coord + cam_t + + if sum(mask) > 0: + coord_list.append(coord.reshape(-1, 3)[mask]) + color_list.append(color.reshape(-1, 3)[mask]) + normal_list.append(normal.reshape(-1, 3)[mask]) + segment_list.append(segment.reshape(-1, 1)[mask]) + else: + print( + f"Skipping {scene}_room{room} panorama view due to all points are filtered out" + ) + + if len(coord_list) > 0: + coord = np.concatenate(coord_list, axis=0) + coord = coord @ np.array([[1, 0, 0], [0, 0, 1], [0, 1, 0]]) + color = np.concatenate(color_list, axis=0) + normal = np.concatenate(normal_list, axis=0) + normal = normal @ np.array([[1, 0, 0], [0, 0, 1], [0, 1, 0]]) + segment = np.concatenate(segment_list, axis=0) + segment25 = np.ones_like(segment, dtype=np.int64) * ignore_index + for idx, value in enumerate(VALID_CLASS_IDS_25): + mask = np.all(segment == value, axis=-1) + segment25[mask] = idx + + data_dict = dict( + coord=coord.astype(np.float32), + color=color.astype(np.uint8), + normal=normal.astype(np.float32), + segment=segment25.astype(np.int16), + ) + # Grid sampling data + if grid_size is not None: + grid_coord = np.floor(coord / grid_size).astype(int) + _, idx = np.unique(grid_coord, axis=0, return_index=True) + coord = coord[idx] + for key in data_dict.keys(): + data_dict[key] = data_dict[key][idx] + + # Save data + save_path = os.path.join( + output_root, split, os.path.basename(scene), f"room_{room}" + ) + os.makedirs(save_path, exist_ok=True) + for key in data_dict.keys(): + np.save(os.path.join(save_path, f"{key}.npy"), data_dict[key]) + + if vis: + from pointcept.utils.visualization import save_point_cloud + + os.makedirs("./vis", exist_ok=True) + save_point_cloud( + coord, color / 255, f"./vis/{scene}_room{room}_color.ply" + ) + save_point_cloud( + coord, (normal + 1) / 2, f"./vis/{scene}_room{room}_normal.ply" + ) + else: + print(f"Skipping {scene}_room{room} due to no valid points") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the ScanNet dataset containing scene folders.", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located.", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + parser.add_argument( + "--grid_size", default=None, type=float, help="Grid size for grid sampling." + ) + parser.add_argument("--ignore_index", default=-1, type=float, help="Ignore index.") + parser.add_argument( + "--fuse_prsp", action="store_true", help="Whether fuse perspective view." + ) + parser.add_argument( + "--fuse_pano", action="store_true", help="Whether fuse panorama view." + ) + config = parser.parse_args() + + reader = Structured3DReader( + [ + os.path.join(config.dataset_root, f) + for f in os.listdir(config.dataset_root) + if f.endswith(".zip") + ] + ) + + scenes_list = reader.listdir("Structured3D") + scenes_list = sorted(scenes_list) + os.makedirs(os.path.join(config.output_root, "train"), exist_ok=True) + os.makedirs(os.path.join(config.output_root, "val"), exist_ok=True) + os.makedirs(os.path.join(config.output_root, "test"), exist_ok=True) + + # Preprocess data. + print("Processing scenes...") + pool = ProcessPoolExecutor(max_workers=config.num_workers) + _ = list( + pool.map( + parse_scene, + scenes_list, + repeat(config.dataset_root), + repeat(config.output_root), + repeat(config.ignore_index), + repeat(config.grid_size), + repeat(config.fuse_prsp), + repeat(config.fuse_pano), + ) + ) + pool.shutdown() diff --git a/pointcept/datasets/preprocessing/waymo/3d_semseg_test_set_frames.txt b/pointcept/datasets/preprocessing/waymo/3d_semseg_test_set_frames.txt new file mode 100644 index 0000000000000000000000000000000000000000..e25dc615331aaff261c25bfc3e6b930fccf880ac --- /dev/null +++ b/pointcept/datasets/preprocessing/waymo/3d_semseg_test_set_frames.txt @@ -0,0 +1,2982 @@ +2830680430134047327_1720_000_1740_000,1558034229922468 +2830680430134047327_1720_000_1740_000,1558034232422787 +2830680430134047327_1720_000_1740_000,1558034222922333 +2830680430134047327_1720_000_1740_000,1558034223422411 +2830680430134047327_1720_000_1740_000,1558034232922788 +2830680430134047327_1720_000_1740_000,1558034234422988 +2830680430134047327_1720_000_1740_000,1558034220422269 +2830680430134047327_1720_000_1740_000,1558034224422390 +2830680430134047327_1720_000_1740_000,1558034230422275 +2830680430134047327_1720_000_1740_000,1558034221922367 +2830680430134047327_1720_000_1740_000,1558034231422462 +2830680430134047327_1720_000_1740_000,1558034233422761 +2830680430134047327_1720_000_1740_000,1558034220922343 +2830680430134047327_1720_000_1740_000,1558034223922363 +2830680430134047327_1720_000_1740_000,1558034221422343 +2830680430134047327_1720_000_1740_000,1558034222422363 +2830680430134047327_1720_000_1740_000,1558034233922841 +2830680430134047327_1720_000_1740_000,1558034231922698 +2830680430134047327_1720_000_1740_000,1558034219922361 +2830680430134047327_1720_000_1740_000,1558034230922233 +14586026017427828517_700_000_720_000,1557363460737615 +14586026017427828517_700_000_720_000,1557363461737535 +14586026017427828517_700_000_720_000,1557363470737492 +14586026017427828517_700_000_720_000,1557363458737857 +14586026017427828517_700_000_720_000,1557363471237472 +14586026017427828517_700_000_720_000,1557363459237861 +14586026017427828517_700_000_720_000,1557363460237653 +14586026017427828517_700_000_720_000,1557363467737151 +14586026017427828517_700_000_720_000,1557363469237474 +14586026017427828517_700_000_720_000,1557363468237103 +14586026017427828517_700_000_720_000,1557363461237576 +14586026017427828517_700_000_720_000,1557363457237421 +14586026017427828517_700_000_720_000,1557363467237306 +14586026017427828517_700_000_720_000,1557363468737287 +14586026017427828517_700_000_720_000,1557363459737792 +14586026017427828517_700_000_720_000,1557363469737664 +14586026017427828517_700_000_720_000,1557363458237773 +14586026017427828517_700_000_720_000,1557363471737431 +14586026017427828517_700_000_720_000,1557363470237558 +14586026017427828517_700_000_720_000,1557363457737578 +6079272500228273268_2480_000_2500_000,1557546171272882 +6079272500228273268_2480_000_2500_000,1557546172772631 +6079272500228273268_2480_000_2500_000,1557546159772245 +6079272500228273268_2480_000_2500_000,1557546169773016 +6079272500228273268_2480_000_2500_000,1557546162272567 +6079272500228273268_2480_000_2500_000,1557546170272998 +6079272500228273268_2480_000_2500_000,1557546161272265 +6079272500228273268_2480_000_2500_000,1557546173772741 +6079272500228273268_2480_000_2500_000,1557546170772958 +6079272500228273268_2480_000_2500_000,1557546163272476 +6079272500228273268_2480_000_2500_000,1557546161772481 +6079272500228273268_2480_000_2500_000,1557546171772762 +6079272500228273268_2480_000_2500_000,1557546163772228 +6079272500228273268_2480_000_2500_000,1557546160772102 +6079272500228273268_2480_000_2500_000,1557546162772554 +6079272500228273268_2480_000_2500_000,1557546172272635 +6079272500228273268_2480_000_2500_000,1557546159272357 +6079272500228273268_2480_000_2500_000,1557546173272639 +6079272500228273268_2480_000_2500_000,1557546169272728 +6079272500228273268_2480_000_2500_000,1557546160272024 +1936395688683397781_2580_000_2600_000,1557546260797397 +1936395688683397781_2580_000_2600_000,1557546263297538 +1936395688683397781_2580_000_2600_000,1557546273797478 +1936395688683397781_2580_000_2600_000,1557546259797381 +1936395688683397781_2580_000_2600_000,1557546261797507 +1936395688683397781_2580_000_2600_000,1557546269797705 +1936395688683397781_2580_000_2600_000,1557546271297274 +1936395688683397781_2580_000_2600_000,1557546260297363 +1936395688683397781_2580_000_2600_000,1557546273297464 +1936395688683397781_2580_000_2600_000,1557546272297368 +1936395688683397781_2580_000_2600_000,1557546261297445 +1936395688683397781_2580_000_2600_000,1557546263797578 +1936395688683397781_2580_000_2600_000,1557546270297559 +1936395688683397781_2580_000_2600_000,1557546269297756 +1936395688683397781_2580_000_2600_000,1557546270797376 +1936395688683397781_2580_000_2600_000,1557546262797541 +1936395688683397781_2580_000_2600_000,1557546259297393 +1936395688683397781_2580_000_2600_000,1557546272797434 +1936395688683397781_2580_000_2600_000,1557546262297511 +1936395688683397781_2580_000_2600_000,1557546271797317 +12537711031998520792_3080_000_3100_000,1559178584137607 +12537711031998520792_3080_000_3100_000,1559178596138055 +12537711031998520792_3080_000_3100_000,1559178598137718 +12537711031998520792_3080_000_3100_000,1559178588137379 +12537711031998520792_3080_000_3100_000,1559178588637547 +12537711031998520792_3080_000_3100_000,1559178597637619 +12537711031998520792_3080_000_3100_000,1559178587137003 +12537711031998520792_3080_000_3100_000,1559178594138804 +12537711031998520792_3080_000_3100_000,1559178584637645 +12537711031998520792_3080_000_3100_000,1559178587637113 +12537711031998520792_3080_000_3100_000,1559178598637642 +12537711031998520792_3080_000_3100_000,1559178595638405 +12537711031998520792_3080_000_3100_000,1559178594638876 +12537711031998520792_3080_000_3100_000,1559178585137535 +12537711031998520792_3080_000_3100_000,1559178586637145 +12537711031998520792_3080_000_3100_000,1559178595138665 +12537711031998520792_3080_000_3100_000,1559178585637309 +12537711031998520792_3080_000_3100_000,1559178586137201 +12537711031998520792_3080_000_3100_000,1559178597137539 +12537711031998520792_3080_000_3100_000,1559178596637690 +614453665074997770_1060_000_1080_000,1557449600160586 +614453665074997770_1060_000_1080_000,1557449609661257 +614453665074997770_1060_000_1080_000,1557449599161330 +614453665074997770_1060_000_1080_000,1557449608161401 +614453665074997770_1060_000_1080_000,1557449607661221 +614453665074997770_1060_000_1080_000,1557449598662064 +614453665074997770_1060_000_1080_000,1557449600660450 +614453665074997770_1060_000_1080_000,1557449596170831 +614453665074997770_1060_000_1080_000,1557449610161229 +614453665074997770_1060_000_1080_000,1557449607161133 +614453665074997770_1060_000_1080_000,1557449610661288 +614453665074997770_1060_000_1080_000,1557449597165803 +614453665074997770_1060_000_1080_000,1557449606161418 +614453665074997770_1060_000_1080_000,1557449599660888 +614453665074997770_1060_000_1080_000,1557449597664327 +614453665074997770_1060_000_1080_000,1557449609161403 +614453665074997770_1060_000_1080_000,1557449606661312 +614453665074997770_1060_000_1080_000,1557449596667769 +614453665074997770_1060_000_1080_000,1557449598163004 +614453665074997770_1060_000_1080_000,1557449608661487 +10488772413132920574_680_000_700_000,1557276689370724 +10488772413132920574_680_000_700_000,1557276681870683 +10488772413132920574_680_000_700_000,1557276687370626 +10488772413132920574_680_000_700_000,1557276691870689 +10488772413132920574_680_000_700_000,1557276677372053 +10488772413132920574_680_000_700_000,1557276688370712 +10488772413132920574_680_000_700_000,1557276689870718 +10488772413132920574_680_000_700_000,1557276687870742 +10488772413132920574_680_000_700_000,1557276690370698 +10488772413132920574_680_000_700_000,1557276690870678 +10488772413132920574_680_000_700_000,1557276677871875 +10488772413132920574_680_000_700_000,1557276679370967 +10488772413132920574_680_000_700_000,1557276678871230 +10488772413132920574_680_000_700_000,1557276688870677 +10488772413132920574_680_000_700_000,1557276691370685 +10488772413132920574_680_000_700_000,1557276680370783 +10488772413132920574_680_000_700_000,1557276678371569 +10488772413132920574_680_000_700_000,1557276679870868 +10488772413132920574_680_000_700_000,1557276680870747 +10488772413132920574_680_000_700_000,1557276681370633 +17174012103392027911_3500_000_3520_000,1570909461425651 +17174012103392027911_3500_000_3520_000,1570909452924866 +17174012103392027911_3500_000_3520_000,1570909453924848 +17174012103392027911_3500_000_3520_000,1570909451425527 +17174012103392027911_3500_000_3520_000,1570909450426192 +17174012103392027911_3500_000_3520_000,1570909460425382 +17174012103392027911_3500_000_3520_000,1570909449427011 +17174012103392027911_3500_000_3520_000,1570909449926640 +17174012103392027911_3500_000_3520_000,1570909459925389 +17174012103392027911_3500_000_3520_000,1570909459425389 +17174012103392027911_3500_000_3520_000,1570909461925903 +17174012103392027911_3500_000_3520_000,1570909462926099 +17174012103392027911_3500_000_3520_000,1570909452425024 +17174012103392027911_3500_000_3520_000,1570909450925798 +17174012103392027911_3500_000_3520_000,1570909463926171 +17174012103392027911_3500_000_3520_000,1570909451925298 +17174012103392027911_3500_000_3520_000,1570909463426072 +17174012103392027911_3500_000_3520_000,1570909462426098 +17174012103392027911_3500_000_3520_000,1570909453424839 +17174012103392027911_3500_000_3520_000,1570909460925464 +16062780403777359835_2580_000_2600_000,1570323463399590 +16062780403777359835_2580_000_2600_000,1570323461899536 +16062780403777359835_2580_000_2600_000,1570323456400014 +16062780403777359835_2580_000_2600_000,1570323465899811 +16062780403777359835_2580_000_2600_000,1570323452400102 +16062780403777359835_2580_000_2600_000,1570323454400277 +16062780403777359835_2580_000_2600_000,1570323454900086 +16062780403777359835_2580_000_2600_000,1570323464399655 +16062780403777359835_2580_000_2600_000,1570323452900168 +16062780403777359835_2580_000_2600_000,1570323453400256 +16062780403777359835_2580_000_2600_000,1570323462899433 +16062780403777359835_2580_000_2600_000,1570323451900017 +16062780403777359835_2580_000_2600_000,1570323466399934 +16062780403777359835_2580_000_2600_000,1570323464899679 +16062780403777359835_2580_000_2600_000,1570323455399863 +16062780403777359835_2580_000_2600_000,1570323453900317 +16062780403777359835_2580_000_2600_000,1570323462399389 +16062780403777359835_2580_000_2600_000,1570323455899851 +16062780403777359835_2580_000_2600_000,1570323465399682 +16062780403777359835_2580_000_2600_000,1570323463899688 +1376304843325714018_3420_000_3440_000,1557855926972381 +1376304843325714018_3420_000_3440_000,1557855912972456 +1376304843325714018_3420_000_3440_000,1557855925972176 +1376304843325714018_3420_000_3440_000,1557855927472462 +1376304843325714018_3420_000_3440_000,1557855917472088 +1376304843325714018_3420_000_3440_000,1557855914472215 +1376304843325714018_3420_000_3440_000,1557855923972406 +1376304843325714018_3420_000_3440_000,1557855914972144 +1376304843325714018_3420_000_3440_000,1557855915972108 +1376304843325714018_3420_000_3440_000,1557855924472251 +1376304843325714018_3420_000_3440_000,1557855926472255 +1376304843325714018_3420_000_3440_000,1557855913472347 +1376304843325714018_3420_000_3440_000,1557855923472548 +1376304843325714018_3420_000_3440_000,1557855915472102 +1376304843325714018_3420_000_3440_000,1557855922972694 +1376304843325714018_3420_000_3440_000,1557855924972252 +1376304843325714018_3420_000_3440_000,1557855916472106 +1376304843325714018_3420_000_3440_000,1557855925472198 +1376304843325714018_3420_000_3440_000,1557855913972269 +1376304843325714018_3420_000_3440_000,1557855916972142 +5648007586817904385_3220_000_3240_000,1569901291300092 +5648007586817904385_3220_000_3240_000,1569901302299589 +5648007586817904385_3220_000_3240_000,1569901290300004 +5648007586817904385_3220_000_3240_000,1569901302799659 +5648007586817904385_3220_000_3240_000,1569901301799512 +5648007586817904385_3220_000_3240_000,1569901290800085 +5648007586817904385_3220_000_3240_000,1569901293800265 +5648007586817904385_3220_000_3240_000,1569901292800206 +5648007586817904385_3220_000_3240_000,1569901300799428 +5648007586817904385_3220_000_3240_000,1569901293300205 +5648007586817904385_3220_000_3240_000,1569901294300108 +5648007586817904385_3220_000_3240_000,1569901301299422 +5648007586817904385_3220_000_3240_000,1569901303299757 +5648007586817904385_3220_000_3240_000,1569901304799913 +5648007586817904385_3220_000_3240_000,1569901303799751 +5648007586817904385_3220_000_3240_000,1569901291800157 +5648007586817904385_3220_000_3240_000,1569901304299911 +5648007586817904385_3220_000_3240_000,1569901292300126 +5648007586817904385_3220_000_3240_000,1569901300299426 +14470988792985854683_760_000_780_000,1567607330924872 +14470988792985854683_760_000_780_000,1567607341924939 +14470988792985854683_760_000_780_000,1567607339424824 +14470988792985854683_760_000_780_000,1567607339924818 +14470988792985854683_760_000_780_000,1567607332924840 +14470988792985854683_760_000_780_000,1567607329924795 +14470988792985854683_760_000_780_000,1567607332424911 +14470988792985854683_760_000_780_000,1567607340924888 +14470988792985854683_760_000_780_000,1567607330424797 +14470988792985854683_760_000_780_000,1567607342924753 +14470988792985854683_760_000_780_000,1567607340424823 +14470988792985854683_760_000_780_000,1567607342424852 +14470988792985854683_760_000_780_000,1567607331424817 +14470988792985854683_760_000_780_000,1567607329424827 +14470988792985854683_760_000_780_000,1567607338924788 +14470988792985854683_760_000_780_000,1567607338424807 +14470988792985854683_760_000_780_000,1567607331924837 +14470988792985854683_760_000_780_000,1567607341424947 +14470988792985854683_760_000_780_000,1567607328424803 +14470988792985854683_760_000_780_000,1567607328924826 +16951245307634830999_1400_000_1420_000,1568599891824038 +16951245307634830999_1400_000_1420_000,1568599901824641 +16951245307634830999_1400_000_1420_000,1568599904324788 +16951245307634830999_1400_000_1420_000,1568599903324933 +16951245307634830999_1400_000_1420_000,1568599902824777 +16951245307634830999_1400_000_1420_000,1568599904824728 +16951245307634830999_1400_000_1420_000,1568599895824670 +16951245307634830999_1400_000_1420_000,1568599891324145 +16951245307634830999_1400_000_1420_000,1568599894824764 +16951245307634830999_1400_000_1420_000,1568599893824574 +16951245307634830999_1400_000_1420_000,1568599894324795 +16951245307634830999_1400_000_1420_000,1568599905824790 +16951245307634830999_1400_000_1420_000,1568599903824893 +16951245307634830999_1400_000_1420_000,1568599902324620 +16951245307634830999_1400_000_1420_000,1568599905324768 +16951245307634830999_1400_000_1420_000,1568599893324265 +16951245307634830999_1400_000_1420_000,1568599892824082 +16951245307634830999_1400_000_1420_000,1568599895324702 +16951245307634830999_1400_000_1420_000,1568599892323941 +17835886859721116155_1860_000_1880_000,1558151678787439 +17835886859721116155_1860_000_1880_000,1558151676287513 +17835886859721116155_1860_000_1880_000,1558151670787584 +17835886859721116155_1860_000_1880_000,1558151680287177 +17835886859721116155_1860_000_1880_000,1558151670287284 +17835886859721116155_1860_000_1880_000,1558151679287439 +17835886859721116155_1860_000_1880_000,1558151679787332 +17835886859721116155_1860_000_1880_000,1558151680787183 +17835886859721116155_1860_000_1880_000,1558151668786681 +17835886859721116155_1860_000_1880_000,1558151678287466 +17835886859721116155_1860_000_1880_000,1558151667787272 +17835886859721116155_1860_000_1880_000,1558151677287479 +17835886859721116155_1860_000_1880_000,1558151669286533 +17835886859721116155_1860_000_1880_000,1558151669786756 +17835886859721116155_1860_000_1880_000,1558151676787561 +17835886859721116155_1860_000_1880_000,1558151668286995 +17835886859721116155_1860_000_1880_000,1558151666786923 +17835886859721116155_1860_000_1880_000,1558151677787410 +17835886859721116155_1860_000_1880_000,1558151667287257 +17835886859721116155_1860_000_1880_000,1558151666286432 +9145030426583202228_1060_000_1080_000,1557424274778866 +9145030426583202228_1060_000_1080_000,1557424275778987 +9145030426583202228_1060_000_1080_000,1557424266779291 +9145030426583202228_1060_000_1080_000,1557424278279219 +9145030426583202228_1060_000_1080_000,1557424276779170 +9145030426583202228_1060_000_1080_000,1557424279279575 +9145030426583202228_1060_000_1080_000,1557424268279175 +9145030426583202228_1060_000_1080_000,1557424277779106 +9145030426583202228_1060_000_1080_000,1557424266279249 +9145030426583202228_1060_000_1080_000,1557424269279152 +9145030426583202228_1060_000_1080_000,1557424268779150 +9145030426583202228_1060_000_1080_000,1557424277279133 +9145030426583202228_1060_000_1080_000,1557424275278791 +9145030426583202228_1060_000_1080_000,1557424265779130 +9145030426583202228_1060_000_1080_000,1557424264779014 +9145030426583202228_1060_000_1080_000,1557424265279048 +9145030426583202228_1060_000_1080_000,1557424267279322 +9145030426583202228_1060_000_1080_000,1557424276279143 +9145030426583202228_1060_000_1080_000,1557424278779413 +9145030426583202228_1060_000_1080_000,1557424267779293 +13781857304705519152_2740_000_2760_000,1558018015472758 +13781857304705519152_2740_000_2760_000,1558018006972289 +13781857304705519152_2740_000_2760_000,1558018007472306 +13781857304705519152_2740_000_2760_000,1558018014472458 +13781857304705519152_2740_000_2760_000,1558018017472179 +13781857304705519152_2740_000_2760_000,1558018014972710 +13781857304705519152_2740_000_2760_000,1558018008472276 +13781857304705519152_2740_000_2760_000,1558018006472299 +13781857304705519152_2740_000_2760_000,1558018004472242 +13781857304705519152_2740_000_2760_000,1558018017972558 +13781857304705519152_2740_000_2760_000,1558018004972259 +13781857304705519152_2740_000_2760_000,1558018007972307 +13781857304705519152_2740_000_2760_000,1558018013972483 +13781857304705519152_2740_000_2760_000,1558018005972338 +13781857304705519152_2740_000_2760_000,1558018016972032 +13781857304705519152_2740_000_2760_000,1558018015972514 +13781857304705519152_2740_000_2760_000,1558018005472310 +13781857304705519152_2740_000_2760_000,1558018003972238 +13781857304705519152_2740_000_2760_000,1558018018472666 +13781857304705519152_2740_000_2760_000,1558018016472185 +5154724129640787887_4840_000_4860_000,1557342396562648 +5154724129640787887_4840_000_4860_000,1557342399062810 +5154724129640787887_4840_000_4860_000,1557342395062806 +5154724129640787887_4840_000_4860_000,1557342405062520 +5154724129640787887_4840_000_4860_000,1557342399562770 +5154724129640787887_4840_000_4860_000,1557342395562652 +5154724129640787887_4840_000_4860_000,1557342406562476 +5154724129640787887_4840_000_4860_000,1557342408562474 +5154724129640787887_4840_000_4860_000,1557342406062444 +5154724129640787887_4840_000_4860_000,1557342397562592 +5154724129640787887_4840_000_4860_000,1557342407562646 +5154724129640787887_4840_000_4860_000,1557342396062602 +5154724129640787887_4840_000_4860_000,1557342409562395 +5154724129640787887_4840_000_4860_000,1557342397062617 +5154724129640787887_4840_000_4860_000,1557342409062401 +5154724129640787887_4840_000_4860_000,1557342398062702 +5154724129640787887_4840_000_4860_000,1557342407062596 +5154724129640787887_4840_000_4860_000,1557342405562490 +5154724129640787887_4840_000_4860_000,1557342408062539 +5154724129640787887_4840_000_4860_000,1557342398562701 +12892154548237137398_2820_000_2840_000,1558018087522764 +12892154548237137398_2820_000_2840_000,1558018098022390 +12892154548237137398_2820_000_2840_000,1558018088022638 +12892154548237137398_2820_000_2840_000,1558018095522691 +12892154548237137398_2820_000_2840_000,1558018087022717 +12892154548237137398_2820_000_2840_000,1558018086022213 +12892154548237137398_2820_000_2840_000,1558018086522385 +12892154548237137398_2820_000_2840_000,1558018085522203 +12892154548237137398_2820_000_2840_000,1558018094522190 +12892154548237137398_2820_000_2840_000,1558018084022848 +12892154548237137398_2820_000_2840_000,1558018085022352 +12892154548237137398_2820_000_2840_000,1558018088522537 +12892154548237137398_2820_000_2840_000,1558018084522834 +12892154548237137398_2820_000_2840_000,1558018097022451 +12892154548237137398_2820_000_2840_000,1558018097522376 +12892154548237137398_2820_000_2840_000,1558018098522395 +12892154548237137398_2820_000_2840_000,1558018096022561 +12892154548237137398_2820_000_2840_000,1558018096522494 +12892154548237137398_2820_000_2840_000,1558018094021934 +12892154548237137398_2820_000_2840_000,1558018095022568 +17262030607996041518_540_000_560_000,1558150357737631 +17262030607996041518_540_000_560_000,1558150360737468 +17262030607996041518_540_000_560_000,1558150358737355 +17262030607996041518_540_000_560_000,1558150346737340 +17262030607996041518_540_000_560_000,1558150350737099 +17262030607996041518_540_000_560_000,1558150347237353 +17262030607996041518_540_000_560_000,1558150349237231 +17262030607996041518_540_000_560_000,1558150348237167 +17262030607996041518_540_000_560_000,1558150359237305 +17262030607996041518_540_000_560_000,1558150348737035 +17262030607996041518_540_000_560_000,1558150359737335 +17262030607996041518_540_000_560_000,1558150347737351 +17262030607996041518_540_000_560_000,1558150350237481 +17262030607996041518_540_000_560_000,1558150356237309 +17262030607996041518_540_000_560_000,1558150349737529 +17262030607996041518_540_000_560_000,1558150356737414 +17262030607996041518_540_000_560_000,1558150346237488 +17262030607996041518_540_000_560_000,1558150358237512 +17262030607996041518_540_000_560_000,1558150360237386 +17262030607996041518_540_000_560_000,1558150357237609 +1735154401471216485_440_000_460_000,1566351679575063 +1735154401471216485_440_000_460_000,1566351680574951 +1735154401471216485_440_000_460_000,1566351667075023 +1735154401471216485_440_000_460_000,1566351668074924 +1735154401471216485_440_000_460_000,1566351681074884 +1735154401471216485_440_000_460_000,1566351679075007 +1735154401471216485_440_000_460_000,1566351671574819 +1735154401471216485_440_000_460_000,1566351670575041 +1735154401471216485_440_000_460_000,1566351681574847 +1735154401471216485_440_000_460_000,1566351678574927 +1735154401471216485_440_000_460_000,1566351667575012 +1735154401471216485_440_000_460_000,1566351668574986 +1735154401471216485_440_000_460_000,1566351678074851 +1735154401471216485_440_000_460_000,1566351670075165 +1735154401471216485_440_000_460_000,1566351671074932 +1735154401471216485_440_000_460_000,1566351680075032 +1735154401471216485_440_000_460_000,1566351677075266 +1735154401471216485_440_000_460_000,1566351669075103 +1735154401471216485_440_000_460_000,1566351669575114 +1735154401471216485_440_000_460_000,1566351677575057 +16721473705085324478_2580_000_2600_000,1559143954073968 +16721473705085324478_2580_000_2600_000,1559143946067629 +16721473705085324478_2580_000_2600_000,1559143948570004 +16721473705085324478_2580_000_2600_000,1559143957574188 +16721473705085324478_2580_000_2600_000,1559143945567167 +16721473705085324478_2580_000_2600_000,1559143945066818 +16721473705085324478_2580_000_2600_000,1559143947068541 +16721473705085324478_2580_000_2600_000,1559143956574149 +16721473705085324478_2580_000_2600_000,1559143958574172 +16721473705085324478_2580_000_2600_000,1559143955573951 +16721473705085324478_2580_000_2600_000,1559143957074228 +16721473705085324478_2580_000_2600_000,1559143947568997 +16721473705085324478_2580_000_2600_000,1559143944066354 +16721473705085324478_2580_000_2600_000,1559143954573995 +16721473705085324478_2580_000_2600_000,1559143946568047 +16721473705085324478_2580_000_2600_000,1559143956074028 +16721473705085324478_2580_000_2600_000,1559143948069446 +16721473705085324478_2580_000_2600_000,1559143944566550 +16721473705085324478_2580_000_2600_000,1559143955073909 +16721473705085324478_2580_000_2600_000,1559143958074171 +5046614299208670619_1760_000_1780_000,1557859931896818 +5046614299208670619_1760_000_1780_000,1557859942397098 +5046614299208670619_1760_000_1780_000,1557859941397484 +5046614299208670619_1760_000_1780_000,1557859941897278 +5046614299208670619_1760_000_1780_000,1557859939397451 +5046614299208670619_1760_000_1780_000,1557859929394856 +5046614299208670619_1760_000_1780_000,1557859938397438 +5046614299208670619_1760_000_1780_000,1557859931396740 +5046614299208670619_1760_000_1780_000,1557859940397424 +5046614299208670619_1760_000_1780_000,1557859930896546 +5046614299208670619_1760_000_1780_000,1557859939897429 +5046614299208670619_1760_000_1780_000,1557859929895843 +5046614299208670619_1760_000_1780_000,1557859928893310 +5046614299208670619_1760_000_1780_000,1557859938897438 +5046614299208670619_1760_000_1780_000,1557859940897554 +5046614299208670619_1760_000_1780_000,1557859942897115 +5046614299208670619_1760_000_1780_000,1557859932897120 +5046614299208670619_1760_000_1780_000,1557859930396386 +5046614299208670619_1760_000_1780_000,1557859928391171 +5046614299208670619_1760_000_1780_000,1557859932396854 +6259508587655502768_780_000_800_000,1557843985062519 +6259508587655502768_780_000_800_000,1557843985562521 +6259508587655502768_780_000_800_000,1557843976562766 +6259508587655502768_780_000_800_000,1557843976062713 +6259508587655502768_780_000_800_000,1557843978562284 +6259508587655502768_780_000_800_000,1557843989062285 +6259508587655502768_780_000_800_000,1557843979062370 +6259508587655502768_780_000_800_000,1557843988562341 +6259508587655502768_780_000_800_000,1557843977562120 +6259508587655502768_780_000_800_000,1557843975562542 +6259508587655502768_780_000_800_000,1557843977062493 +6259508587655502768_780_000_800_000,1557843978062117 +6259508587655502768_780_000_800_000,1557843986562332 +6259508587655502768_780_000_800_000,1557843975062365 +6259508587655502768_780_000_800_000,1557843988062465 +6259508587655502768_780_000_800_000,1557843986062494 +6259508587655502768_780_000_800_000,1557843987062399 +6259508587655502768_780_000_800_000,1557843979562469 +6259508587655502768_780_000_800_000,1557843987562501 +6259508587655502768_780_000_800_000,1557843989562412 +11436803605426256250_1720_000_1740_000,1558151527787782 +11436803605426256250_1720_000_1740_000,1558151526787865 +11436803605426256250_1720_000_1740_000,1558151528287716 +11436803605426256250_1720_000_1740_000,1558151530287466 +11436803605426256250_1720_000_1740_000,1558151537786930 +11436803605426256250_1720_000_1740_000,1558151528787637 +11436803605426256250_1720_000_1740_000,1558151538786570 +11436803605426256250_1720_000_1740_000,1558151540786822 +11436803605426256250_1720_000_1740_000,1558151530787441 +11436803605426256250_1720_000_1740_000,1558151527287885 +11436803605426256250_1720_000_1740_000,1558151539786751 +11436803605426256250_1720_000_1740_000,1558151529787489 +11436803605426256250_1720_000_1740_000,1558151539286648 +11436803605426256250_1720_000_1740_000,1558151526287909 +11436803605426256250_1720_000_1740_000,1558151536786870 +11436803605426256250_1720_000_1740_000,1558151536287214 +11436803605426256250_1720_000_1740_000,1558151529287531 +11436803605426256250_1720_000_1740_000,1558151540286973 +11436803605426256250_1720_000_1740_000,1558151538286751 +11436803605426256250_1720_000_1740_000,1558151537286653 +15410814825574326536_2620_000_2640_000,1557860798372836 +15410814825574326536_2620_000_2640_000,1557860800872838 +15410814825574326536_2620_000_2640_000,1557860790372597 +15410814825574326536_2620_000_2640_000,1557860791372832 +15410814825574326536_2620_000_2640_000,1557860799872854 +15410814825574326536_2620_000_2640_000,1557860789372743 +15410814825574326536_2620_000_2640_000,1557860791872904 +15410814825574326536_2620_000_2640_000,1557860798872877 +15410814825574326536_2620_000_2640_000,1557860788372735 +15410814825574326536_2620_000_2640_000,1557860801372803 +15410814825574326536_2620_000_2640_000,1557860802372685 +15410814825574326536_2620_000_2640_000,1557860801872720 +15410814825574326536_2620_000_2640_000,1557860802872671 +15410814825574326536_2620_000_2640_000,1557860792372830 +15410814825574326536_2620_000_2640_000,1557860790872704 +15410814825574326536_2620_000_2640_000,1557860799372902 +15410814825574326536_2620_000_2640_000,1557860792872709 +15410814825574326536_2620_000_2640_000,1557860788872750 +15410814825574326536_2620_000_2640_000,1557860800372906 +15410814825574326536_2620_000_2640_000,1557860789872662 +13585389505831587326_2560_000_2580_000,1557241472137342 +13585389505831587326_2560_000_2580_000,1557241476637531 +13585389505831587326_2560_000_2580_000,1557241484636865 +13585389505831587326_2560_000_2580_000,1557241473137454 +13585389505831587326_2560_000_2580_000,1557241476137536 +13585389505831587326_2560_000_2580_000,1557241472637386 +13585389505831587326_2560_000_2580_000,1557241485637136 +13585389505831587326_2560_000_2580_000,1557241484136968 +13585389505831587326_2560_000_2580_000,1557241485137091 +13585389505831587326_2560_000_2580_000,1557241473637451 +13585389505831587326_2560_000_2580_000,1557241482137115 +13585389505831587326_2560_000_2580_000,1557241475637469 +13585389505831587326_2560_000_2580_000,1557241483636983 +13585389505831587326_2560_000_2580_000,1557241474637506 +13585389505831587326_2560_000_2580_000,1557241483136950 +13585389505831587326_2560_000_2580_000,1557241486137285 +13585389505831587326_2560_000_2580_000,1557241474137501 +13585389505831587326_2560_000_2580_000,1557241486637439 +13585389505831587326_2560_000_2580_000,1557241475137435 +13585389505831587326_2560_000_2580_000,1557241482636985 +15739335479094705947_1420_000_1440_000,1557240344647374 +15739335479094705947_1420_000_1440_000,1557240333147825 +15739335479094705947_1420_000_1440_000,1557240332647832 +15739335479094705947_1420_000_1440_000,1557240336647687 +15739335479094705947_1420_000_1440_000,1557240345147370 +15739335479094705947_1420_000_1440_000,1557240334147846 +15739335479094705947_1420_000_1440_000,1557240335648112 +15739335479094705947_1420_000_1440_000,1557240345647376 +15739335479094705947_1420_000_1440_000,1557240332147799 +15739335479094705947_1420_000_1440_000,1557240344147429 +15739335479094705947_1420_000_1440_000,1557240342147432 +15739335479094705947_1420_000_1440_000,1557240343647467 +15739335479094705947_1420_000_1440_000,1557240346647461 +15739335479094705947_1420_000_1440_000,1557240343147461 +15739335479094705947_1420_000_1440_000,1557240333647840 +15739335479094705947_1420_000_1440_000,1557240335147955 +15739335479094705947_1420_000_1440_000,1557240342647438 +15739335479094705947_1420_000_1440_000,1557240334647920 +15739335479094705947_1420_000_1440_000,1557240346147451 +15739335479094705947_1420_000_1440_000,1557240336147836 +16743182245734335352_1260_000_1280_000,1557888790949495 +16743182245734335352_1260_000_1280_000,1557888787449383 +16743182245734335352_1260_000_1280_000,1557888788948833 +16743182245734335352_1260_000_1280_000,1557888786949263 +16743182245734335352_1260_000_1280_000,1557888776449903 +16743182245734335352_1260_000_1280_000,1557888780449779 +16743182245734335352_1260_000_1280_000,1557888786448960 +16743182245734335352_1260_000_1280_000,1557888777950853 +16743182245734335352_1260_000_1280_000,1557888789448778 +16743182245734335352_1260_000_1280_000,1557888790449312 +16743182245734335352_1260_000_1280_000,1557888779950298 +16743182245734335352_1260_000_1280_000,1557888778451116 +16743182245734335352_1260_000_1280_000,1557888788449105 +16743182245734335352_1260_000_1280_000,1557888779450837 +16743182245734335352_1260_000_1280_000,1557888776950096 +16743182245734335352_1260_000_1280_000,1557888789949015 +16743182245734335352_1260_000_1280_000,1557888787949303 +16743182245734335352_1260_000_1280_000,1557888778951257 +16743182245734335352_1260_000_1280_000,1557888780949350 +16743182245734335352_1260_000_1280_000,1557888777450467 +4037952268810331899_2420_000_2440_000,1567028476924185 +4037952268810331899_2420_000_2440_000,1567028464925058 +4037952268810331899_2420_000_2440_000,1567028466425018 +4037952268810331899_2420_000_2440_000,1567028477924371 +4037952268810331899_2420_000_2440_000,1567028475423773 +4037952268810331899_2420_000_2440_000,1567028475923773 +4037952268810331899_2420_000_2440_000,1567028478424492 +4037952268810331899_2420_000_2440_000,1567028468424910 +4037952268810331899_2420_000_2440_000,1567028466924954 +4037952268810331899_2420_000_2440_000,1567028477424335 +4037952268810331899_2420_000_2440_000,1567028465925047 +4037952268810331899_2420_000_2440_000,1567028476424000 +4037952268810331899_2420_000_2440_000,1567028474424271 +4037952268810331899_2420_000_2440_000,1567028467924880 +4037952268810331899_2420_000_2440_000,1567028478924633 +4037952268810331899_2420_000_2440_000,1567028467424848 +4037952268810331899_2420_000_2440_000,1567028465425099 +4037952268810331899_2420_000_2440_000,1567028464424994 +4037952268810331899_2420_000_2440_000,1567028468924846 +4037952268810331899_2420_000_2440_000,1567028474924011 +17052666463197337241_4560_000_4580_000,1558019835965165 +17052666463197337241_4560_000_4580_000,1558019834964122 +17052666463197337241_4560_000_4580_000,1558019826962706 +17052666463197337241_4560_000_4580_000,1558019837466540 +17052666463197337241_4560_000_4580_000,1558019823962469 +17052666463197337241_4560_000_4580_000,1558019826462862 +17052666463197337241_4560_000_4580_000,1558019834463718 +17052666463197337241_4560_000_4580_000,1558019827962424 +17052666463197337241_4560_000_4580_000,1558019836465729 +17052666463197337241_4560_000_4580_000,1558019827462613 +17052666463197337241_4560_000_4580_000,1558019833963377 +17052666463197337241_4560_000_4580_000,1558019824462615 +17052666463197337241_4560_000_4580_000,1558019836966268 +17052666463197337241_4560_000_4580_000,1558019835464590 +17052666463197337241_4560_000_4580_000,1558019828462295 +17052666463197337241_4560_000_4580_000,1558019825962899 +17052666463197337241_4560_000_4580_000,1558019824962730 +17052666463197337241_4560_000_4580_000,1558019837966298 +17052666463197337241_4560_000_4580_000,1558019825462832 +17052666463197337241_4560_000_4580_000,1558019838465664 +8197312656120253218_3120_000_3140_000,1569346275474782 +8197312656120253218_3120_000_3140_000,1569346279974791 +8197312656120253218_3120_000_3140_000,1569346268974889 +8197312656120253218_3120_000_3140_000,1569346266474964 +8197312656120253218_3120_000_3140_000,1569346267974935 +8197312656120253218_3120_000_3140_000,1569346269974854 +8197312656120253218_3120_000_3140_000,1569346268474908 +8197312656120253218_3120_000_3140_000,1569346266975023 +8197312656120253218_3120_000_3140_000,1569346265475116 +8197312656120253218_3120_000_3140_000,1569346267475024 +8197312656120253218_3120_000_3140_000,1569346276974820 +8197312656120253218_3120_000_3140_000,1569346275974860 +8197312656120253218_3120_000_3140_000,1569346276474878 +8197312656120253218_3120_000_3140_000,1569346279474792 +8197312656120253218_3120_000_3140_000,1569346269474905 +8197312656120253218_3120_000_3140_000,1569346278974783 +8197312656120253218_3120_000_3140_000,1569346265975042 +8197312656120253218_3120_000_3140_000,1569346277974754 +8197312656120253218_3120_000_3140_000,1569346278474771 +8197312656120253218_3120_000_3140_000,1569346277474745 +7844300897851889216_500_000_520_000,1569180269849584 +7844300897851889216_500_000_520_000,1569180283349326 +7844300897851889216_500_000_520_000,1569180270349514 +7844300897851889216_500_000_520_000,1569180281349367 +7844300897851889216_500_000_520_000,1569180273349112 +7844300897851889216_500_000_520_000,1569180280349315 +7844300897851889216_500_000_520_000,1569180280849273 +7844300897851889216_500_000_520_000,1569180283849207 +7844300897851889216_500_000_520_000,1569180272849305 +7844300897851889216_500_000_520_000,1569180270849484 +7844300897851889216_500_000_520_000,1569180282849497 +7844300897851889216_500_000_520_000,1569180271349596 +7844300897851889216_500_000_520_000,1569180271849879 +7844300897851889216_500_000_520_000,1569180284349457 +7844300897851889216_500_000_520_000,1569180282349589 +7844300897851889216_500_000_520_000,1569180281849491 +7844300897851889216_500_000_520_000,1569180272349632 +7844300897851889216_500_000_520_000,1569180274349414 +7844300897851889216_500_000_520_000,1569180279849307 +7844300897851889216_500_000_520_000,1569180273849225 +14918167237855418464_1420_000_1440_000,1557265451487590 +14918167237855418464_1420_000_1440_000,1557265453487513 +14918167237855418464_1420_000_1440_000,1557265440987220 +14918167237855418464_1420_000_1440_000,1557265452987516 +14918167237855418464_1420_000_1440_000,1557265441487272 +14918167237855418464_1420_000_1440_000,1557265449987389 +14918167237855418464_1420_000_1440_000,1557265450487458 +14918167237855418464_1420_000_1440_000,1557265450987504 +14918167237855418464_1420_000_1440_000,1557265440487216 +14918167237855418464_1420_000_1440_000,1557265452487693 +14918167237855418464_1420_000_1440_000,1557265443487465 +14918167237855418464_1420_000_1440_000,1557265451987681 +14918167237855418464_1420_000_1440_000,1557265453987788 +14918167237855418464_1420_000_1440_000,1557265449487404 +14918167237855418464_1420_000_1440_000,1557265442487348 +14918167237855418464_1420_000_1440_000,1557265439487550 +14918167237855418464_1420_000_1440_000,1557265441987298 +14918167237855418464_1420_000_1440_000,1557265439987371 +14918167237855418464_1420_000_1440_000,1557265443987430 +14918167237855418464_1420_000_1440_000,1557265442987426 +1765211916310163252_4400_000_4420_000,1557548091247400 +1765211916310163252_4400_000_4420_000,1557548092247422 +1765211916310163252_4400_000_4420_000,1557548082747340 +1765211916310163252_4400_000_4420_000,1557548080247436 +1765211916310163252_4400_000_4420_000,1557548081747442 +1765211916310163252_4400_000_4420_000,1557548079747433 +1765211916310163252_4400_000_4420_000,1557548093747379 +1765211916310163252_4400_000_4420_000,1557548079247435 +1765211916310163252_4400_000_4420_000,1557548089247264 +1765211916310163252_4400_000_4420_000,1557548092747360 +1765211916310163252_4400_000_4420_000,1557548093247395 +1765211916310163252_4400_000_4420_000,1557548090747296 +1765211916310163252_4400_000_4420_000,1557548083747413 +1765211916310163252_4400_000_4420_000,1557548091747409 +1765211916310163252_4400_000_4420_000,1557548080747512 +1765211916310163252_4400_000_4420_000,1557548090247209 +1765211916310163252_4400_000_4420_000,1557548089747220 +1765211916310163252_4400_000_4420_000,1557548082247344 +1765211916310163252_4400_000_4420_000,1557548081247513 +1765211916310163252_4400_000_4420_000,1557548083247412 +365416647046203224_1080_000_1100_000,1557424297779078 +365416647046203224_1080_000_1100_000,1557424298279187 +365416647046203224_1080_000_1100_000,1557424284779145 +365416647046203224_1080_000_1100_000,1557424299279496 +365416647046203224_1080_000_1100_000,1557424285779375 +365416647046203224_1080_000_1100_000,1557424286279493 +365416647046203224_1080_000_1100_000,1557424288279208 +365416647046203224_1080_000_1100_000,1557424289279220 +365416647046203224_1080_000_1100_000,1557424286779477 +365416647046203224_1080_000_1100_000,1557424294779296 +365416647046203224_1080_000_1100_000,1557424297279126 +365416647046203224_1080_000_1100_000,1557424288779176 +365416647046203224_1080_000_1100_000,1557424287779352 +365416647046203224_1080_000_1100_000,1557424296779274 +365416647046203224_1080_000_1100_000,1557424298779408 +365416647046203224_1080_000_1100_000,1557424295779354 +365416647046203224_1080_000_1100_000,1557424295279343 +365416647046203224_1080_000_1100_000,1557424287279453 +365416647046203224_1080_000_1100_000,1557424285279259 +365416647046203224_1080_000_1100_000,1557424296279315 +3122599254941105215_2980_000_3000_000,1557267013486064 +3122599254941105215_2980_000_3000_000,1557266999471976 +3122599254941105215_2980_000_3000_000,1557267003971991 +3122599254941105215_2980_000_3000_000,1557267002972068 +3122599254941105215_2980_000_3000_000,1557267011978743 +3122599254941105215_2980_000_3000_000,1557267010473667 +3122599254941105215_2980_000_3000_000,1557267001472099 +3122599254941105215_2980_000_3000_000,1557267009973013 +3122599254941105215_2980_000_3000_000,1557267001972106 +3122599254941105215_2980_000_3000_000,1557267009472852 +3122599254941105215_2980_000_3000_000,1557267013987647 +3122599254941105215_2980_000_3000_000,1557267000972170 +3122599254941105215_2980_000_3000_000,1557267011476593 +3122599254941105215_2980_000_3000_000,1557267012983667 +3122599254941105215_2980_000_3000_000,1557266999972086 +3122599254941105215_2980_000_3000_000,1557267012481088 +3122599254941105215_2980_000_3000_000,1557267010974840 +3122599254941105215_2980_000_3000_000,1557267000472146 +3122599254941105215_2980_000_3000_000,1557267002472069 +3122599254941105215_2980_000_3000_000,1557267003472050 +11672844176539348333_4440_000_4460_000,1557548130247264 +11672844176539348333_4440_000_4460_000,1557548119247298 +11672844176539348333_4440_000_4460_000,1557548120747400 +11672844176539348333_4440_000_4460_000,1557548129247403 +11672844176539348333_4440_000_4460_000,1557548121747436 +11672844176539348333_4440_000_4460_000,1557548131747575 +11672844176539348333_4440_000_4460_000,1557548122747361 +11672844176539348333_4440_000_4460_000,1557548132247553 +11672844176539348333_4440_000_4460_000,1557548129747331 +11672844176539348333_4440_000_4460_000,1557548119747262 +11672844176539348333_4440_000_4460_000,1557548121247414 +11672844176539348333_4440_000_4460_000,1557548133747542 +11672844176539348333_4440_000_4460_000,1557548131247534 +11672844176539348333_4440_000_4460_000,1557548122247407 +11672844176539348333_4440_000_4460_000,1557548120247254 +11672844176539348333_4440_000_4460_000,1557548132747504 +11672844176539348333_4440_000_4460_000,1557548123247374 +11672844176539348333_4440_000_4460_000,1557548133247537 +11672844176539348333_4440_000_4460_000,1557548130747376 +11672844176539348333_4440_000_4460_000,1557548123747487 +17212025549630306883_2500_000_2520_000,1558035014396914 +17212025549630306883_2500_000_2520_000,1558035010397071 +17212025549630306883_2500_000_2520_000,1558035000879571 +17212025549630306883_2500_000_2520_000,1558035010897075 +17212025549630306883_2500_000_2520_000,1558035003389800 +17212025549630306883_2500_000_2520_000,1558034999877494 +17212025549630306883_2500_000_2520_000,1558035001883076 +17212025549630306883_2500_000_2520_000,1558035013896904 +17212025549630306883_2500_000_2520_000,1558035002385104 +17212025549630306883_2500_000_2520_000,1558035013397429 +17212025549630306883_2500_000_2520_000,1558035012398066 +17212025549630306883_2500_000_2520_000,1558035009897309 +17212025549630306883_2500_000_2520_000,1558035011397333 +17212025549630306883_2500_000_2520_000,1558035003892217 +17212025549630306883_2500_000_2520_000,1558035002887308 +17212025549630306883_2500_000_2520_000,1558035004394149 +17212025549630306883_2500_000_2520_000,1558035001381105 +17212025549630306883_2500_000_2520_000,1558035012897961 +17212025549630306883_2500_000_2520_000,1558035011897875 +17212025549630306883_2500_000_2520_000,1558035000378455 +5444585006397501511_160_000_180_000,1557843369612501 +5444585006397501511_160_000_180_000,1557843356612649 +5444585006397501511_160_000_180_000,1557843357612587 +5444585006397501511_160_000_180_000,1557843366112688 +5444585006397501511_160_000_180_000,1557843369112577 +5444585006397501511_160_000_180_000,1557843356112502 +5444585006397501511_160_000_180_000,1557843357112699 +5444585006397501511_160_000_180_000,1557843359112424 +5444585006397501511_160_000_180_000,1557843368612608 +5444585006397501511_160_000_180_000,1557843358612418 +5444585006397501511_160_000_180_000,1557843359612545 +5444585006397501511_160_000_180_000,1557843365112636 +5444585006397501511_160_000_180_000,1557843365612657 +5444585006397501511_160_000_180_000,1557843367112626 +5444585006397501511_160_000_180_000,1557843366612681 +5444585006397501511_160_000_180_000,1557843367612623 +5444585006397501511_160_000_180_000,1557843358112458 +5444585006397501511_160_000_180_000,1557843355112397 +5444585006397501511_160_000_180_000,1557843355612457 +5444585006397501511_160_000_180_000,1557843368112622 +17595457728136868510_860_000_880_000,1568570142949954 +17595457728136868510_860_000_880_000,1568570144449980 +17595457728136868510_860_000_880_000,1568570133450011 +17595457728136868510_860_000_880_000,1568570132449985 +17595457728136868510_860_000_880_000,1568570146949999 +17595457728136868510_860_000_880_000,1568570145450102 +17595457728136868510_860_000_880_000,1568570136950003 +17595457728136868510_860_000_880_000,1568570146449992 +17595457728136868510_860_000_880_000,1568570145950029 +17595457728136868510_860_000_880_000,1568570134450024 +17595457728136868510_860_000_880_000,1568570135449980 +17595457728136868510_860_000_880_000,1568570133950026 +17595457728136868510_860_000_880_000,1568570143449845 +17595457728136868510_860_000_880_000,1568570143949863 +17595457728136868510_860_000_880_000,1568570144950031 +17595457728136868510_860_000_880_000,1568570132950020 +17595457728136868510_860_000_880_000,1568570142449990 +17595457728136868510_860_000_880_000,1568570135950008 +17595457728136868510_860_000_880_000,1568570134950042 +17595457728136868510_860_000_880_000,1568570136450044 +10534368980139017457_4480_000_4500_000,1557548163747266 +10534368980139017457_4480_000_4500_000,1557548173324042 +10534368980139017457_4480_000_4500_000,1557548171335072 +10534368980139017457_4480_000_4500_000,1557548171831184 +10534368980139017457_4480_000_4500_000,1557548172327947 +10534368980139017457_4480_000_4500_000,1557548160747474 +10534368980139017457_4480_000_4500_000,1557548159747321 +10534368980139017457_4480_000_4500_000,1557548170342486 +10534368980139017457_4480_000_4500_000,1557548169845348 +10534368980139017457_4480_000_4500_000,1557548170838932 +10534368980139017457_4480_000_4500_000,1557548162247217 +10534368980139017457_4480_000_4500_000,1557548169346776 +10534368980139017457_4480_000_4500_000,1557548173822919 +10534368980139017457_4480_000_4500_000,1557548162747305 +10534368980139017457_4480_000_4500_000,1557548160247434 +10534368980139017457_4480_000_4500_000,1557548163247304 +10534368980139017457_4480_000_4500_000,1557548159247329 +10534368980139017457_4480_000_4500_000,1557548161247379 +10534368980139017457_4480_000_4500_000,1557548161747254 +10534368980139017457_4480_000_4500_000,1557548172825501 +4593468568253300598_1620_000_1640_000,1558034119947167 +4593468568253300598_1620_000_1640_000,1558034131947521 +4593468568253300598_1620_000_1640_000,1558034130447767 +4593468568253300598_1620_000_1640_000,1558034123947147 +4593468568253300598_1620_000_1640_000,1558034123447155 +4593468568253300598_1620_000_1640_000,1558034131447564 +4593468568253300598_1620_000_1640_000,1558034132447509 +4593468568253300598_1620_000_1640_000,1558034133947605 +4593468568253300598_1620_000_1640_000,1558034130947609 +4593468568253300598_1620_000_1640_000,1558034120947198 +4593468568253300598_1620_000_1640_000,1558034129947874 +4593468568253300598_1620_000_1640_000,1558034121947243 +4593468568253300598_1620_000_1640_000,1558034134447535 +4593468568253300598_1620_000_1640_000,1558034122447204 +4593468568253300598_1620_000_1640_000,1558034120447070 +4593468568253300598_1620_000_1640_000,1558034132947552 +4593468568253300598_1620_000_1640_000,1558034121447241 +4593468568253300598_1620_000_1640_000,1558034124447344 +4593468568253300598_1620_000_1640_000,1558034122947127 +4593468568253300598_1620_000_1640_000,1558034133447706 +5810494922060252082_3720_000_3740_000,1557324791637281 +5810494922060252082_3720_000_3740_000,1557324799137989 +5810494922060252082_3720_000_3740_000,1557324792137386 +5810494922060252082_3720_000_3740_000,1557324793137531 +5810494922060252082_3720_000_3740_000,1557324802137484 +5810494922060252082_3720_000_3740_000,1557324802637768 +5810494922060252082_3720_000_3740_000,1557324793637492 +5810494922060252082_3720_000_3740_000,1557324789137692 +5810494922060252082_3720_000_3740_000,1557324803137953 +5810494922060252082_3720_000_3740_000,1557324800137837 +5810494922060252082_3720_000_3740_000,1557324789637648 +5810494922060252082_3720_000_3740_000,1557324800637433 +5810494922060252082_3720_000_3740_000,1557324792637516 +5810494922060252082_3720_000_3740_000,1557324803638106 +5810494922060252082_3720_000_3740_000,1557324791137526 +5810494922060252082_3720_000_3740_000,1557324790637757 +5810494922060252082_3720_000_3740_000,1557324801637223 +5810494922060252082_3720_000_3740_000,1557324801137028 +5810494922060252082_3720_000_3740_000,1557324799638056 +5810494922060252082_3720_000_3740_000,1557324790137645 +2942662230423855469_880_000_900_000,1559348363724177 +2942662230423855469_880_000_900_000,1559348365724257 +2942662230423855469_880_000_900_000,1559348367224483 +2942662230423855469_880_000_900_000,1559348356718454 +2942662230423855469_880_000_900_000,1559348367724406 +2942662230423855469_880_000_900_000,1559348355717289 +2942662230423855469_880_000_900_000,1559348357719518 +2942662230423855469_880_000_900_000,1559348357218926 +2942662230423855469_880_000_900_000,1559348363224053 +2942662230423855469_880_000_900_000,1559348365224224 +2942662230423855469_880_000_900_000,1559348364724292 +2942662230423855469_880_000_900_000,1559348353716247 +2942662230423855469_880_000_900_000,1559348354716626 +2942662230423855469_880_000_900_000,1559348366224290 +2942662230423855469_880_000_900_000,1559348366724409 +2942662230423855469_880_000_900_000,1559348353216332 +2942662230423855469_880_000_900_000,1559348355216840 +2942662230423855469_880_000_900_000,1559348364224254 +2942662230423855469_880_000_900_000,1559348356217995 +2942662230423855469_880_000_900_000,1559348354216438 +5927928428387529213_1640_000_1660_000,1557240564663983 +5927928428387529213_1640_000_1660_000,1557240563163984 +5927928428387529213_1640_000_1660_000,1557240553162670 +5927928428387529213_1640_000_1660_000,1557240566163388 +5927928428387529213_1640_000_1660_000,1557240556162436 +5927928428387529213_1640_000_1660_000,1557240554162851 +5927928428387529213_1640_000_1660_000,1557240552662244 +5927928428387529213_1640_000_1660_000,1557240555162405 +5927928428387529213_1640_000_1660_000,1557240564164016 +5927928428387529213_1640_000_1660_000,1557240552162020 +5927928428387529213_1640_000_1660_000,1557240554662508 +5927928428387529213_1640_000_1660_000,1557240562163098 +5927928428387529213_1640_000_1660_000,1557240566663035 +5927928428387529213_1640_000_1660_000,1557240555662402 +5927928428387529213_1640_000_1660_000,1557240565663746 +5927928428387529213_1640_000_1660_000,1557240562663614 +5927928428387529213_1640_000_1660_000,1557240563664057 +5927928428387529213_1640_000_1660_000,1557240556662471 +5927928428387529213_1640_000_1660_000,1557240553662931 +5927928428387529213_1640_000_1660_000,1557240565163970 +3645211352574995740_3540_000_3560_000,1558018817992298 +3645211352574995740_3540_000_3560_000,1558018804471765 +3645211352574995740_3540_000_3560_000,1558018808472120 +3645211352574995740_3540_000_3560_000,1558018807972072 +3645211352574995740_3540_000_3560_000,1558018815475883 +3645211352574995740_3540_000_3560_000,1558018804971761 +3645211352574995740_3540_000_3560_000,1558018816984976 +3645211352574995740_3540_000_3560_000,1558018815978327 +3645211352574995740_3540_000_3560_000,1558018816481398 +3645211352574995740_3540_000_3560_000,1558018818494946 +3645211352574995740_3540_000_3560_000,1558018817488679 +3645211352574995740_3540_000_3560_000,1558018805471754 +3645211352574995740_3540_000_3560_000,1558018806471940 +3645211352574995740_3540_000_3560_000,1558018807472066 +3645211352574995740_3540_000_3560_000,1558018805971789 +3645211352574995740_3540_000_3560_000,1558018806972056 +3645211352574995740_3540_000_3560_000,1558018814473516 +3645211352574995740_3540_000_3560_000,1558018813973212 +3645211352574995740_3540_000_3560_000,1558018814974307 +3645211352574995740_3540_000_3560_000,1558018803972077 +3510690431623954420_7700_000_7720_000,1567022016599663 +3510690431623954420_7700_000_7720_000,1567022018599669 +3510690431623954420_7700_000_7720_000,1567022028099832 +3510690431623954420_7700_000_7720_000,1567022017099671 +3510690431623954420_7700_000_7720_000,1567022021099696 +3510690431623954420_7700_000_7720_000,1567022019599567 +3510690431623954420_7700_000_7720_000,1567022020599579 +3510690431623954420_7700_000_7720_000,1567022029600020 +3510690431623954420_7700_000_7720_000,1567022017599659 +3510690431623954420_7700_000_7720_000,1567022026599976 +3510690431623954420_7700_000_7720_000,1567022030099989 +3510690431623954420_7700_000_7720_000,1567022028599676 +3510690431623954420_7700_000_7720_000,1567022019099605 +3510690431623954420_7700_000_7720_000,1567022018099661 +3510690431623954420_7700_000_7720_000,1567022030599800 +3510690431623954420_7700_000_7720_000,1567022027599919 +3510690431623954420_7700_000_7720_000,1567022029099795 +3510690431623954420_7700_000_7720_000,1567022020099574 +3510690431623954420_7700_000_7720_000,1567022027099913 +3510690431623954420_7700_000_7720_000,1567022031099758 +39847154216997509_6440_000_6460_000,1568954824924191 +39847154216997509_6440_000_6460_000,1568954813424574 +39847154216997509_6440_000_6460_000,1568954813924259 +39847154216997509_6440_000_6460_000,1568954811424618 +39847154216997509_6440_000_6460_000,1568954822924591 +39847154216997509_6440_000_6460_000,1568954812924890 +39847154216997509_6440_000_6460_000,1568954820924315 +39847154216997509_6440_000_6460_000,1568954810424785 +39847154216997509_6440_000_6460_000,1568954811924639 +39847154216997509_6440_000_6460_000,1568954810924694 +39847154216997509_6440_000_6460_000,1568954814924374 +39847154216997509_6440_000_6460_000,1568954823424463 +39847154216997509_6440_000_6460_000,1568954824424244 +39847154216997509_6440_000_6460_000,1568954814424260 +39847154216997509_6440_000_6460_000,1568954821924250 +39847154216997509_6440_000_6460_000,1568954821424322 +39847154216997509_6440_000_6460_000,1568954820424237 +39847154216997509_6440_000_6460_000,1568954823924349 +39847154216997509_6440_000_6460_000,1568954812424884 +39847154216997509_6440_000_6460_000,1568954822424440 +8623236016759087157_3500_000_3520_000,1557324582561015 +8623236016759087157_3500_000_3520_000,1557324569137429 +8623236016759087157_3500_000_3520_000,1557324582058259 +8623236016759087157_3500_000_3520_000,1557324583062491 +8623236016759087157_3500_000_3520_000,1557324573638623 +8623236016759087157_3500_000_3520_000,1557324572138081 +8623236016759087157_3500_000_3520_000,1557324583562892 +8623236016759087157_3500_000_3520_000,1557324570637898 +8623236016759087157_3500_000_3520_000,1557324581052320 +8623236016759087157_3500_000_3520_000,1557324571638078 +8623236016759087157_3500_000_3520_000,1557324570137931 +8623236016759087157_3500_000_3520_000,1557324580549644 +8623236016759087157_3500_000_3520_000,1557324571138013 +8623236016759087157_3500_000_3520_000,1557324579042078 +8623236016759087157_3500_000_3520_000,1557324581555154 +8623236016759087157_3500_000_3520_000,1557324572638279 +8623236016759087157_3500_000_3520_000,1557324579544532 +8623236016759087157_3500_000_3520_000,1557324580047183 +8623236016759087157_3500_000_3520_000,1557324569637588 +8623236016759087157_3500_000_3520_000,1557324573138464 +8920841445900141920_1700_000_1720_000,1557859882950427 +8920841445900141920_1700_000_1720_000,1557859870947785 +8920841445900141920_1700_000_1720_000,1557859868947629 +8920841445900141920_1700_000_1720_000,1557859882449903 +8920841445900141920_1700_000_1720_000,1557859878447989 +8920841445900141920_1700_000_1720_000,1557859872447754 +8920841445900141920_1700_000_1720_000,1557859879448016 +8920841445900141920_1700_000_1720_000,1557859879948093 +8920841445900141920_1700_000_1720_000,1557859869447788 +8920841445900141920_1700_000_1720_000,1557859881448912 +8920841445900141920_1700_000_1720_000,1557859870447773 +8920841445900141920_1700_000_1720_000,1557859880448231 +8920841445900141920_1700_000_1720_000,1557859878947993 +8920841445900141920_1700_000_1720_000,1557859880948478 +8920841445900141920_1700_000_1720_000,1557859869947772 +8920841445900141920_1700_000_1720_000,1557859881949366 +8920841445900141920_1700_000_1720_000,1557859872947901 +8920841445900141920_1700_000_1720_000,1557859871947716 +8920841445900141920_1700_000_1720_000,1557859871447819 +8920841445900141920_1700_000_1720_000,1557859868447474 +1417898473608326362_2560_000_2580_000,1557546242797474 +1417898473608326362_2560_000_2580_000,1557546239797468 +1417898473608326362_2560_000_2580_000,1557546241797368 +1417898473608326362_2560_000_2580_000,1557546252797649 +1417898473608326362_2560_000_2580_000,1557546252297680 +1417898473608326362_2560_000_2580_000,1557546239297163 +1417898473608326362_2560_000_2580_000,1557546253797788 +1417898473608326362_2560_000_2580_000,1557546249297427 +1417898473608326362_2560_000_2580_000,1557546242297446 +1417898473608326362_2560_000_2580_000,1557546251297740 +1417898473608326362_2560_000_2580_000,1557546240297658 +1417898473608326362_2560_000_2580_000,1557546240797643 +1417898473608326362_2560_000_2580_000,1557546250297550 +1417898473608326362_2560_000_2580_000,1557546249797555 +1417898473608326362_2560_000_2580_000,1557546251797725 +1417898473608326362_2560_000_2580_000,1557546250797666 +1417898473608326362_2560_000_2580_000,1557546253297756 +1417898473608326362_2560_000_2580_000,1557546243797028 +1417898473608326362_2560_000_2580_000,1557546243297291 +1417898473608326362_2560_000_2580_000,1557546241297483 +9584760613582366524_1620_000_1640_000,1557879417399208 +9584760613582366524_1620_000_1640_000,1557879416899412 +9584760613582366524_1620_000_1640_000,1557879428399102 +9584760613582366524_1620_000_1640_000,1557879420399302 +9584760613582366524_1620_000_1640_000,1557879427399045 +9584760613582366524_1620_000_1640_000,1557879420899353 +9584760613582366524_1620_000_1640_000,1557879426899061 +9584760613582366524_1620_000_1640_000,1557879418899485 +9584760613582366524_1620_000_1640_000,1557879418399553 +9584760613582366524_1620_000_1640_000,1557879429898992 +9584760613582366524_1620_000_1640_000,1557879428899097 +9584760613582366524_1620_000_1640_000,1557879430898987 +9584760613582366524_1620_000_1640_000,1557879429399097 +9584760613582366524_1620_000_1640_000,1557879421399451 +9584760613582366524_1620_000_1640_000,1557879431398990 +9584760613582366524_1620_000_1640_000,1557879419899335 +9584760613582366524_1620_000_1640_000,1557879419399372 +9584760613582366524_1620_000_1640_000,1557879430398927 +9584760613582366524_1620_000_1640_000,1557879417899434 +9584760613582366524_1620_000_1640_000,1557879427899058 +6503078254504013503_3440_000_3460_000,1557855947547440 +6503078254504013503_3440_000_3460_000,1557855934472627 +6503078254504013503_3440_000_3460_000,1557855932972711 +6503078254504013503_3440_000_3460_000,1557855934972072 +6503078254504013503_3440_000_3460_000,1557855946547513 +6503078254504013503_3440_000_3460_000,1557855933972741 +6503078254504013503_3440_000_3460_000,1557855945047402 +6503078254504013503_3440_000_3460_000,1557855936962356 +6503078254504013503_3440_000_3460_000,1557855945547411 +6503078254504013503_3440_000_3460_000,1557855947047525 +6503078254504013503_3440_000_3460_000,1557855944547167 +6503078254504013503_3440_000_3460_000,1557855944046932 +6503078254504013503_3440_000_3460_000,1557855937459144 +6503078254504013503_3440_000_3460_000,1557855933472775 +6503078254504013503_3440_000_3460_000,1557855946047387 +6503078254504013503_3440_000_3460_000,1557855935470483 +6503078254504013503_3440_000_3460_000,1557855943047114 +6503078254504013503_3440_000_3460_000,1557855935968223 +6503078254504013503_3440_000_3460_000,1557855943547034 +6503078254504013503_3440_000_3460_000,1557855936465449 +11867874114645674271_600_000_620_000,1556074736854433 +11867874114645674271_600_000_620_000,1556074726349701 +11867874114645674271_600_000_620_000,1556074735851657 +11867874114645674271_600_000_620_000,1556074725849692 +11867874114645674271_600_000_620_000,1556074737859199 +11867874114645674271_600_000_620_000,1556074738362254 +11867874114645674271_600_000_620_000,1556074727849804 +11867874114645674271_600_000_620_000,1556074733850341 +11867874114645674271_600_000_620_000,1556074724350381 +11867874114645674271_600_000_620_000,1556074735350931 +11867874114645674271_600_000_620_000,1556074728349730 +11867874114645674271_600_000_620_000,1556074724849999 +11867874114645674271_600_000_620_000,1556074725349782 +11867874114645674271_600_000_620_000,1556074726849817 +11867874114645674271_600_000_620_000,1556074727349951 +11867874114645674271_600_000_620_000,1556074723850636 +11867874114645674271_600_000_620_000,1556074736352936 +11867874114645674271_600_000_620_000,1556074737356543 +11867874114645674271_600_000_620_000,1556074734850605 +2374138435300423201_2600_000_2620_000,1557546281297269 +2374138435300423201_2600_000_2620_000,1557546290797384 +2374138435300423201_2600_000_2620_000,1557546279797210 +2374138435300423201_2600_000_2620_000,1557546282797429 +2374138435300423201_2600_000_2620_000,1557546279297244 +2374138435300423201_2600_000_2620_000,1557546280797280 +2374138435300423201_2600_000_2620_000,1557546280297328 +2374138435300423201_2600_000_2620_000,1557546289297324 +2374138435300423201_2600_000_2620_000,1557546289797335 +2374138435300423201_2600_000_2620_000,1557546283297421 +2374138435300423201_2600_000_2620_000,1557546293797422 +2374138435300423201_2600_000_2620_000,1557546283797387 +2374138435300423201_2600_000_2620_000,1557546291297442 +2374138435300423201_2600_000_2620_000,1557546292797289 +2374138435300423201_2600_000_2620_000,1557546293297352 +2374138435300423201_2600_000_2620_000,1557546282297473 +2374138435300423201_2600_000_2620_000,1557546290297367 +2374138435300423201_2600_000_2620_000,1557546281797389 +2374138435300423201_2600_000_2620_000,1557546292297340 +2374138435300423201_2600_000_2620_000,1557546291797459 +16050146835908439029_4500_000_4520_000,1557862669362069 +16050146835908439029_4500_000_4520_000,1557862668362475 +16050146835908439029_4500_000_4520_000,1557862680862489 +16050146835908439029_4500_000_4520_000,1557862682362527 +16050146835908439029_4500_000_4520_000,1557862679362451 +16050146835908439029_4500_000_4520_000,1557862669862200 +16050146835908439029_4500_000_4520_000,1557862680362483 +16050146835908439029_4500_000_4520_000,1557862670362417 +16050146835908439029_4500_000_4520_000,1557862668862219 +16050146835908439029_4500_000_4520_000,1557862682862598 +16050146835908439029_4500_000_4520_000,1557862681362512 +16050146835908439029_4500_000_4520_000,1557862672362384 +16050146835908439029_4500_000_4520_000,1557862672862388 +16050146835908439029_4500_000_4520_000,1557862670862532 +16050146835908439029_4500_000_4520_000,1557862671862452 +16050146835908439029_4500_000_4520_000,1557862681862522 +16050146835908439029_4500_000_4520_000,1557862678862529 +16050146835908439029_4500_000_4520_000,1557862671362531 +16050146835908439029_4500_000_4520_000,1557862679862445 +16050146835908439029_4500_000_4520_000,1557862678362527 +3400465735719851775_1400_000_1420_000,1572136113149873 +3400465735719851775_1400_000_1420_000,1572136101149909 +3400465735719851775_1400_000_1420_000,1572136102649696 +3400465735719851775_1400_000_1420_000,1572136113649695 +3400465735719851775_1400_000_1420_000,1572136110649929 +3400465735719851775_1400_000_1420_000,1572136111649912 +3400465735719851775_1400_000_1420_000,1572136112649991 +3400465735719851775_1400_000_1420_000,1572136114149800 +3400465735719851775_1400_000_1420_000,1572136100649954 +3400465735719851775_1400_000_1420_000,1572136114649927 +3400465735719851775_1400_000_1420_000,1572136103149884 +3400465735719851775_1400_000_1420_000,1572136112149956 +3400465735719851775_1400_000_1420_000,1572136105149812 +3400465735719851775_1400_000_1420_000,1572136103650115 +3400465735719851775_1400_000_1420_000,1572136115150038 +3400465735719851775_1400_000_1420_000,1572136102149777 +3400465735719851775_1400_000_1420_000,1572136111149884 +3400465735719851775_1400_000_1420_000,1572136104150206 +3400465735719851775_1400_000_1420_000,1572136101649856 +3400465735719851775_1400_000_1420_000,1572136104650062 +13347759874869607317_1540_000_1560_000,1557240455162639 +13347759874869607317_1540_000_1560_000,1557240466662585 +13347759874869607317_1540_000_1560_000,1557240462162692 +13347759874869607317_1540_000_1560_000,1557240454162643 +13347759874869607317_1540_000_1560_000,1557240453162676 +13347759874869607317_1540_000_1560_000,1557240464162672 +13347759874869607317_1540_000_1560_000,1557240462662650 +13347759874869607317_1540_000_1560_000,1557240463162605 +13347759874869607317_1540_000_1560_000,1557240466162631 +13347759874869607317_1540_000_1560_000,1557240452662663 +13347759874869607317_1540_000_1560_000,1557240465662701 +13347759874869607317_1540_000_1560_000,1557240464662665 +13347759874869607317_1540_000_1560_000,1557240452162656 +13347759874869607317_1540_000_1560_000,1557240455662596 +13347759874869607317_1540_000_1560_000,1557240456662719 +13347759874869607317_1540_000_1560_000,1557240456162634 +13347759874869607317_1540_000_1560_000,1557240463662676 +13347759874869607317_1540_000_1560_000,1557240465162708 +13347759874869607317_1540_000_1560_000,1557240454662705 +13347759874869607317_1540_000_1560_000,1557240453662610 +792520390268391604_780_000_800_000,1557276779322398 +792520390268391604_780_000_800_000,1557276789819241 +792520390268391604_780_000_800_000,1557276781822470 +792520390268391604_780_000_800_000,1557276790814750 +792520390268391604_780_000_800_000,1557276787822228 +792520390268391604_780_000_800_000,1557276777822335 +792520390268391604_780_000_800_000,1557276779822444 +792520390268391604_780_000_800_000,1557276791809911 +792520390268391604_780_000_800_000,1557276787322288 +792520390268391604_780_000_800_000,1557276781322325 +792520390268391604_780_000_800_000,1557276778822345 +792520390268391604_780_000_800_000,1557276788821750 +792520390268391604_780_000_800_000,1557276791312453 +792520390268391604_780_000_800_000,1557276780822385 +792520390268391604_780_000_800_000,1557276789320894 +792520390268391604_780_000_800_000,1557276788322108 +792520390268391604_780_000_800_000,1557276778322306 +792520390268391604_780_000_800_000,1557276790317051 +792520390268391604_780_000_800_000,1557276780322459 +792520390268391604_780_000_800_000,1557276777322444 +12555145882162126399_1180_000_1200_000,1558016457446728 +12555145882162126399_1180_000_1200_000,1558016457946731 +12555145882162126399_1180_000_1200_000,1558016443947236 +12555145882162126399_1180_000_1200_000,1558016455946753 +12555145882162126399_1180_000_1200_000,1558016456946665 +12555145882162126399_1180_000_1200_000,1558016445447090 +12555145882162126399_1180_000_1200_000,1558016446446923 +12555145882162126399_1180_000_1200_000,1558016453946646 +12555145882162126399_1180_000_1200_000,1558016446946859 +12555145882162126399_1180_000_1200_000,1558016445947010 +12555145882162126399_1180_000_1200_000,1558016444947161 +12555145882162126399_1180_000_1200_000,1558016455446712 +12555145882162126399_1180_000_1200_000,1558016448446785 +12555145882162126399_1180_000_1200_000,1558016447946858 +12555145882162126399_1180_000_1200_000,1558016458446676 +12555145882162126399_1180_000_1200_000,1558016444447200 +12555145882162126399_1180_000_1200_000,1558016454446636 +12555145882162126399_1180_000_1200_000,1558016454946704 +12555145882162126399_1180_000_1200_000,1558016447446885 +12555145882162126399_1180_000_1200_000,1558016456446713 +2363225200168330815_760_000_780_000,1557363529737707 +2363225200168330815_760_000_780_000,1557363527237757 +2363225200168330815_760_000_780_000,1557363531737831 +2363225200168330815_760_000_780_000,1557363517737551 +2363225200168330815_760_000_780_000,1557363521737684 +2363225200168330815_760_000_780_000,1557363520237879 +2363225200168330815_760_000_780_000,1557363520737853 +2363225200168330815_760_000_780_000,1557363530737748 +2363225200168330815_760_000_780_000,1557363530237741 +2363225200168330815_760_000_780_000,1557363527737802 +2363225200168330815_760_000_780_000,1557363519238031 +2363225200168330815_760_000_780_000,1557363518738025 +2363225200168330815_760_000_780_000,1557363519737941 +2363225200168330815_760_000_780_000,1557363528237770 +2363225200168330815_760_000_780_000,1557363517237246 +2363225200168330815_760_000_780_000,1557363518237827 +2363225200168330815_760_000_780_000,1557363528737726 +2363225200168330815_760_000_780_000,1557363529237740 +2363225200168330815_760_000_780_000,1557363521237784 +2363225200168330815_760_000_780_000,1557363531237786 +3328513486129168664_2080_000_2100_000,1567831757349830 +3328513486129168664_2080_000_2100_000,1567831748349777 +3328513486129168664_2080_000_2100_000,1567831749349801 +3328513486129168664_2080_000_2100_000,1567831759349962 +3328513486129168664_2080_000_2100_000,1567831748849802 +3328513486129168664_2080_000_2100_000,1567831755849834 +3328513486129168664_2080_000_2100_000,1567831745849801 +3328513486129168664_2080_000_2100_000,1567831756349773 +3328513486129168664_2080_000_2100_000,1567831746849942 +3328513486129168664_2080_000_2100_000,1567831750350037 +3328513486129168664_2080_000_2100_000,1567831749849925 +3328513486129168664_2080_000_2100_000,1567831759849964 +3328513486129168664_2080_000_2100_000,1567831747849819 +3328513486129168664_2080_000_2100_000,1567831747349894 +3328513486129168664_2080_000_2100_000,1567831758849989 +3328513486129168664_2080_000_2100_000,1567831758350003 +3328513486129168664_2080_000_2100_000,1567831746349855 +3328513486129168664_2080_000_2100_000,1567831757849928 +3328513486129168664_2080_000_2100_000,1567831756849772 +3328513486129168664_2080_000_2100_000,1567831760349924 +4632556232973423919_2940_000_2960_000,1557266971471958 +4632556232973423919_2940_000_2960_000,1557266969972031 +4632556232973423919_2940_000_2960_000,1557266970971991 +4632556232973423919_2940_000_2960_000,1557266961970295 +4632556232973423919_2940_000_2960_000,1557266959471575 +4632556232973423919_2940_000_2960_000,1557266973972043 +4632556232973423919_2940_000_2960_000,1557266970471994 +4632556232973423919_2940_000_2960_000,1557266959971091 +4632556232973423919_2940_000_2960_000,1557266969472028 +4632556232973423919_2940_000_2960_000,1557266973471990 +4632556232973423919_2940_000_2960_000,1557266962470698 +4632556232973423919_2940_000_2960_000,1557266972972019 +4632556232973423919_2940_000_2960_000,1557266961470258 +4632556232973423919_2940_000_2960_000,1557266972472025 +4632556232973423919_2940_000_2960_000,1557266963471197 +4632556232973423919_2940_000_2960_000,1557266963971575 +4632556232973423919_2940_000_2960_000,1557266962971008 +4632556232973423919_2940_000_2960_000,1557266960970249 +4632556232973423919_2940_000_2960_000,1557266960470534 +4632556232973423919_2940_000_2960_000,1557266971972014 +7855150647548977812_3900_000_3920_000,1557963222297587 +7855150647548977812_3900_000_3920_000,1557963229797360 +7855150647548977812_3900_000_3920_000,1557963219297462 +7855150647548977812_3900_000_3920_000,1557963222797358 +7855150647548977812_3900_000_3920_000,1557963219797630 +7855150647548977812_3900_000_3920_000,1557963221297729 +7855150647548977812_3900_000_3920_000,1557963228797686 +7855150647548977812_3900_000_3920_000,1557963232797406 +7855150647548977812_3900_000_3920_000,1557963230297234 +7855150647548977812_3900_000_3920_000,1557963230797164 +7855150647548977812_3900_000_3920_000,1557963232297399 +7855150647548977812_3900_000_3920_000,1557963221797707 +7855150647548977812_3900_000_3920_000,1557963231797409 +7855150647548977812_3900_000_3920_000,1557963220297852 +7855150647548977812_3900_000_3920_000,1557963233297506 +7855150647548977812_3900_000_3920_000,1557963218797413 +7855150647548977812_3900_000_3920_000,1557963231297227 +7855150647548977812_3900_000_3920_000,1557963229297664 +7855150647548977812_3900_000_3920_000,1557963220797815 +7855150647548977812_3900_000_3920_000,1557963223297048 +6228701001600487900_720_000_740_000,1557196124797396 +6228701001600487900_720_000_740_000,1557196115797133 +6228701001600487900_720_000_740_000,1557196127297339 +6228701001600487900_720_000_740_000,1557196118797295 +6228701001600487900_720_000_740_000,1557196114797019 +6228701001600487900_720_000_740_000,1557196125297368 +6228701001600487900_720_000_740_000,1557196117797593 +6228701001600487900_720_000_740_000,1557196128297444 +6228701001600487900_720_000_740_000,1557196126797262 +6228701001600487900_720_000_740_000,1557196116297146 +6228701001600487900_720_000_740_000,1557196114297509 +6228701001600487900_720_000_740_000,1557196125797316 +6228701001600487900_720_000_740_000,1557196124297381 +6228701001600487900_720_000_740_000,1557196128797516 +6228701001600487900_720_000_740_000,1557196126297244 +6228701001600487900_720_000_740_000,1557196118297420 +6228701001600487900_720_000_740_000,1557196117297567 +6228701001600487900_720_000_740_000,1557196116797296 +6228701001600487900_720_000_740_000,1557196127797382 +6228701001600487900_720_000_740_000,1557196115297027 +5683383258122801095_1040_000_1060_000,1557363799237684 +5683383258122801095_1040_000_1060_000,1557363798737684 +5683383258122801095_1040_000_1060_000,1557363809737739 +5683383258122801095_1040_000_1060_000,1557363801237681 +5683383258122801095_1040_000_1060_000,1557363808237726 +5683383258122801095_1040_000_1060_000,1557363810237674 +5683383258122801095_1040_000_1060_000,1557363797237694 +5683383258122801095_1040_000_1060_000,1557363808737723 +5683383258122801095_1040_000_1060_000,1557363801737692 +5683383258122801095_1040_000_1060_000,1557363807737730 +5683383258122801095_1040_000_1060_000,1557363797737739 +5683383258122801095_1040_000_1060_000,1557363800737713 +5683383258122801095_1040_000_1060_000,1557363799737724 +5683383258122801095_1040_000_1060_000,1557363811737549 +5683383258122801095_1040_000_1060_000,1557363798237739 +5683383258122801095_1040_000_1060_000,1557363810737600 +5683383258122801095_1040_000_1060_000,1557363807237704 +5683383258122801095_1040_000_1060_000,1557363811237550 +5683383258122801095_1040_000_1060_000,1557363800237778 +5683383258122801095_1040_000_1060_000,1557363809237695 +14631629219048194483_2720_000_2740_000,1558017994972187 +14631629219048194483_2720_000_2740_000,1558017997472318 +14631629219048194483_2720_000_2740_000,1558017985972625 +14631629219048194483_2720_000_2740_000,1558017985472656 +14631629219048194483_2720_000_2740_000,1558017986972355 +14631629219048194483_2720_000_2740_000,1558017998472347 +14631629219048194483_2720_000_2740_000,1558017996472360 +14631629219048194483_2720_000_2740_000,1558017983971974 +14631629219048194483_2720_000_2740_000,1558017987972319 +14631629219048194483_2720_000_2740_000,1558017984472180 +14631629219048194483_2720_000_2740_000,1558017984972436 +14631629219048194483_2720_000_2740_000,1558017995972332 +14631629219048194483_2720_000_2740_000,1558017997972298 +14631629219048194483_2720_000_2740_000,1558017994472307 +14631629219048194483_2720_000_2740_000,1558017995472189 +14631629219048194483_2720_000_2740_000,1558017988472347 +14631629219048194483_2720_000_2740_000,1558017986472464 +14631629219048194483_2720_000_2740_000,1558017987472321 +14631629219048194483_2720_000_2740_000,1558017996972405 +14631629219048194483_2720_000_2740_000,1558017993972404 +2906594041697319079_3040_000_3060_000,1557267072486781 +2906594041697319079_3040_000_3060_000,1557267060487498 +2906594041697319079_3040_000_3060_000,1557267073986709 +2906594041697319079_3040_000_3060_000,1557267070986791 +2906594041697319079_3040_000_3060_000,1557267059987458 +2906594041697319079_3040_000_3060_000,1557267071486876 +2906594041697319079_3040_000_3060_000,1557267062487451 +2906594041697319079_3040_000_3060_000,1557267063987482 +2906594041697319079_3040_000_3060_000,1557267063487438 +2906594041697319079_3040_000_3060_000,1557267071986868 +2906594041697319079_3040_000_3060_000,1557267072986667 +2906594041697319079_3040_000_3060_000,1557267069487459 +2906594041697319079_3040_000_3060_000,1557267073486626 +2906594041697319079_3040_000_3060_000,1557267062987469 +2906594041697319079_3040_000_3060_000,1557267061487517 +2906594041697319079_3040_000_3060_000,1557267061987452 +2906594041697319079_3040_000_3060_000,1557267060987578 +2906594041697319079_3040_000_3060_000,1557267070487093 +2906594041697319079_3040_000_3060_000,1557267069987397 +2906594041697319079_3040_000_3060_000,1557267059487462 +2383902674438058857_4420_000_4440_000,1567796626524800 +2383902674438058857_4420_000_4440_000,1567796634024717 +2383902674438058857_4420_000_4440_000,1567796623525141 +2383902674438058857_4420_000_4440_000,1567796634524720 +2383902674438058857_4420_000_4440_000,1567796637024790 +2383902674438058857_4420_000_4440_000,1567796633524726 +2383902674438058857_4420_000_4440_000,1567796623025071 +2383902674438058857_4420_000_4440_000,1567796624525076 +2383902674438058857_4420_000_4440_000,1567796627024804 +2383902674438058857_4420_000_4440_000,1567796627524846 +2383902674438058857_4420_000_4440_000,1567796635024610 +2383902674438058857_4420_000_4440_000,1567796624025081 +2383902674438058857_4420_000_4440_000,1567796625524889 +2383902674438058857_4420_000_4440_000,1567796635524621 +2383902674438058857_4420_000_4440_000,1567796626024858 +2383902674438058857_4420_000_4440_000,1567796636024637 +2383902674438058857_4420_000_4440_000,1567796625024984 +2383902674438058857_4420_000_4440_000,1567796633024733 +2383902674438058857_4420_000_4440_000,1567796637524761 +2383902674438058857_4420_000_4440_000,1567796636524720 +6862795755554967162_2280_000_2300_000,1558152098797641 +6862795755554967162_2280_000_2300_000,1558152096797728 +6862795755554967162_2280_000_2300_000,1558152098297483 +6862795755554967162_2280_000_2300_000,1558152086297472 +6862795755554967162_2280_000_2300_000,1558152088297543 +6862795755554967162_2280_000_2300_000,1558152090297619 +6862795755554967162_2280_000_2300_000,1558152088797546 +6862795755554967162_2280_000_2300_000,1558152096297876 +6862795755554967162_2280_000_2300_000,1558152087797448 +6862795755554967162_2280_000_2300_000,1558152100297819 +6862795755554967162_2280_000_2300_000,1558152089297513 +6862795755554967162_2280_000_2300_000,1558152086797503 +6862795755554967162_2280_000_2300_000,1558152097297600 +6862795755554967162_2280_000_2300_000,1558152099297843 +6862795755554967162_2280_000_2300_000,1558152089797536 +6862795755554967162_2280_000_2300_000,1558152090797668 +6862795755554967162_2280_000_2300_000,1558152099797835 +6862795755554967162_2280_000_2300_000,1558152100797780 +6862795755554967162_2280_000_2300_000,1558152097797483 +6862795755554967162_2280_000_2300_000,1558152087297497 +8085856200343017603_4120_000_4140_000,1557963441810800 +8085856200343017603_4120_000_4140_000,1557963452811392 +8085856200343017603_4120_000_4140_000,1557963442310429 +8085856200343017603_4120_000_4140_000,1557963448811394 +8085856200343017603_4120_000_4140_000,1557963440312587 +8085856200343017603_4120_000_4140_000,1557963452311343 +8085856200343017603_4120_000_4140_000,1557963438812840 +8085856200343017603_4120_000_4140_000,1557963449311428 +8085856200343017603_4120_000_4140_000,1557963450311446 +8085856200343017603_4120_000_4140_000,1557963450811460 +8085856200343017603_4120_000_4140_000,1557963451311480 +8085856200343017603_4120_000_4140_000,1557963441311391 +8085856200343017603_4120_000_4140_000,1557963439312755 +8085856200343017603_4120_000_4140_000,1557963442810720 +8085856200343017603_4120_000_4140_000,1557963453311401 +8085856200343017603_4120_000_4140_000,1557963449811379 +8085856200343017603_4120_000_4140_000,1557963439812771 +8085856200343017603_4120_000_4140_000,1557963443310973 +8085856200343017603_4120_000_4140_000,1557963451811373 +8085856200343017603_4120_000_4140_000,1557963440812062 +15370024704033662533_1240_000_1260_000,1558016507522714 +15370024704033662533_1240_000_1260_000,1558016504022513 +15370024704033662533_1240_000_1260_000,1558016508023021 +15370024704033662533_1240_000_1260_000,1558016516522659 +15370024704033662533_1240_000_1260_000,1558016518522508 +15370024704033662533_1240_000_1260_000,1558016506522488 +15370024704033662533_1240_000_1260_000,1558016516022182 +15370024704033662533_1240_000_1260_000,1558016518022743 +15370024704033662533_1240_000_1260_000,1558016517022970 +15370024704033662533_1240_000_1260_000,1558016514522028 +15370024704033662533_1240_000_1260_000,1558016507022487 +15370024704033662533_1240_000_1260_000,1558016505022580 +15370024704033662533_1240_000_1260_000,1558016517522896 +15370024704033662533_1240_000_1260_000,1558016506022489 +15370024704033662533_1240_000_1260_000,1558016504522546 +15370024704033662533_1240_000_1260_000,1558016514022344 +15370024704033662533_1240_000_1260_000,1558016505522521 +15370024704033662533_1240_000_1260_000,1558016515022010 +15370024704033662533_1240_000_1260_000,1558016515522158 +15370024704033662533_1240_000_1260_000,1558016508523056 +13887882285811432765_740_000_760_000,1557427670612416 +13887882285811432765_740_000_760_000,1557427657104220 +13887882285811432765_740_000_760_000,1557427656098234 +13887882285811432765_740_000_760_000,1557427670112488 +13887882285811432765_740_000_760_000,1557427657607241 +13887882285811432765_740_000_760_000,1557427659611359 +13887882285811432765_740_000_760_000,1557427668112690 +13887882285811432765_740_000_760_000,1557427669112938 +13887882285811432765_740_000_760_000,1557427668612977 +13887882285811432765_740_000_760_000,1557427667112287 +13887882285811432765_740_000_760_000,1557427667612500 +13887882285811432765_740_000_760_000,1557427660611691 +13887882285811432765_740_000_760_000,1557427658610018 +13887882285811432765_740_000_760_000,1557427660111611 +13887882285811432765_740_000_760_000,1557427658109105 +13887882285811432765_740_000_760_000,1557427656601007 +13887882285811432765_740_000_760_000,1557427659110867 +13887882285811432765_740_000_760_000,1557427666112373 +13887882285811432765_740_000_760_000,1557427666612282 +13887882285811432765_740_000_760_000,1557427669612708 +7886090431228432618_1060_000_1080_000,1557427978090024 +7886090431228432618_1060_000_1080_000,1557427986587357 +7886090431228432618_1060_000_1080_000,1557427979089033 +7886090431228432618_1060_000_1080_000,1557427980587825 +7886090431228432618_1060_000_1080_000,1557427988586899 +7886090431228432618_1060_000_1080_000,1557427989086904 +7886090431228432618_1060_000_1080_000,1557427977091041 +7886090431228432618_1060_000_1080_000,1557427976591045 +7886090431228432618_1060_000_1080_000,1557427987587267 +7886090431228432618_1060_000_1080_000,1557427980088231 +7886090431228432618_1060_000_1080_000,1557427987087350 +7886090431228432618_1060_000_1080_000,1557427990587971 +7886090431228432618_1060_000_1080_000,1557427978589494 +7886090431228432618_1060_000_1080_000,1557427979588581 +7886090431228432618_1060_000_1080_000,1557427977590623 +7886090431228432618_1060_000_1080_000,1557427990087424 +7886090431228432618_1060_000_1080_000,1557427988087048 +7886090431228432618_1060_000_1080_000,1557427989587118 +7886090431228432618_1060_000_1080_000,1557427986087349 +7886090431228432618_1060_000_1080_000,1557427976090905 +11096867396355523348_1460_000_1480_000,1557240385647315 +11096867396355523348_1460_000_1480_000,1557240376147639 +11096867396355523348_1460_000_1480_000,1557240383646953 +11096867396355523348_1460_000_1480_000,1557240373147399 +11096867396355523348_1460_000_1480_000,1557240385147284 +11096867396355523348_1460_000_1480_000,1557240383147053 +11096867396355523348_1460_000_1480_000,1557240375647537 +11096867396355523348_1460_000_1480_000,1557240376647555 +11096867396355523348_1460_000_1480_000,1557240382647278 +11096867396355523348_1460_000_1480_000,1557240374147381 +11096867396355523348_1460_000_1480_000,1557240373647402 +11096867396355523348_1460_000_1480_000,1557240382147351 +11096867396355523348_1460_000_1480_000,1557240375147338 +11096867396355523348_1460_000_1480_000,1557240386147261 +11096867396355523348_1460_000_1480_000,1557240384647073 +11096867396355523348_1460_000_1480_000,1557240372647451 +11096867396355523348_1460_000_1480_000,1557240384146914 +11096867396355523348_1460_000_1480_000,1557240386647265 +11096867396355523348_1460_000_1480_000,1557240374647330 +11096867396355523348_1460_000_1480_000,1557240372147515 +5993415832220804439_1020_000_1040_000,1557427938162330 +5993415832220804439_1020_000_1040_000,1557427940162319 +5993415832220804439_1020_000_1040_000,1557427937662244 +5993415832220804439_1020_000_1040_000,1557427946662314 +5993415832220804439_1020_000_1040_000,1557427946162333 +5993415832220804439_1020_000_1040_000,1557427938662319 +5993415832220804439_1020_000_1040_000,1557427948162669 +5993415832220804439_1020_000_1040_000,1557427947162431 +5993415832220804439_1020_000_1040_000,1557427947662672 +5993415832220804439_1020_000_1040_000,1557427949662420 +5993415832220804439_1020_000_1040_000,1557427950162677 +5993415832220804439_1020_000_1040_000,1557427948662689 +5993415832220804439_1020_000_1040_000,1557427950662930 +5993415832220804439_1020_000_1040_000,1557427940662334 +5993415832220804439_1020_000_1040_000,1557427939662313 +5993415832220804439_1020_000_1040_000,1557427936661931 +5993415832220804439_1020_000_1040_000,1557427936161893 +5993415832220804439_1020_000_1040_000,1557427939162305 +5993415832220804439_1020_000_1040_000,1557427937162046 +5993415832220804439_1020_000_1040_000,1557427949162510 +684234579698396203_2540_000_2560_000,1557546221272675 +684234579698396203_2540_000_2560_000,1557546223272676 +684234579698396203_2540_000_2560_000,1557546229272374 +684234579698396203_2540_000_2560_000,1557546232272632 +684234579698396203_2540_000_2560_000,1557546222772668 +684234579698396203_2540_000_2560_000,1557546233775554 +684234579698396203_2540_000_2560_000,1557546230272562 +684234579698396203_2540_000_2560_000,1557546219772629 +684234579698396203_2540_000_2560_000,1557546231272784 +684234579698396203_2540_000_2560_000,1557546221772604 +684234579698396203_2540_000_2560_000,1557546229772445 +684234579698396203_2540_000_2560_000,1557546233273525 +684234579698396203_2540_000_2560_000,1557546220772768 +684234579698396203_2540_000_2560_000,1557546230772716 +684234579698396203_2540_000_2560_000,1557546223772715 +684234579698396203_2540_000_2560_000,1557546231772736 +684234579698396203_2540_000_2560_000,1557546232772749 +684234579698396203_2540_000_2560_000,1557546222272631 +684234579698396203_2540_000_2560_000,1557546220272744 +684234579698396203_2540_000_2560_000,1557546219272563 +16367045247642649300_3060_000_3080_000,1557267091988308 +16367045247642649300_3060_000_3080_000,1557267090487889 +16367045247642649300_3060_000_3080_000,1557267089487659 +16367045247642649300_3060_000_3080_000,1557267093487520 +16367045247642649300_3060_000_3080_000,1557267093987555 +16367045247642649300_3060_000_3080_000,1557267082986958 +16367045247642649300_3060_000_3080_000,1557267080987657 +16367045247642649300_3060_000_3080_000,1557267083987136 +16367045247642649300_3060_000_3080_000,1557267082487269 +16367045247642649300_3060_000_3080_000,1557267080487535 +16367045247642649300_3060_000_3080_000,1557267081987538 +16367045247642649300_3060_000_3080_000,1557267083486940 +16367045247642649300_3060_000_3080_000,1557267079987387 +16367045247642649300_3060_000_3080_000,1557267079487248 +16367045247642649300_3060_000_3080_000,1557267089987808 +16367045247642649300_3060_000_3080_000,1557267092987360 +16367045247642649300_3060_000_3080_000,1557267092487706 +16367045247642649300_3060_000_3080_000,1557267090987837 +16367045247642649300_3060_000_3080_000,1557267081487585 +16367045247642649300_3060_000_3080_000,1557267091488223 +10940141908690367388_4420_000_4440_000,1557325501087726 +10940141908690367388_4420_000_4440_000,1557325493087410 +10940141908690367388_4420_000_4440_000,1557325490587432 +10940141908690367388_4420_000_4440_000,1557325503087783 +10940141908690367388_4420_000_4440_000,1557325501587681 +10940141908690367388_4420_000_4440_000,1557325492087435 +10940141908690367388_4420_000_4440_000,1557325503587721 +10940141908690367388_4420_000_4440_000,1557325491587341 +10940141908690367388_4420_000_4440_000,1557325489587388 +10940141908690367388_4420_000_4440_000,1557325489087347 +10940141908690367388_4420_000_4440_000,1557325490087423 +10940141908690367388_4420_000_4440_000,1557325499587637 +10940141908690367388_4420_000_4440_000,1557325491087364 +10940141908690367388_4420_000_4440_000,1557325493587440 +10940141908690367388_4420_000_4440_000,1557325502087695 +10940141908690367388_4420_000_4440_000,1557325500087574 +10940141908690367388_4420_000_4440_000,1557325502587743 +10940141908690367388_4420_000_4440_000,1557325492587377 +10940141908690367388_4420_000_4440_000,1557325500587663 +10940141908690367388_4420_000_4440_000,1557325499087629 +15865907199900332614_760_000_780_000,1559313080537412 +15865907199900332614_760_000_780_000,1559313078037376 +15865907199900332614_760_000_780_000,1559313080037454 +15865907199900332614_760_000_780_000,1559313079537512 +15865907199900332614_760_000_780_000,1559313078537459 +15865907199900332614_760_000_780_000,1559313089537338 +15865907199900332614_760_000_780_000,1559313077537461 +15865907199900332614_760_000_780_000,1559313091537372 +15865907199900332614_760_000_780_000,1559313081037481 +15865907199900332614_760_000_780_000,1559313087537628 +15865907199900332614_760_000_780_000,1559313077037424 +15865907199900332614_760_000_780_000,1559313079037502 +15865907199900332614_760_000_780_000,1559313090537600 +15865907199900332614_760_000_780_000,1559313089037261 +15865907199900332614_760_000_780_000,1559313088037246 +15865907199900332614_760_000_780_000,1559313091037429 +15865907199900332614_760_000_780_000,1559313087037841 +15865907199900332614_760_000_780_000,1559313081537390 +15865907199900332614_760_000_780_000,1559313090037603 +15865907199900332614_760_000_780_000,1559313088537022 +16418654553014119039_4340_000_4360_000,1557548032247842 +16418654553014119039_4340_000_4360_000,1557548021247344 +16418654553014119039_4340_000_4360_000,1557548020747349 +16418654553014119039_4340_000_4360_000,1557548019247610 +16418654553014119039_4340_000_4360_000,1557548019747557 +16418654553014119039_4340_000_4360_000,1557548022747669 +16418654553014119039_4340_000_4360_000,1557548032748077 +16418654553014119039_4340_000_4360_000,1557548022247554 +16418654553014119039_4340_000_4360_000,1557548020247425 +16418654553014119039_4340_000_4360_000,1557548031247283 +16418654553014119039_4340_000_4360_000,1557548031747513 +16418654553014119039_4340_000_4360_000,1557548021747406 +16418654553014119039_4340_000_4360_000,1557548023747615 +16418654553014119039_4340_000_4360_000,1557548029247116 +16418654553014119039_4340_000_4360_000,1557548030247196 +16418654553014119039_4340_000_4360_000,1557548030747259 +16418654553014119039_4340_000_4360_000,1557548023247650 +16418654553014119039_4340_000_4360_000,1557548029747131 +16418654553014119039_4340_000_4360_000,1557548033248036 +16418654553014119039_4340_000_4360_000,1557548033747756 +2795127582672852315_4140_000_4160_000,1557963462811402 +2795127582672852315_4140_000_4160_000,1557963459811328 +2795127582672852315_4140_000_4160_000,1557963461311393 +2795127582672852315_4140_000_4160_000,1557963468811200 +2795127582672852315_4140_000_4160_000,1557963460311323 +2795127582672852315_4140_000_4160_000,1557963472811254 +2795127582672852315_4140_000_4160_000,1557963459311361 +2795127582672852315_4140_000_4160_000,1557963472311331 +2795127582672852315_4140_000_4160_000,1557963469811253 +2795127582672852315_4140_000_4160_000,1557963473311173 +2795127582672852315_4140_000_4160_000,1557963458811300 +2795127582672852315_4140_000_4160_000,1557963461811317 +2795127582672852315_4140_000_4160_000,1557963460811362 +2795127582672852315_4140_000_4160_000,1557963471811333 +2795127582672852315_4140_000_4160_000,1557963462311357 +2795127582672852315_4140_000_4160_000,1557963463311436 +2795127582672852315_4140_000_4160_000,1557963469311205 +2795127582672852315_4140_000_4160_000,1557963470811412 +2795127582672852315_4140_000_4160_000,1557963471311372 +2795127582672852315_4140_000_4160_000,1557963470311335 +10084636266401282188_1120_000_1140_000,1558407846397548 +10084636266401282188_1120_000_1140_000,1558407843897545 +10084636266401282188_1120_000_1140_000,1558407844397659 +10084636266401282188_1120_000_1140_000,1558407855397331 +10084636266401282188_1120_000_1140_000,1558407854397201 +10084636266401282188_1120_000_1140_000,1558407856897229 +10084636266401282188_1120_000_1140_000,1558407843397428 +10084636266401282188_1120_000_1140_000,1558407857397306 +10084636266401282188_1120_000_1140_000,1558407845897532 +10084636266401282188_1120_000_1140_000,1558407846897582 +10084636266401282188_1120_000_1140_000,1558407855897228 +10084636266401282188_1120_000_1140_000,1558407852897242 +10084636266401282188_1120_000_1140_000,1558407845397550 +10084636266401282188_1120_000_1140_000,1558407856397205 +10084636266401282188_1120_000_1140_000,1558407853897063 +10084636266401282188_1120_000_1140_000,1558407844897621 +10084636266401282188_1120_000_1140_000,1558407847397707 +10084636266401282188_1120_000_1140_000,1558407854897351 +10084636266401282188_1120_000_1140_000,1558407853397165 +10084636266401282188_1120_000_1140_000,1558407842897345 +2709541197299883157_1140_000_1160_000,1558407875897558 +2709541197299883157_1140_000_1160_000,1558407877397532 +2709541197299883157_1140_000_1160_000,1558407873397482 +2709541197299883157_1140_000_1160_000,1558407866897397 +2709541197299883157_1140_000_1160_000,1558407865397535 +2709541197299883157_1140_000_1160_000,1558407862897305 +2709541197299883157_1140_000_1160_000,1558407865897598 +2709541197299883157_1140_000_1160_000,1558407867397220 +2709541197299883157_1140_000_1160_000,1558407866397538 +2709541197299883157_1140_000_1160_000,1558407874397414 +2709541197299883157_1140_000_1160_000,1558407876897664 +2709541197299883157_1140_000_1160_000,1558407876397661 +2709541197299883157_1140_000_1160_000,1558407874897399 +2709541197299883157_1140_000_1160_000,1558407864897431 +2709541197299883157_1140_000_1160_000,1558407863397357 +2709541197299883157_1140_000_1160_000,1558407863897366 +2709541197299883157_1140_000_1160_000,1558407873897410 +2709541197299883157_1140_000_1160_000,1558407872897442 +2709541197299883157_1140_000_1160_000,1558407875397469 +2709541197299883157_1140_000_1160_000,1558407864397400 +13849332693800388551_960_000_980_000,1557264991038089 +13849332693800388551_960_000_980_000,1557264981037854 +13849332693800388551_960_000_980_000,1557264980537799 +13849332693800388551_960_000_980_000,1557264990038023 +13849332693800388551_960_000_980_000,1557264981537583 +13849332693800388551_960_000_980_000,1557264990537919 +13849332693800388551_960_000_980_000,1557264989537908 +13849332693800388551_960_000_980_000,1557264993538114 +13849332693800388551_960_000_980_000,1557264992037794 +13849332693800388551_960_000_980_000,1557264982537109 +13849332693800388551_960_000_980_000,1557264991537831 +13849332693800388551_960_000_980_000,1557264983537470 +13849332693800388551_960_000_980_000,1557264984037733 +13849332693800388551_960_000_980_000,1557264980037600 +13849332693800388551_960_000_980_000,1557264979537445 +13849332693800388551_960_000_980_000,1557264983037216 +13849332693800388551_960_000_980_000,1557264992537947 +13849332693800388551_960_000_980_000,1557264993038041 +13849332693800388551_960_000_980_000,1557264994038073 +13849332693800388551_960_000_980_000,1557264982037313 +10649066155322078676_1660_000_1680_000,1557240584087768 +10649066155322078676_1660_000_1680_000,1557240585587367 +10649066155322078676_1660_000_1680_000,1557240573663029 +10649066155322078676_1660_000_1680_000,1557240584587589 +10649066155322078676_1660_000_1680_000,1557240586087486 +10649066155322078676_1660_000_1680_000,1557240585087446 +10649066155322078676_1660_000_1680_000,1557240575671293 +10649066155322078676_1660_000_1680_000,1557240576677868 +10649066155322078676_1660_000_1680_000,1557240576174505 +10649066155322078676_1660_000_1680_000,1557240582087726 +10649066155322078676_1660_000_1680_000,1557240574666010 +10649066155322078676_1660_000_1680_000,1557240572662201 +10649066155322078676_1660_000_1680_000,1557240572162174 +10649066155322078676_1660_000_1680_000,1557240583587849 +10649066155322078676_1660_000_1680_000,1557240573162360 +10649066155322078676_1660_000_1680_000,1557240582587734 +10649066155322078676_1660_000_1680_000,1557240586587594 +10649066155322078676_1660_000_1680_000,1557240574164269 +10649066155322078676_1660_000_1680_000,1557240575168313 +10649066155322078676_1660_000_1680_000,1557240583087847 +14386836877680112549_4460_000_4480_000,1559179974137579 +14386836877680112549_4460_000_4480_000,1559179965637497 +14386836877680112549_4460_000_4480_000,1559179975137452 +14386836877680112549_4460_000_4480_000,1559179965137491 +14386836877680112549_4460_000_4480_000,1559179967137475 +14386836877680112549_4460_000_4480_000,1559179968137424 +14386836877680112549_4460_000_4480_000,1559179968637431 +14386836877680112549_4460_000_4480_000,1559179977137567 +14386836877680112549_4460_000_4480_000,1559179977637531 +14386836877680112549_4460_000_4480_000,1559179974637544 +14386836877680112549_4460_000_4480_000,1559179975637343 +14386836877680112549_4460_000_4480_000,1559179966637434 +14386836877680112549_4460_000_4480_000,1559179964137409 +14386836877680112549_4460_000_4480_000,1559179967637439 +14386836877680112549_4460_000_4480_000,1559179976637532 +14386836877680112549_4460_000_4480_000,1559179978137338 +14386836877680112549_4460_000_4480_000,1559179978637228 +14386836877680112549_4460_000_4480_000,1559179964637420 +14386836877680112549_4460_000_4480_000,1559179966137487 +14386836877680112549_4460_000_4480_000,1559179976137422 +1703056599550681101_4380_000_4400_000,1557548063747285 +1703056599550681101_4380_000_4400_000,1557548069747442 +1703056599550681101_4380_000_4400_000,1557548060747134 +1703056599550681101_4380_000_4400_000,1557548059247135 +1703056599550681101_4380_000_4400_000,1557548062747196 +1703056599550681101_4380_000_4400_000,1557548061747138 +1703056599550681101_4380_000_4400_000,1557548059747103 +1703056599550681101_4380_000_4400_000,1557548071747485 +1703056599550681101_4380_000_4400_000,1557548062247198 +1703056599550681101_4380_000_4400_000,1557548071247487 +1703056599550681101_4380_000_4400_000,1557548070747406 +1703056599550681101_4380_000_4400_000,1557548073247485 +1703056599550681101_4380_000_4400_000,1557548072747519 +1703056599550681101_4380_000_4400_000,1557548061247054 +1703056599550681101_4380_000_4400_000,1557548070247363 +1703056599550681101_4380_000_4400_000,1557548063247235 +1703056599550681101_4380_000_4400_000,1557548060247093 +1703056599550681101_4380_000_4400_000,1557548072247479 +1703056599550681101_4380_000_4400_000,1557548069247567 +1703056599550681101_4380_000_4400_000,1557548073747477 +9806821842001738961_4460_000_4480_000,1557548152749185 +9806821842001738961_4460_000_4480_000,1557548152249507 +9806821842001738961_4460_000_4480_000,1557548139248527 +9806821842001738961_4460_000_4480_000,1557548139748613 +9806821842001738961_4460_000_4480_000,1557548149748710 +9806821842001738961_4460_000_4480_000,1557548143745069 +9806821842001738961_4460_000_4480_000,1557548141247955 +9806821842001738961_4460_000_4480_000,1557548150749859 +9806821842001738961_4460_000_4480_000,1557548153248836 +9806821842001738961_4460_000_4480_000,1557548142746485 +9806821842001738961_4460_000_4480_000,1557548151749796 +9806821842001738961_4460_000_4480_000,1557548140248466 +9806821842001738961_4460_000_4480_000,1557548143245860 +9806821842001738961_4460_000_4480_000,1557548141747585 +9806821842001738961_4460_000_4480_000,1557548149247731 +9806821842001738961_4460_000_4480_000,1557548153748607 +9806821842001738961_4460_000_4480_000,1557548142247085 +9806821842001738961_4460_000_4480_000,1557548150249485 +9806821842001738961_4460_000_4480_000,1557548151249946 +9806821842001738961_4460_000_4480_000,1557548140748234 +4008112367880337022_3680_000_3700_000,1569854705325111 +4008112367880337022_3680_000_3700_000,1569854713325049 +4008112367880337022_3680_000_3700_000,1569854717325186 +4008112367880337022_3680_000_3700_000,1569854717825065 +4008112367880337022_3680_000_3700_000,1569854716325211 +4008112367880337022_3680_000_3700_000,1569854716825240 +4008112367880337022_3680_000_3700_000,1569854714325134 +4008112367880337022_3680_000_3700_000,1569854706825153 +4008112367880337022_3680_000_3700_000,1569854704325165 +4008112367880337022_3680_000_3700_000,1569854714825260 +4008112367880337022_3680_000_3700_000,1569854706325106 +4008112367880337022_3680_000_3700_000,1569854705825068 +4008112367880337022_3680_000_3700_000,1569854704825168 +4008112367880337022_3680_000_3700_000,1569854707325043 +4008112367880337022_3680_000_3700_000,1569854707824970 +4008112367880337022_3680_000_3700_000,1569854715325243 +4008112367880337022_3680_000_3700_000,1569854715825244 +4008112367880337022_3680_000_3700_000,1569854703825152 +4008112367880337022_3680_000_3700_000,1569854713825019 +4008112367880337022_3680_000_3700_000,1569854703325067 +3275806206237593341_1260_000_1280_000,1557544942819254 +3275806206237593341_1260_000_1280_000,1557544950297870 +3275806206237593341_1260_000_1280_000,1557544951297442 +3275806206237593341_1260_000_1280_000,1557544951797369 +3275806206237593341_1260_000_1280_000,1557544950797707 +3275806206237593341_1260_000_1280_000,1557544952297300 +3275806206237593341_1260_000_1280_000,1557544949299141 +3275806206237593341_1260_000_1280_000,1557544940320359 +3275806206237593341_1260_000_1280_000,1557544940820248 +3275806206237593341_1260_000_1280_000,1557544942319553 +3275806206237593341_1260_000_1280_000,1557544941320001 +3275806206237593341_1260_000_1280_000,1557544949798358 +3275806206237593341_1260_000_1280_000,1557544939320505 +3275806206237593341_1260_000_1280_000,1557544953797222 +3275806206237593341_1260_000_1280_000,1557544953297262 +3275806206237593341_1260_000_1280_000,1557544943318943 +3275806206237593341_1260_000_1280_000,1557544941819785 +3275806206237593341_1260_000_1280_000,1557544943818501 +3275806206237593341_1260_000_1280_000,1557544939820502 +3275806206237593341_1260_000_1280_000,1557544952797231 +16942495693882305487_4340_000_4360_000,1559179844137784 +16942495693882305487_4340_000_4360_000,1559179844637716 +16942495693882305487_4340_000_4360_000,1559179846637950 +16942495693882305487_4340_000_4360_000,1559179855137769 +16942495693882305487_4340_000_4360_000,1559179854137701 +16942495693882305487_4340_000_4360_000,1559179846137883 +16942495693882305487_4340_000_4360_000,1559179845637785 +16942495693882305487_4340_000_4360_000,1559179857137780 +16942495693882305487_4340_000_4360_000,1559179848137768 +16942495693882305487_4340_000_4360_000,1559179847637805 +16942495693882305487_4340_000_4360_000,1559179848637749 +16942495693882305487_4340_000_4360_000,1559179855637782 +16942495693882305487_4340_000_4360_000,1559179845137739 +16942495693882305487_4340_000_4360_000,1559179858137740 +16942495693882305487_4340_000_4360_000,1559179856637781 +16942495693882305487_4340_000_4360_000,1559179854637737 +16942495693882305487_4340_000_4360_000,1559179857637814 +16942495693882305487_4340_000_4360_000,1559179856137797 +16942495693882305487_4340_000_4360_000,1559179858637797 +16942495693882305487_4340_000_4360_000,1559179847137875 +5764319372514665214_2480_000_2500_000,1558034992472993 +5764319372514665214_2480_000_2500_000,1558034983472954 +5764319372514665214_2480_000_2500_000,1558034982972924 +5764319372514665214_2480_000_2500_000,1558034989972975 +5764319372514665214_2480_000_2500_000,1558034981473075 +5764319372514665214_2480_000_2500_000,1558034990472969 +5764319372514665214_2480_000_2500_000,1558034984472951 +5764319372514665214_2480_000_2500_000,1558034991472965 +5764319372514665214_2480_000_2500_000,1558034980973024 +5764319372514665214_2480_000_2500_000,1558034979972956 +5764319372514665214_2480_000_2500_000,1558034981973026 +5764319372514665214_2480_000_2500_000,1558034991973002 +5764319372514665214_2480_000_2500_000,1558034990972960 +5764319372514665214_2480_000_2500_000,1558034993973011 +5764319372514665214_2480_000_2500_000,1558034982472951 +5764319372514665214_2480_000_2500_000,1558034983972951 +5764319372514665214_2480_000_2500_000,1558034993473006 +5764319372514665214_2480_000_2500_000,1558034980472954 +5764319372514665214_2480_000_2500_000,1558034994473066 +5764319372514665214_2480_000_2500_000,1558034992972995 +3485136235103477552_600_000_620_000,1559312920037900 +3485136235103477552_600_000_620_000,1559312918536992 +3485136235103477552_600_000_620_000,1559312929037490 +3485136235103477552_600_000_620_000,1559312931537400 +3485136235103477552_600_000_620_000,1559312921537438 +3485136235103477552_600_000_620_000,1559312917537421 +3485136235103477552_600_000_620_000,1559312927536888 +3485136235103477552_600_000_620_000,1559312921037521 +3485136235103477552_600_000_620_000,1559312919537665 +3485136235103477552_600_000_620_000,1559312928037154 +3485136235103477552_600_000_620_000,1559312930537328 +3485136235103477552_600_000_620_000,1559312917037757 +3485136235103477552_600_000_620_000,1559312930037396 +3485136235103477552_600_000_620_000,1559312918037188 +3485136235103477552_600_000_620_000,1559312929537548 +3485136235103477552_600_000_620_000,1559312927037001 +3485136235103477552_600_000_620_000,1559312928537375 +3485136235103477552_600_000_620_000,1559312931037329 +3485136235103477552_600_000_620_000,1559312919037170 +3485136235103477552_600_000_620_000,1559312920537711 +13732041959462600641_720_000_740_000,1558742853976814 +13732041959462600641_720_000_740_000,1558742855976028 +13732041959462600641_720_000_740_000,1558742843475326 +13732041959462600641_720_000_740_000,1558742854976703 +13732041959462600641_720_000_740_000,1558742843975547 +13732041959462600641_720_000_740_000,1558742846475978 +13732041959462600641_720_000_740_000,1558742844975697 +13732041959462600641_720_000_740_000,1558742856975912 +13732041959462600641_720_000_740_000,1558742855476179 +13732041959462600641_720_000_740_000,1558742842975141 +13732041959462600641_720_000_740_000,1558742847476056 +13732041959462600641_720_000_740_000,1558742857475609 +13732041959462600641_720_000_740_000,1558742844475636 +13732041959462600641_720_000_740_000,1558742845475848 +13732041959462600641_720_000_740_000,1558742845975911 +13732041959462600641_720_000_740_000,1558742846976015 +13732041959462600641_720_000_740_000,1558742854477097 +13732041959462600641_720_000_740_000,1558742852976440 +13732041959462600641_720_000_740_000,1558742853476695 +13732041959462600641_720_000_740_000,1558742856476018 +8684065200957554260_2700_000_2720_000,1566246362851376 +8684065200957554260_2700_000_2720_000,1566246374351315 +8684065200957554260_2700_000_2720_000,1566246373851362 +8684065200957554260_2700_000_2720_000,1566246372351287 +8684065200957554260_2700_000_2720_000,1566246363351451 +8684065200957554260_2700_000_2720_000,1566246362351295 +8684065200957554260_2700_000_2720_000,1566246363851429 +8684065200957554260_2700_000_2720_000,1566246366351318 +8684065200957554260_2700_000_2720_000,1566246375351264 +8684065200957554260_2700_000_2720_000,1566246373351328 +8684065200957554260_2700_000_2720_000,1566246376351894 +8684065200957554260_2700_000_2720_000,1566246376852628 +8684065200957554260_2700_000_2720_000,1566246364851337 +8684065200957554260_2700_000_2720_000,1566246375851419 +8684065200957554260_2700_000_2720_000,1566246365351325 +8684065200957554260_2700_000_2720_000,1566246366851318 +8684065200957554260_2700_000_2720_000,1566246365851320 +8684065200957554260_2700_000_2720_000,1566246364351329 +8684065200957554260_2700_000_2720_000,1566246372851306 +8684065200957554260_2700_000_2720_000,1566246374851263 +10410418118434245359_5140_000_5160_000,1557326223047734 +10410418118434245359_5140_000_5160_000,1557326221547648 +10410418118434245359_5140_000_5160_000,1557326223547764 +10410418118434245359_5140_000_5160_000,1557326209047560 +10410418118434245359_5140_000_5160_000,1557326213047602 +10410418118434245359_5140_000_5160_000,1557326212047572 +10410418118434245359_5140_000_5160_000,1557326221047770 +10410418118434245359_5140_000_5160_000,1557326211047663 +10410418118434245359_5140_000_5160_000,1557326211547653 +10410418118434245359_5140_000_5160_000,1557326220547772 +10410418118434245359_5140_000_5160_000,1557326212547575 +10410418118434245359_5140_000_5160_000,1557326209547585 +10410418118434245359_5140_000_5160_000,1557326210047617 +10410418118434245359_5140_000_5160_000,1557326220047729 +10410418118434245359_5140_000_5160_000,1557326222047648 +10410418118434245359_5140_000_5160_000,1557326222547699 +10410418118434245359_5140_000_5160_000,1557326219047730 +10410418118434245359_5140_000_5160_000,1557326219547770 +10410418118434245359_5140_000_5160_000,1557326210547626 +10410418118434245359_5140_000_5160_000,1557326213547578 +7240042450405902042_580_000_600_000,1559312901037775 +7240042450405902042_580_000_600_000,1559312897037515 +7240042450405902042_580_000_600_000,1559312899537484 +7240042450405902042_580_000_600_000,1559312898537394 +7240042450405902042_580_000_600_000,1559312911537589 +7240042450405902042_580_000_600_000,1559312900037413 +7240042450405902042_580_000_600_000,1559312907037317 +7240042450405902042_580_000_600_000,1559312901538082 +7240042450405902042_580_000_600_000,1559312909537272 +7240042450405902042_580_000_600_000,1559312908537793 +7240042450405902042_580_000_600_000,1559312899037443 +7240042450405902042_580_000_600_000,1559312910036813 +7240042450405902042_580_000_600_000,1559312910537019 +7240042450405902042_580_000_600_000,1559312908037618 +7240042450405902042_580_000_600_000,1559312909037663 +7240042450405902042_580_000_600_000,1559312911037369 +7240042450405902042_580_000_600_000,1559312898037440 +7240042450405902042_580_000_600_000,1559312900537375 +7240042450405902042_580_000_600_000,1559312897537487 +7240042450405902042_580_000_600_000,1559312907537791 +5585555620508986875_720_000_740_000,1559313037538117 +5585555620508986875_720_000_740_000,1559313050537687 +5585555620508986875_720_000_740_000,1559313047537497 +5585555620508986875_720_000_740_000,1559313048037350 +5585555620508986875_720_000_740_000,1559313040037581 +5585555620508986875_720_000_740_000,1559313039037173 +5585555620508986875_720_000_740_000,1559313038037778 +5585555620508986875_720_000_740_000,1559313051537445 +5585555620508986875_720_000_740_000,1559313040537431 +5585555620508986875_720_000_740_000,1559313047037528 +5585555620508986875_720_000_740_000,1559313049537681 +5585555620508986875_720_000_740_000,1559313048537310 +5585555620508986875_720_000_740_000,1559313041537128 +5585555620508986875_720_000_740_000,1559313049037464 +5585555620508986875_720_000_740_000,1559313037038225 +5585555620508986875_720_000_740_000,1559313041037197 +5585555620508986875_720_000_740_000,1559313051037544 +5585555620508986875_720_000_740_000,1559313050037678 +5585555620508986875_720_000_740_000,1559313038537390 +5585555620508986875_720_000_740_000,1559313039537279 +2714318267497393311_480_000_500_000,1558150298237122 +2714318267497393311_480_000_500_000,1558150287237469 +2714318267497393311_480_000_500_000,1558150290237709 +2714318267497393311_480_000_500_000,1558150296737452 +2714318267497393311_480_000_500_000,1558150287737484 +2714318267497393311_480_000_500_000,1558150299237252 +2714318267497393311_480_000_500_000,1558150288237628 +2714318267497393311_480_000_500_000,1558150300237538 +2714318267497393311_480_000_500_000,1558150297737232 +2714318267497393311_480_000_500_000,1558150289737802 +2714318267497393311_480_000_500_000,1558150290737726 +2714318267497393311_480_000_500_000,1558150296237346 +2714318267497393311_480_000_500_000,1558150297237200 +2714318267497393311_480_000_500_000,1558150288737667 +2714318267497393311_480_000_500_000,1558150286237588 +2714318267497393311_480_000_500_000,1558150289237769 +2714318267497393311_480_000_500_000,1558150286737552 +2714318267497393311_480_000_500_000,1558150298737101 +2714318267497393311_480_000_500_000,1558150299737398 +2714318267497393311_480_000_500_000,1558150300737648 +13790309965076620852_6520_000_6540_000,1574126957899706 +13790309965076620852_6520_000_6540_000,1574126959900038 +13790309965076620852_6520_000_6540_000,1574126955399851 +13790309965076620852_6520_000_6540_000,1574126968399982 +13790309965076620852_6520_000_6540_000,1574126965399821 +13790309965076620852_6520_000_6540_000,1574126958399589 +13790309965076620852_6520_000_6540_000,1574126957399943 +13790309965076620852_6520_000_6540_000,1574126967399978 +13790309965076620852_6520_000_6540_000,1574126958899663 +13790309965076620852_6520_000_6540_000,1574126956399869 +13790309965076620852_6520_000_6540_000,1574126966400006 +13790309965076620852_6520_000_6540_000,1574126956899942 +13790309965076620852_6520_000_6540_000,1574126968900008 +13790309965076620852_6520_000_6540_000,1574126966900090 +13790309965076620852_6520_000_6540_000,1574126959399883 +13790309965076620852_6520_000_6540_000,1574126965899849 +13790309965076620852_6520_000_6540_000,1574126967900033 +13790309965076620852_6520_000_6540_000,1574126955899899 +13790309965076620852_6520_000_6540_000,1574126969400087 +13790309965076620852_6520_000_6540_000,1574126969900021 +17387485694427326992_760_000_780_000,1557843958062722 +17387485694427326992_760_000_780_000,1557843968062691 +17387485694427326992_760_000_780_000,1557843968562687 +17387485694427326992_760_000_780_000,1557843959062736 +17387485694427326992_760_000_780_000,1557843967562765 +17387485694427326992_760_000_780_000,1557843956562821 +17387485694427326992_760_000_780_000,1557843955062802 +17387485694427326992_760_000_780_000,1557843965062813 +17387485694427326992_760_000_780_000,1557843969062758 +17387485694427326992_760_000_780_000,1557843969562794 +17387485694427326992_760_000_780_000,1557843966062703 +17387485694427326992_760_000_780_000,1557843967062734 +17387485694427326992_760_000_780_000,1557843965562735 +17387485694427326992_760_000_780_000,1557843959562659 +17387485694427326992_760_000_780_000,1557843957062778 +17387485694427326992_760_000_780_000,1557843957562803 +17387485694427326992_760_000_780_000,1557843966562710 +17387485694427326992_760_000_780_000,1557843956062840 +17387485694427326992_760_000_780_000,1557843958562737 +17387485694427326992_760_000_780_000,1557843955562873 +9350911198443552989_680_000_700_000,1557363451237372 +9350911198443552989_680_000_700_000,1557363441737465 +9350911198443552989_680_000_700_000,1557363449237250 +9350911198443552989_680_000_700_000,1557363439737622 +9350911198443552989_680_000_700_000,1557363438237327 +9350911198443552989_680_000_700_000,1557363440237403 +9350911198443552989_680_000_700_000,1557363441237340 +9350911198443552989_680_000_700_000,1557363447237793 +9350911198443552989_680_000_700_000,1557363451737437 +9350911198443552989_680_000_700_000,1557363449737386 +9350911198443552989_680_000_700_000,1557363437237375 +9350911198443552989_680_000_700_000,1557363437737418 +9350911198443552989_680_000_700_000,1557363440737261 +9350911198443552989_680_000_700_000,1557363448737285 +9350911198443552989_680_000_700_000,1557363439237622 +9350911198443552989_680_000_700_000,1557363447737794 +9350911198443552989_680_000_700_000,1557363438737444 +9350911198443552989_680_000_700_000,1557363450237422 +9350911198443552989_680_000_700_000,1557363450737354 +9350911198443552989_680_000_700_000,1557363448237517 +6174376739759381004_3240_000_3260_000,1557877015199162 +6174376739759381004_3240_000_3260_000,1557877006199178 +6174376739759381004_3240_000_3260_000,1557877004199181 +6174376739759381004_3240_000_3260_000,1557877005699128 +6174376739759381004_3240_000_3260_000,1557877008199313 +6174376739759381004_3240_000_3260_000,1557877016199192 +6174376739759381004_3240_000_3260_000,1557877014699134 +6174376739759381004_3240_000_3260_000,1557877007699341 +6174376739759381004_3240_000_3260_000,1557877017199143 +6174376739759381004_3240_000_3260_000,1557877014199207 +6174376739759381004_3240_000_3260_000,1557877016699133 +6174376739759381004_3240_000_3260_000,1557877004699166 +6174376739759381004_3240_000_3260_000,1557877018699207 +6174376739759381004_3240_000_3260_000,1557877015699193 +6174376739759381004_3240_000_3260_000,1557877008699136 +6174376739759381004_3240_000_3260_000,1557877005199071 +6174376739759381004_3240_000_3260_000,1557877018199234 +6174376739759381004_3240_000_3260_000,1557877007199256 +6174376739759381004_3240_000_3260_000,1557877006699224 +6174376739759381004_3240_000_3260_000,1557877017699172 +12153647356523920032_2560_000_2580_000,1572710128774788 +12153647356523920032_2560_000_2580_000,1572710126774725 +12153647356523920032_2560_000_2580_000,1572710120774792 +12153647356523920032_2560_000_2580_000,1572710129274823 +12153647356523920032_2560_000_2580_000,1572710116774778 +12153647356523920032_2560_000_2580_000,1572710119774754 +12153647356523920032_2560_000_2580_000,1572710117774722 +12153647356523920032_2560_000_2580_000,1572710130274767 +12153647356523920032_2560_000_2580_000,1572710128274786 +12153647356523920032_2560_000_2580_000,1572710120274760 +12153647356523920032_2560_000_2580_000,1572710130774726 +12153647356523920032_2560_000_2580_000,1572710118274683 +12153647356523920032_2560_000_2580_000,1572710127274765 +12153647356523920032_2560_000_2580_000,1572710127774811 +12153647356523920032_2560_000_2580_000,1572710126274748 +12153647356523920032_2560_000_2580_000,1572710118774754 +12153647356523920032_2560_000_2580_000,1572710117274754 +12153647356523920032_2560_000_2580_000,1572710129774796 +12153647356523920032_2560_000_2580_000,1572710119274760 +12153647356523920032_2560_000_2580_000,1572710116274782 +11933765568165455008_2940_000_2960_000,1557198335387209 +11933765568165455008_2940_000_2960_000,1557198338387310 +11933765568165455008_2940_000_2960_000,1557198347387352 +11933765568165455008_2940_000_2960_000,1557198338887301 +11933765568165455008_2940_000_2960_000,1557198348387315 +11933765568165455008_2940_000_2960_000,1557198345387416 +11933765568165455008_2940_000_2960_000,1557198335887178 +11933765568165455008_2940_000_2960_000,1557198344387408 +11933765568165455008_2940_000_2960_000,1557198344887332 +11933765568165455008_2940_000_2960_000,1557198337387225 +11933765568165455008_2940_000_2960_000,1557198345887369 +11933765568165455008_2940_000_2960_000,1557198347887352 +11933765568165455008_2940_000_2960_000,1557198346887349 +11933765568165455008_2940_000_2960_000,1557198336387249 +11933765568165455008_2940_000_2960_000,1557198348887399 +11933765568165455008_2940_000_2960_000,1557198334887218 +11933765568165455008_2940_000_2960_000,1557198334387221 +11933765568165455008_2940_000_2960_000,1557198337887303 +11933765568165455008_2940_000_2960_000,1557198336887239 +11933765568165455008_2940_000_2960_000,1557198346387373 +10161761842905385678_760_000_780_000,1557196157797448 +10161761842905385678_760_000_780_000,1557196158797350 +10161761842905385678_760_000_780_000,1557196168297765 +10161761842905385678_760_000_780_000,1557196155797333 +10161761842905385678_760_000_780_000,1557196167797613 +10161761842905385678_760_000_780_000,1557196166297587 +10161761842905385678_760_000_780_000,1557196156797479 +10161761842905385678_760_000_780_000,1557196167297622 +10161761842905385678_760_000_780_000,1557196154797258 +10161761842905385678_760_000_780_000,1557196154297327 +10161761842905385678_760_000_780_000,1557196165297463 +10161761842905385678_760_000_780_000,1557196165797474 +10161761842905385678_760_000_780_000,1557196156297413 +10161761842905385678_760_000_780_000,1557196164297460 +10161761842905385678_760_000_780_000,1557196158297419 +10161761842905385678_760_000_780_000,1557196168797617 +10161761842905385678_760_000_780_000,1557196166797651 +10161761842905385678_760_000_780_000,1557196155297293 +10161761842905385678_760_000_780_000,1557196164797422 +10161761842905385678_760_000_780_000,1557196157297472 +6922883602463663456_2220_000_2240_000,1558152040772834 +6922883602463663456_2220_000_2240_000,1558152026756411 +6922883602463663456_2220_000_2240_000,1558152028764982 +6922883602463663456_2220_000_2240_000,1558152036772202 +6922883602463663456_2220_000_2240_000,1558152029768625 +6922883602463663456_2220_000_2240_000,1558152037272084 +6922883602463663456_2220_000_2240_000,1558152038772394 +6922883602463663456_2220_000_2240_000,1558152036272158 +6922883602463663456_2220_000_2240_000,1558152030771388 +6922883602463663456_2220_000_2240_000,1558152038272239 +6922883602463663456_2220_000_2240_000,1558152040272803 +6922883602463663456_2220_000_2240_000,1558152030270137 +6922883602463663456_2220_000_2240_000,1558152037772191 +6922883602463663456_2220_000_2240_000,1558152027760810 +6922883602463663456_2220_000_2240_000,1558152027258557 +6922883602463663456_2220_000_2240_000,1558152026254441 +6922883602463663456_2220_000_2240_000,1558152039272550 +6922883602463663456_2220_000_2240_000,1558152039772680 +6922883602463663456_2220_000_2240_000,1558152029266975 +6922883602463663456_2220_000_2240_000,1558152028262942 +3341890853207909601_1020_000_1040_000,1573927803625099 +3341890853207909601_1020_000_1040_000,1573927790125189 +3341890853207909601_1020_000_1040_000,1573927802125062 +3341890853207909601_1020_000_1040_000,1573927801625067 +3341890853207909601_1020_000_1040_000,1573927794625079 +3341890853207909601_1020_000_1040_000,1573927790625242 +3341890853207909601_1020_000_1040_000,1573927792624930 +3341890853207909601_1020_000_1040_000,1573927791125208 +3341890853207909601_1020_000_1040_000,1573927800624954 +3341890853207909601_1020_000_1040_000,1573927804625096 +3341890853207909601_1020_000_1040_000,1573927800124914 +3341890853207909601_1020_000_1040_000,1573927802625074 +3341890853207909601_1020_000_1040_000,1573927792124827 +3341890853207909601_1020_000_1040_000,1573927794125084 +3341890853207909601_1020_000_1040_000,1573927801125097 +3341890853207909601_1020_000_1040_000,1573927793624995 +3341890853207909601_1020_000_1040_000,1573927793124963 +3341890853207909601_1020_000_1040_000,1573927804125097 +3341890853207909601_1020_000_1040_000,1573927803125097 +3341890853207909601_1020_000_1040_000,1573927791625026 +17756183617755834457_1940_000_1960_000,1558017204447293 +17756183617755834457_1940_000_1960_000,1558017214436996 +17756183617755834457_1940_000_1960_000,1558017215429120 +17756183617755834457_1940_000_1960_000,1558017206446333 +17756183617755834457_1940_000_1960_000,1558017207446078 +17756183617755834457_1940_000_1960_000,1558017218421930 +17756183617755834457_1940_000_1960_000,1558017213940930 +17756183617755834457_1940_000_1960_000,1558017217922014 +17756183617755834457_1940_000_1960_000,1558017206945999 +17756183617755834457_1940_000_1960_000,1558017205447104 +17756183617755834457_1940_000_1960_000,1558017214932926 +17756183617755834457_1940_000_1960_000,1558017217422255 +17756183617755834457_1940_000_1960_000,1558017215925793 +17756183617755834457_1940_000_1960_000,1558017208447290 +17756183617755834457_1940_000_1960_000,1558017216423608 +17756183617755834457_1940_000_1960_000,1558017207946577 +17756183617755834457_1940_000_1960_000,1558017216922725 +17756183617755834457_1940_000_1960_000,1558017204947246 +17756183617755834457_1940_000_1960_000,1558017205946707 +17756183617755834457_1940_000_1960_000,1558017203947410 +2218963221891181906_4360_000_4380_000,1573932454073919 +2218963221891181906_4360_000_4380_000,1573932458574312 +2218963221891181906_4360_000_4380_000,1573932456574286 +2218963221891181906_4360_000_4380_000,1573932445149910 +2218963221891181906_4360_000_4380_000,1573932457574335 +2218963221891181906_4360_000_4380_000,1573932444649899 +2218963221891181906_4360_000_4380_000,1573932446649928 +2218963221891181906_4360_000_4380_000,1573932445649884 +2218963221891181906_4360_000_4380_000,1573932448150256 +2218963221891181906_4360_000_4380_000,1573932444149933 +2218963221891181906_4360_000_4380_000,1573932447149977 +2218963221891181906_4360_000_4380_000,1573932454574319 +2218963221891181906_4360_000_4380_000,1573932456074299 +2218963221891181906_4360_000_4380_000,1573932455574265 +2218963221891181906_4360_000_4380_000,1573932457074331 +2218963221891181906_4360_000_4380_000,1573932458074340 +2218963221891181906_4360_000_4380_000,1573932448650899 +2218963221891181906_4360_000_4380_000,1573932446149941 +2218963221891181906_4360_000_4380_000,1573932447650058 +2218963221891181906_4360_000_4380_000,1573932455074331 +10149575340910243572_2720_000_2740_000,1558035231962663 +10149575340910243572_2720_000_2740_000,1558035232462596 +10149575340910243572_2720_000_2740_000,1558035234462274 +10149575340910243572_2720_000_2740_000,1558035232962512 +10149575340910243572_2720_000_2740_000,1558035233462396 +10149575340910243572_2720_000_2740_000,1558035230462351 +10149575340910243572_2720_000_2740_000,1558035231462594 +10149575340910243572_2720_000_2740_000,1558035230962448 +10149575340910243572_2720_000_2740_000,1558035229962648 +10149575340910243572_2720_000_2740_000,1558035233962327 +3459095437766396887_1600_000_1620_000,1559177116218096 +3459095437766396887_1600_000_1620_000,1559177116717458 +3459095437766396887_1600_000_1620_000,1559177108222874 +3459095437766396887_1600_000_1620_000,1559177115219166 +3459095437766396887_1600_000_1620_000,1559177117217000 +3459095437766396887_1600_000_1620_000,1559177114719614 +3459095437766396887_1600_000_1620_000,1559177115718671 +3459095437766396887_1600_000_1620_000,1559177105721360 +3459095437766396887_1600_000_1620_000,1559177108722993 +3459095437766396887_1600_000_1620_000,1559177107221934 +3459095437766396887_1600_000_1620_000,1559177106221852 +3459095437766396887_1600_000_1620_000,1559177114219949 +3459095437766396887_1600_000_1620_000,1559177105220562 +3459095437766396887_1600_000_1620_000,1559177107722383 +3459095437766396887_1600_000_1620_000,1559177118216369 +3459095437766396887_1600_000_1620_000,1559177117716745 +3459095437766396887_1600_000_1620_000,1559177104218831 +3459095437766396887_1600_000_1620_000,1559177104719526 +3459095437766396887_1600_000_1620_000,1559177118715883 +3459095437766396887_1600_000_1620_000,1559177106721948 +8249122135171526629_520_000_540_000,1559184839587788 +8249122135171526629_520_000_540_000,1559184839087463 +8249122135171526629_520_000_540_000,1559184838086814 +8249122135171526629_520_000_540_000,1559184829585106 +8249122135171526629_520_000_540_000,1559184829085741 +8249122135171526629_520_000_540_000,1559184841587404 +8249122135171526629_520_000_540_000,1559184832087286 +8249122135171526629_520_000_540_000,1559184831086036 +8249122135171526629_520_000_540_000,1559184830585419 +8249122135171526629_520_000_540_000,1559184838587000 +8249122135171526629_520_000_540_000,1559184842087749 +8249122135171526629_520_000_540_000,1559184827587385 +8249122135171526629_520_000_540_000,1559184828087141 +8249122135171526629_520_000_540_000,1559184837586942 +8249122135171526629_520_000_540_000,1559184840587321 +8249122135171526629_520_000_540_000,1559184830085083 +8249122135171526629_520_000_540_000,1559184828586572 +8249122135171526629_520_000_540_000,1559184841087135 +8249122135171526629_520_000_540_000,1559184840087626 +8249122135171526629_520_000_540_000,1559184831586778 +1664548685643064400_2240_000_2260_000,1572730660024796 +1664548685643064400_2240_000_2260_000,1572730661524793 +1664548685643064400_2240_000_2260_000,1572730664024893 +1664548685643064400_2240_000_2260_000,1572730661024763 +1664548685643064400_2240_000_2260_000,1572730659524712 +1664548685643064400_2240_000_2260_000,1572730651024914 +1664548685643064400_2240_000_2260_000,1572730652024805 +1664548685643064400_2240_000_2260_000,1572730663524712 +1664548685643064400_2240_000_2260_000,1572730662524607 +1664548685643064400_2240_000_2260_000,1572730654024948 +1664548685643064400_2240_000_2260_000,1572730660524763 +1664548685643064400_2240_000_2260_000,1572730649525094 +1664548685643064400_2240_000_2260_000,1572730651524841 +1664548685643064400_2240_000_2260_000,1572730653024965 +1664548685643064400_2240_000_2260_000,1572730662024682 +1664548685643064400_2240_000_2260_000,1572730652524780 +1664548685643064400_2240_000_2260_000,1572730650524867 +1664548685643064400_2240_000_2260_000,1572730663024572 +1664548685643064400_2240_000_2260_000,1572730650024950 +1664548685643064400_2240_000_2260_000,1572730653525063 +4916600861562283346_3880_000_3900_000,1559179394137429 +4916600861562283346_3880_000_3900_000,1559179396137504 +4916600861562283346_3880_000_3900_000,1559179396637496 +4916600861562283346_3880_000_3900_000,1559179398137489 +4916600861562283346_3880_000_3900_000,1559179388637375 +4916600861562283346_3880_000_3900_000,1559179398637508 +4916600861562283346_3880_000_3900_000,1559179386637413 +4916600861562283346_3880_000_3900_000,1559179386137493 +4916600861562283346_3880_000_3900_000,1559179397137450 +4916600861562283346_3880_000_3900_000,1559179387637365 +4916600861562283346_3880_000_3900_000,1559179384137390 +4916600861562283346_3880_000_3900_000,1559179387137336 +4916600861562283346_3880_000_3900_000,1559179384637499 +4916600861562283346_3880_000_3900_000,1559179388137403 +4916600861562283346_3880_000_3900_000,1559179397637459 +4916600861562283346_3880_000_3900_000,1559179395137442 +4916600861562283346_3880_000_3900_000,1559179385137537 +4916600861562283346_3880_000_3900_000,1559179385637530 +4916600861562283346_3880_000_3900_000,1559179395637456 +4916600861562283346_3880_000_3900_000,1559179394637383 +10802932587105534078_1280_000_1300_000,1557888796948097 +10802932587105534078_1280_000_1300_000,1557888798448099 +10802932587105534078_1280_000_1300_000,1557888806449251 +10802932587105534078_1280_000_1300_000,1557888809449360 +10802932587105534078_1280_000_1300_000,1557888810448859 +10802932587105534078_1280_000_1300_000,1557888800447985 +10802932587105534078_1280_000_1300_000,1557888807948674 +10802932587105534078_1280_000_1300_000,1557888809949023 +10802932587105534078_1280_000_1300_000,1557888810949122 +10802932587105534078_1280_000_1300_000,1557888799948216 +10802932587105534078_1280_000_1300_000,1557888798948041 +10802932587105534078_1280_000_1300_000,1557888800948126 +10802932587105534078_1280_000_1300_000,1557888806949187 +10802932587105534078_1280_000_1300_000,1557888807448803 +10802932587105534078_1280_000_1300_000,1557888799448247 +10802932587105534078_1280_000_1300_000,1557888808449065 +10802932587105534078_1280_000_1300_000,1557888797948166 +10802932587105534078_1280_000_1300_000,1557888796448121 +10802932587105534078_1280_000_1300_000,1557888808949531 +10802932587105534078_1280_000_1300_000,1557888797448185 +13748565785898537200_680_000_700_000,1573621439474829 +13748565785898537200_680_000_700_000,1573621439974767 +13748565785898537200_680_000_700_000,1573621429474915 +13748565785898537200_680_000_700_000,1573621429974924 +13748565785898537200_680_000_700_000,1573621440974804 +13748565785898537200_680_000_700_000,1573621441474863 +13748565785898537200_680_000_700_000,1573621443974860 +13748565785898537200_680_000_700_000,1573621431474829 +13748565785898537200_680_000_700_000,1573621441974787 +13748565785898537200_680_000_700_000,1573621432474859 +13748565785898537200_680_000_700_000,1573621443474808 +13748565785898537200_680_000_700_000,1573621430974792 +13748565785898537200_680_000_700_000,1573621433974860 +13748565785898537200_680_000_700_000,1573621431974875 +13748565785898537200_680_000_700_000,1573621442974839 +13748565785898537200_680_000_700_000,1573621430474807 +13748565785898537200_680_000_700_000,1573621442474772 +13748565785898537200_680_000_700_000,1573621440474758 +13748565785898537200_680_000_700_000,1573621433474826 +13748565785898537200_680_000_700_000,1573621432974900 +14643284977980826278_520_000_540_000,1558150336737516 +14643284977980826278_520_000_540_000,1558150328237510 +14643284977980826278_520_000_540_000,1558150337237361 +14643284977980826278_520_000_540_000,1558150327737447 +14643284977980826278_520_000_540_000,1558150327237391 +14643284977980826278_520_000_540_000,1558150339737419 +14643284977980826278_520_000_540_000,1558150326737588 +14643284977980826278_520_000_540_000,1558150326237829 +14643284977980826278_520_000_540_000,1558150330737609 +14643284977980826278_520_000_540_000,1558150329237498 +14643284977980826278_520_000_540_000,1558150330237732 +14643284977980826278_520_000_540_000,1558150339237427 +14643284977980826278_520_000_540_000,1558150340237494 +14643284977980826278_520_000_540_000,1558150340737501 +14643284977980826278_520_000_540_000,1558150329737609 +14643284977980826278_520_000_540_000,1558150328737468 +14643284977980826278_520_000_540_000,1558150337737326 +14643284977980826278_520_000_540_000,1558150338237396 +14643284977980826278_520_000_540_000,1558150336237586 +14643284977980826278_520_000_540_000,1558150338737431 +4045613324047897473_940_000_960_000,1558493338074162 +4045613324047897473_940_000_960_000,1558493348073783 +4045613324047897473_940_000_960_000,1558493350074053 +4045613324047897473_940_000_960_000,1558493345573992 +4045613324047897473_940_000_960_000,1558493347574007 +4045613324047897473_940_000_960_000,1558493338574044 +4045613324047897473_940_000_960_000,1558493335574296 +4045613324047897473_940_000_960_000,1558493339573912 +4045613324047897473_940_000_960_000,1558493336574269 +4045613324047897473_940_000_960_000,1558493347074035 +4045613324047897473_940_000_960_000,1558493346574102 +4045613324047897473_940_000_960_000,1558493346073989 +4045613324047897473_940_000_960_000,1558493337574148 +4045613324047897473_940_000_960_000,1558493348573778 +4045613324047897473_940_000_960_000,1558493349074012 +4045613324047897473_940_000_960_000,1558493337074219 +4045613324047897473_940_000_960_000,1558493349574122 +4045613324047897473_940_000_960_000,1558493340074053 +4045613324047897473_940_000_960_000,1558493336074290 +4045613324047897473_940_000_960_000,1558493339073948 +2257381802419655779_820_000_840_000,1558402111847622 +2257381802419655779_820_000_840_000,1558402122847222 +2257381802419655779_820_000_840_000,1558402108847992 +2257381802419655779_820_000_840_000,1558402118847287 +2257381802419655779_820_000_840_000,1558402120847365 +2257381802419655779_820_000_840_000,1558402110847064 +2257381802419655779_820_000_840_000,1558402119847426 +2257381802419655779_820_000_840_000,1558402122347099 +2257381802419655779_820_000_840_000,1558402121847019 +2257381802419655779_820_000_840_000,1558402121347177 +2257381802419655779_820_000_840_000,1558402109847716 +2257381802419655779_820_000_840_000,1558402112347811 +2257381802419655779_820_000_840_000,1558402123347308 +2257381802419655779_820_000_840_000,1558402112847819 +2257381802419655779_820_000_840_000,1558402109347833 +2257381802419655779_820_000_840_000,1558402120347479 +2257381802419655779_820_000_840_000,1558402111347219 +2257381802419655779_820_000_840_000,1558402110347368 +2257381802419655779_820_000_840_000,1558402119347368 +2257381802419655779_820_000_840_000,1558402113347613 +4054036670499089296_2300_000_2320_000,1557187714649115 +4054036670499089296_2300_000_2320_000,1557187716649135 +4054036670499089296_2300_000_2320_000,1557187704649276 +4054036670499089296_2300_000_2320_000,1557187707149136 +4054036670499089296_2300_000_2320_000,1557187716149170 +4054036670499089296_2300_000_2320_000,1557187704149193 +4054036670499089296_2300_000_2320_000,1557187717148945 +4054036670499089296_2300_000_2320_000,1557187707649076 +4054036670499089296_2300_000_2320_000,1557187706649119 +4054036670499089296_2300_000_2320_000,1557187705149208 +4054036670499089296_2300_000_2320_000,1557187715649133 +4054036670499089296_2300_000_2320_000,1557187713649046 +4054036670499089296_2300_000_2320_000,1557187706149101 +4054036670499089296_2300_000_2320_000,1557187715149153 +4054036670499089296_2300_000_2320_000,1557187703148999 +4054036670499089296_2300_000_2320_000,1557187703649173 +4054036670499089296_2300_000_2320_000,1557187713149076 +4054036670499089296_2300_000_2320_000,1557187714149098 +4054036670499089296_2300_000_2320_000,1557187717649252 +4054036670499089296_2300_000_2320_000,1557187705649134 +12056192874455954437_140_000_160_000,1557843345612667 +12056192874455954437_140_000_160_000,1557843349612578 +12056192874455954437_140_000_160_000,1557843345112543 +12056192874455954437_140_000_160_000,1557843335112508 +12056192874455954437_140_000_160_000,1557843338612551 +12056192874455954437_140_000_160_000,1557843336612494 +12056192874455954437_140_000_160_000,1557843338112693 +12056192874455954437_140_000_160_000,1557843337112658 +12056192874455954437_140_000_160_000,1557843339612639 +12056192874455954437_140_000_160_000,1557843348612302 +12056192874455954437_140_000_160_000,1557843335612429 +12056192874455954437_140_000_160_000,1557843336112396 +12056192874455954437_140_000_160_000,1557843349112419 +12056192874455954437_140_000_160_000,1557843337612796 +12056192874455954437_140_000_160_000,1557843346612497 +12056192874455954437_140_000_160_000,1557843347612615 +12056192874455954437_140_000_160_000,1557843348112448 +12056192874455954437_140_000_160_000,1557843346112603 +12056192874455954437_140_000_160_000,1557843339112601 +12056192874455954437_140_000_160_000,1557843347112468 +13034900465317073842_1700_000_1720_000,1559143078524545 +13034900465317073842_1700_000_1720_000,1559143065016062 +13034900465317073842_1700_000_1720_000,1559143064015948 +13034900465317073842_1700_000_1720_000,1559143074021060 +13034900465317073842_1700_000_1720_000,1559143068016067 +13034900465317073842_1700_000_1720_000,1559143076523597 +13034900465317073842_1700_000_1720_000,1559143067016248 +13034900465317073842_1700_000_1720_000,1559143075522514 +13034900465317073842_1700_000_1720_000,1559143077023973 +13034900465317073842_1700_000_1720_000,1559143064515955 +13034900465317073842_1700_000_1720_000,1559143066516551 +13034900465317073842_1700_000_1720_000,1559143077524362 +13034900465317073842_1700_000_1720_000,1559143068516366 +13034900465317073842_1700_000_1720_000,1559143076023064 +13034900465317073842_1700_000_1720_000,1559143074521426 +13034900465317073842_1700_000_1720_000,1559143067516020 +13034900465317073842_1700_000_1720_000,1559143065516232 +13034900465317073842_1700_000_1720_000,1559143066016549 +13034900465317073842_1700_000_1720_000,1559143075021878 +13034900465317073842_1700_000_1720_000,1559143078024530 +7511993111693456743_3880_000_3900_000,1557963202297466 +7511993111693456743_3880_000_3900_000,1557963212297515 +7511993111693456743_3880_000_3900_000,1557963200297419 +7511993111693456743_3880_000_3900_000,1557963202797419 +7511993111693456743_3880_000_3900_000,1557963211297319 +7511993111693456743_3880_000_3900_000,1557963211797549 +7511993111693456743_3880_000_3900_000,1557963201297473 +7511993111693456743_3880_000_3900_000,1557963209797116 +7511993111693456743_3880_000_3900_000,1557963210297172 +7511993111693456743_3880_000_3900_000,1557963200797464 +7511993111693456743_3880_000_3900_000,1557963209297327 +7511993111693456743_3880_000_3900_000,1557963208797520 +7511993111693456743_3880_000_3900_000,1557963198797401 +7511993111693456743_3880_000_3900_000,1557963213297448 +7511993111693456743_3880_000_3900_000,1557963210797182 +7511993111693456743_3880_000_3900_000,1557963201797503 +7511993111693456743_3880_000_3900_000,1557963199297286 +7511993111693456743_3880_000_3900_000,1557963199797330 +7511993111693456743_3880_000_3900_000,1557963203297377 +7511993111693456743_3880_000_3900_000,1557963212797472 +9355489589631690177_4800_000_4820_000,1557342366562650 +9355489589631690177_4800_000_4820_000,1557342358062536 +9355489589631690177_4800_000_4820_000,1557342369562809 +9355489589631690177_4800_000_4820_000,1557342357562530 +9355489589631690177_4800_000_4820_000,1557342367062748 +9355489589631690177_4800_000_4820_000,1557342356562423 +9355489589631690177_4800_000_4820_000,1557342355562520 +9355489589631690177_4800_000_4820_000,1557342358562309 +9355489589631690177_4800_000_4820_000,1557342368562561 +9355489589631690177_4800_000_4820_000,1557342367562723 +9355489589631690177_4800_000_4820_000,1557342365562451 +9355489589631690177_4800_000_4820_000,1557342369062698 +9355489589631690177_4800_000_4820_000,1557342366062493 +9355489589631690177_4800_000_4820_000,1557342368062616 +9355489589631690177_4800_000_4820_000,1557342357062509 +9355489589631690177_4800_000_4820_000,1557342359062110 +9355489589631690177_4800_000_4820_000,1557342355062436 +9355489589631690177_4800_000_4820_000,1557342359562031 +9355489589631690177_4800_000_4820_000,1557342365062568 +9355489589631690177_4800_000_4820_000,1557342356062469 +3522804493060229409_3400_000_3420_000,1557855904472271 +3522804493060229409_3400_000_3420_000,1557855907472634 +3522804493060229409_3400_000_3420_000,1557855896472328 +3522804493060229409_3400_000_3420_000,1557855892972587 +3522804493060229409_3400_000_3420_000,1557855906972551 +3522804493060229409_3400_000_3420_000,1557855905472302 +3522804493060229409_3400_000_3420_000,1557855904972296 +3522804493060229409_3400_000_3420_000,1557855905972396 +3522804493060229409_3400_000_3420_000,1557855906472495 +3522804493060229409_3400_000_3420_000,1557855893972382 +3522804493060229409_3400_000_3420_000,1557855897472206 +3522804493060229409_3400_000_3420_000,1557855902972245 +3522804493060229409_3400_000_3420_000,1557855894972377 +3522804493060229409_3400_000_3420_000,1557855893472505 +3522804493060229409_3400_000_3420_000,1557855895472388 +3522804493060229409_3400_000_3420_000,1557855896972244 +3522804493060229409_3400_000_3420_000,1557855903472293 +3522804493060229409_3400_000_3420_000,1557855895972316 +3522804493060229409_3400_000_3420_000,1557855894472345 +3522804493060229409_3400_000_3420_000,1557855903972289 +8566480970798227989_500_000_520_000,1557239425612429 +8566480970798227989_500_000_520_000,1557239414112699 +8566480970798227989_500_000_520_000,1557239413112667 +8566480970798227989_500_000_520_000,1557239415112533 +8566480970798227989_500_000_520_000,1557239416612460 +8566480970798227989_500_000_520_000,1557239423112799 +8566480970798227989_500_000_520_000,1557239415612490 +8566480970798227989_500_000_520_000,1557239422112884 +8566480970798227989_500_000_520_000,1557239412612624 +8566480970798227989_500_000_520_000,1557239424612659 +8566480970798227989_500_000_520_000,1557239412112652 +8566480970798227989_500_000_520_000,1557239422612861 +8566480970798227989_500_000_520_000,1557239416112464 +8566480970798227989_500_000_520_000,1557239423612728 +8566480970798227989_500_000_520_000,1557239413612747 +8566480970798227989_500_000_520_000,1557239426112320 +8566480970798227989_500_000_520_000,1557239426612303 +8566480970798227989_500_000_520_000,1557239414612596 +8566480970798227989_500_000_520_000,1557239425112554 +8566480970798227989_500_000_520_000,1557239424112739 +6278307160249415497_1700_000_1720_000,1558034213921937 +6278307160249415497_1700_000_1720_000,1558034201922721 +6278307160249415497_1700_000_1720_000,1558034202422649 +6278307160249415497_1700_000_1720_000,1558034202922472 +6278307160249415497_1700_000_1720_000,1558034204422154 +6278307160249415497_1700_000_1720_000,1558034214422280 +6278307160249415497_1700_000_1720_000,1558034213421817 +6278307160249415497_1700_000_1720_000,1558034211421372 +6278307160249415497_1700_000_1720_000,1558034203922216 +6278307160249415497_1700_000_1720_000,1558034200922728 +6278307160249415497_1700_000_1720_000,1558034212921821 +6278307160249415497_1700_000_1720_000,1558034210421304 +6278307160249415497_1700_000_1720_000,1558034201422689 +6278307160249415497_1700_000_1720_000,1558034211921700 +6278307160249415497_1700_000_1720_000,1558034209921189 +6278307160249415497_1700_000_1720_000,1558034212421831 +6278307160249415497_1700_000_1720_000,1558034200422683 +6278307160249415497_1700_000_1720_000,1558034210921320 +6278307160249415497_1700_000_1720_000,1558034203422353 +6278307160249415497_1700_000_1720_000,1558034199922726 +13787943721654585343_1220_000_1240_000,1558483374422389 +13787943721654585343_1220_000_1240_000,1558483360422540 +13787943721654585343_1220_000_1240_000,1558483362922326 +13787943721654585343_1220_000_1240_000,1558483361422280 +13787943721654585343_1220_000_1240_000,1558483370422349 +13787943721654585343_1220_000_1240_000,1558483359922533 +13787943721654585343_1220_000_1240_000,1558483372922276 +13787943721654585343_1220_000_1240_000,1558483364422414 +13787943721654585343_1220_000_1240_000,1558483369922463 +13787943721654585343_1220_000_1240_000,1558483373422253 +13787943721654585343_1220_000_1240_000,1558483360922432 +13787943721654585343_1220_000_1240_000,1558483370922205 +13787943721654585343_1220_000_1240_000,1558483371922349 +13787943721654585343_1220_000_1240_000,1558483371422242 +13787943721654585343_1220_000_1240_000,1558483361922245 +13787943721654585343_1220_000_1240_000,1558483362422314 +13787943721654585343_1220_000_1240_000,1558483363422326 +13787943721654585343_1220_000_1240_000,1558483363922364 +13787943721654585343_1220_000_1240_000,1558483372422320 +13787943721654585343_1220_000_1240_000,1558483373922325 +10998289306141768318_1280_000_1300_000,1558483433397038 +10998289306141768318_1280_000_1300_000,1558483430411803 +10998289306141768318_1280_000_1300_000,1558483420422343 +10998289306141768318_1280_000_1300_000,1558483434396435 +10998289306141768318_1280_000_1300_000,1558483421422280 +10998289306141768318_1280_000_1300_000,1558483423422502 +10998289306141768318_1280_000_1300_000,1558483430908205 +10998289306141768318_1280_000_1300_000,1558483424422579 +10998289306141768318_1280_000_1300_000,1558483433896475 +10998289306141768318_1280_000_1300_000,1558483423922620 +10998289306141768318_1280_000_1300_000,1558483419922414 +10998289306141768318_1280_000_1300_000,1558483422422324 +10998289306141768318_1280_000_1300_000,1558483431404397 +10998289306141768318_1280_000_1300_000,1558483431901030 +10998289306141768318_1280_000_1300_000,1558483429915076 +10998289306141768318_1280_000_1300_000,1558483420922273 +10998289306141768318_1280_000_1300_000,1558483421922318 +10998289306141768318_1280_000_1300_000,1558483422922327 +10998289306141768318_1280_000_1300_000,1558483432398938 +10998289306141768318_1280_000_1300_000,1558483432897848 +7435516779413778621_4440_000_4460_000,1557325510087987 +7435516779413778621_4440_000_4460_000,1557325509088023 +7435516779413778621_4440_000_4460_000,1557325509588017 +7435516779413778621_4440_000_4460_000,1557325522112585 +7435516779413778621_4440_000_4460_000,1557325511088136 +7435516779413778621_4440_000_4460_000,1557325513590433 +7435516779413778621_4440_000_4460_000,1557325512588488 +7435516779413778621_4440_000_4460_000,1557325521112794 +7435516779413778621_4440_000_4460_000,1557325513089176 +7435516779413778621_4440_000_4460_000,1557325522612689 +7435516779413778621_4440_000_4460_000,1557325520112870 +7435516779413778621_4440_000_4460_000,1557325523612525 +7435516779413778621_4440_000_4460_000,1557325511588133 +7435516779413778621_4440_000_4460_000,1557325521612655 +7435516779413778621_4440_000_4460_000,1557325519113921 +7435516779413778621_4440_000_4460_000,1557325520612844 +7435516779413778621_4440_000_4460_000,1557325510588071 +7435516779413778621_4440_000_4460_000,1557325523112680 +7435516779413778621_4440_000_4460_000,1557325519613322 +7435516779413778621_4440_000_4460_000,1557325512088233 +13944616099709049906_1020_000_1040_000,1558493425524322 +13944616099709049906_1020_000_1040_000,1558493417024071 +13944616099709049906_1020_000_1040_000,1558493426024320 +13944616099709049906_1020_000_1040_000,1558493416024098 +13944616099709049906_1020_000_1040_000,1558493429524171 +13944616099709049906_1020_000_1040_000,1558493426524287 +13944616099709049906_1020_000_1040_000,1558493419024193 +13944616099709049906_1020_000_1040_000,1558493430024138 +13944616099709049906_1020_000_1040_000,1558493427524280 +13944616099709049906_1020_000_1040_000,1558493415524136 +13944616099709049906_1020_000_1040_000,1558493427024273 +13944616099709049906_1020_000_1040_000,1558493429024223 +13944616099709049906_1020_000_1040_000,1558493428524220 +13944616099709049906_1020_000_1040_000,1558493420024171 +13944616099709049906_1020_000_1040_000,1558493418024131 +13944616099709049906_1020_000_1040_000,1558493418524161 +13944616099709049906_1020_000_1040_000,1558493417524102 +13944616099709049906_1020_000_1040_000,1558493419524165 +13944616099709049906_1020_000_1040_000,1558493416524077 +13944616099709049906_1020_000_1040_000,1558493428024253 +8229317157758012712_3860_000_3880_000,1559179375137657 +8229317157758012712_3860_000_3880_000,1559179375637448 +8229317157758012712_3860_000_3880_000,1559179366637361 +8229317157758012712_3860_000_3880_000,1559179368137382 +8229317157758012712_3860_000_3880_000,1559179367137366 +8229317157758012712_3860_000_3880_000,1559179376137327 +8229317157758012712_3860_000_3880_000,1559179378637568 +8229317157758012712_3860_000_3880_000,1559179374137643 +8229317157758012712_3860_000_3880_000,1559179374637715 +8229317157758012712_3860_000_3880_000,1559179376637419 +8229317157758012712_3860_000_3880_000,1559179364137325 +8229317157758012712_3860_000_3880_000,1559179377637503 +8229317157758012712_3860_000_3880_000,1559179366137360 +8229317157758012712_3860_000_3880_000,1559179368637389 +8229317157758012712_3860_000_3880_000,1559179377137484 +8229317157758012712_3860_000_3880_000,1559179364637326 +8229317157758012712_3860_000_3880_000,1559179365137367 +8229317157758012712_3860_000_3880_000,1559179367637354 +8229317157758012712_3860_000_3880_000,1559179378137535 +8229317157758012712_3860_000_3880_000,1559179365637366 +5638240639308158118_4220_000_4240_000,1555267988099080 +5638240639308158118_4220_000_4240_000,1555267981099003 +5638240639308158118_4220_000_4240_000,1555267980599134 +5638240639308158118_4220_000_4240_000,1555267989099215 +5638240639308158118_4220_000_4240_000,1555267977599158 +5638240639308158118_4220_000_4240_000,1555267987599108 +5638240639308158118_4220_000_4240_000,1555267986599172 +5638240639308158118_4220_000_4240_000,1555267979599132 +5638240639308158118_4220_000_4240_000,1555267988599141 +5638240639308158118_4220_000_4240_000,1555267990098844 +5638240639308158118_4220_000_4240_000,1555267990598105 +5638240639308158118_4220_000_4240_000,1555267979099131 +5638240639308158118_4220_000_4240_000,1555267978599123 +5638240639308158118_4220_000_4240_000,1555267987099206 +5638240639308158118_4220_000_4240_000,1555267976599172 +5638240639308158118_4220_000_4240_000,1555267977099159 +5638240639308158118_4220_000_4240_000,1555267989599152 +5638240639308158118_4220_000_4240_000,1555267980099165 +5638240639308158118_4220_000_4240_000,1555267978099155 +5638240639308158118_4220_000_4240_000,1555267991096855 +15272375112495403395_620_000_640_000,1559189217599985 +15272375112495403395_620_000_640_000,1559189230099846 +15272375112495403395_620_000_640_000,1559189221600285 +15272375112495403395_620_000_640_000,1559189228599908 +15272375112495403395_620_000_640_000,1559189228100026 +15272375112495403395_620_000_640_000,1559189231099755 +15272375112495403395_620_000_640_000,1559189229599850 +15272375112495403395_620_000_640_000,1559189217099978 +15272375112495403395_620_000_640_000,1559189220599788 +15272375112495403395_620_000_640_000,1559189229099841 +15272375112495403395_620_000_640_000,1559189227100268 +15272375112495403395_620_000_640_000,1559189231599710 +15272375112495403395_620_000_640_000,1559189218599758 +15272375112495403395_620_000_640_000,1559189219599785 +15272375112495403395_620_000_640_000,1559189218099858 +15272375112495403395_620_000_640_000,1559189230599799 +15272375112495403395_620_000_640_000,1559189219099720 +15272375112495403395_620_000_640_000,1559189221099879 +15272375112495403395_620_000_640_000,1559189227600224 +15272375112495403395_620_000_640_000,1559189220099860 +8993680275027614595_2520_000_2540_000,1555280202399639 +8993680275027614595_2520_000_2540_000,1555280199899606 +8993680275027614595_2520_000_2540_000,1555280211375470 +8993680275027614595_2520_000_2540_000,1555280199399568 +8993680275027614595_2520_000_2540_000,1555280212875223 +8993680275027614595_2520_000_2540_000,1555280208875515 +8993680275027614595_2520_000_2540_000,1555280202899788 +8993680275027614595_2520_000_2540_000,1555280210374654 +8993680275027614595_2520_000_2540_000,1555280210875023 +8993680275027614595_2520_000_2540_000,1555280200899582 +8993680275027614595_2520_000_2540_000,1555280201399518 +8993680275027614595_2520_000_2540_000,1555280212375553 +8993680275027614595_2520_000_2540_000,1555280209874639 +8993680275027614595_2520_000_2540_000,1555280211875697 +8993680275027614595_2520_000_2540_000,1555280209374829 +8993680275027614595_2520_000_2540_000,1555280203399700 +8993680275027614595_2520_000_2540_000,1555280201899495 +8993680275027614595_2520_000_2540_000,1555280213374830 +8993680275027614595_2520_000_2540_000,1555280200399612 +8993680275027614595_2520_000_2540_000,1555280198899595 +8688567562597583972_940_000_960_000,1555217344950039 +8688567562597583972_940_000_960_000,1555217347949948 +8688567562597583972_940_000_960_000,1555217356449802 +8688567562597583972_940_000_960_000,1555217353949933 +8688567562597583972_940_000_960_000,1555217345450004 +8688567562597583972_940_000_960_000,1555217346449944 +8688567562597583972_940_000_960_000,1555217343450016 +8688567562597583972_940_000_960_000,1555217344449945 +8688567562597583972_940_000_960_000,1555217346949874 +8688567562597583972_940_000_960_000,1555217355449905 +8688567562597583972_940_000_960_000,1555217353449883 +8688567562597583972_940_000_960_000,1555217355949898 +8688567562597583972_940_000_960_000,1555217354949900 +8688567562597583972_940_000_960_000,1555217357449853 +8688567562597583972_940_000_960_000,1555217345949937 +8688567562597583972_940_000_960_000,1555217354449934 +8688567562597583972_940_000_960_000,1555217356949774 +8688567562597583972_940_000_960_000,1555217343949948 +8688567562597583972_940_000_960_000,1555217357949939 +8688567562597583972_940_000_960_000,1555217347449863 +7247823803417339098_2320_000_2340_000,1557197726848807 +7247823803417339098_2320_000_2340_000,1557197726349233 +7247823803417339098_2320_000_2340_000,1557197727348551 +7247823803417339098_2320_000_2340_000,1557197714347252 +7247823803417339098_2320_000_2340_000,1557197716347129 +7247823803417339098_2320_000_2340_000,1557197725349846 +7247823803417339098_2320_000_2340_000,1557197718347455 +7247823803417339098_2320_000_2340_000,1557197716847198 +7247823803417339098_2320_000_2340_000,1557197715847235 +7247823803417339098_2320_000_2340_000,1557197724349365 +7247823803417339098_2320_000_2340_000,1557197714847182 +7247823803417339098_2320_000_2340_000,1557197717847546 +7247823803417339098_2320_000_2340_000,1557197728348372 +7247823803417339098_2320_000_2340_000,1557197715347156 +7247823803417339098_2320_000_2340_000,1557197727848417 +7247823803417339098_2320_000_2340_000,1557197718847355 +7247823803417339098_2320_000_2340_000,1557197728848372 +7247823803417339098_2320_000_2340_000,1557197724849707 +7247823803417339098_2320_000_2340_000,1557197725849623 +7247823803417339098_2320_000_2340_000,1557197717347349 +2601205676330128831_4880_000_4900_000,1555183240199075 +2601205676330128831_4880_000_4900_000,1555183251775192 +2601205676330128831_4880_000_4900_000,1555183242695259 +2601205676330128831_4880_000_4900_000,1555183239698969 +2601205676330128831_4880_000_4900_000,1555183252774590 +2601205676330128831_4880_000_4900_000,1555183239198898 +2601205676330128831_4880_000_4900_000,1555183241697881 +2601205676330128831_4880_000_4900_000,1555183250274996 +2601205676330128831_4880_000_4900_000,1555183248775035 +2601205676330128831_4880_000_4900_000,1555183242196604 +2601205676330128831_4880_000_4900_000,1555183241198707 +2601205676330128831_4880_000_4900_000,1555183252274928 +2601205676330128831_4880_000_4900_000,1555183253274584 +2601205676330128831_4880_000_4900_000,1555183249775067 +2601205676330128831_4880_000_4900_000,1555183238698908 +2601205676330128831_4880_000_4900_000,1555183240699040 +2601205676330128831_4880_000_4900_000,1555183243193747 +2601205676330128831_4880_000_4900_000,1555183251275298 +2601205676330128831_4880_000_4900_000,1555183249275187 +2601205676330128831_4880_000_4900_000,1555183250775187 +14737335824319407706_1980_000_2000_000,1556068257625722 +14737335824319407706_1980_000_2000_000,1556068264624994 +14737335824319407706_1980_000_2000_000,1556068253125108 +14737335824319407706_1980_000_2000_000,1556068256626068 +14737335824319407706_1980_000_2000_000,1556068256125917 +14737335824319407706_1980_000_2000_000,1556068267124989 +14737335824319407706_1980_000_2000_000,1556068254125759 +14737335824319407706_1980_000_2000_000,1556068265124999 +14737335824319407706_1980_000_2000_000,1556068263125013 +14737335824319407706_1980_000_2000_000,1556068266125077 +14737335824319407706_1980_000_2000_000,1556068254626070 +14737335824319407706_1980_000_2000_000,1556068265625046 +14737335824319407706_1980_000_2000_000,1556068255126360 +14737335824319407706_1980_000_2000_000,1556068267624889 +14737335824319407706_1980_000_2000_000,1556068255626085 +14737335824319407706_1980_000_2000_000,1556068266625069 +14737335824319407706_1980_000_2000_000,1556068264124922 +14737335824319407706_1980_000_2000_000,1556068257126022 +14737335824319407706_1980_000_2000_000,1556068253625378 +14737335824319407706_1980_000_2000_000,1556068263624987 +10504764403039842352_460_000_480_000,1558060925875055 +10504764403039842352_460_000_480_000,1558060940374703 +10504764403039842352_460_000_480_000,1558060939874709 +10504764403039842352_460_000_480_000,1558060937374792 +10504764403039842352_460_000_480_000,1558060927874686 +10504764403039842352_460_000_480_000,1558060926874887 +10504764403039842352_460_000_480_000,1558060930375221 +10504764403039842352_460_000_480_000,1558060926375083 +10504764403039842352_460_000_480_000,1558060935875120 +10504764403039842352_460_000_480_000,1558060936375015 +10504764403039842352_460_000_480_000,1558060936874787 +10504764403039842352_460_000_480_000,1558060938875168 +10504764403039842352_460_000_480_000,1558060928875075 +10504764403039842352_460_000_480_000,1558060937874938 +10504764403039842352_460_000_480_000,1558060928374842 +10504764403039842352_460_000_480_000,1558060929375235 +10504764403039842352_460_000_480_000,1558060938375035 +10504764403039842352_460_000_480_000,1558060939374902 +4140965781175793864_460_000_480_000,1559189068049919 +4140965781175793864_460_000_480_000,1559189060549423 +4140965781175793864_460_000_480_000,1559189058052659 +4140965781175793864_460_000_480_000,1559189070549944 +4140965781175793864_460_000_480_000,1559189071550057 +4140965781175793864_460_000_480_000,1559189067049957 +4140965781175793864_460_000_480_000,1559189061049573 +4140965781175793864_460_000_480_000,1559189059549297 +4140965781175793864_460_000_480_000,1559189067549997 +4140965781175793864_460_000_480_000,1559189058551289 +4140965781175793864_460_000_480_000,1559189057056840 +4140965781175793864_460_000_480_000,1559189069550001 +4140965781175793864_460_000_480_000,1559189068549926 +4140965781175793864_460_000_480_000,1559189069049952 +4140965781175793864_460_000_480_000,1559189059049934 +4140965781175793864_460_000_480_000,1559189057554573 +4140965781175793864_460_000_480_000,1559189070049942 +4140965781175793864_460_000_480_000,1559189061549638 +4140965781175793864_460_000_480_000,1559189071050027 +4140965781175793864_460_000_480_000,1559189060049248 +14188689528137485670_2660_000_2680_000,1555687836099829 +14188689528137485670_2660_000_2680_000,1555687847574536 +14188689528137485670_2660_000_2680_000,1555687834599917 +14188689528137485670_2660_000_2680_000,1555687835599804 +14188689528137485670_2660_000_2680_000,1555687844576878 +14188689528137485670_2660_000_2680_000,1555687838099816 +14188689528137485670_2660_000_2680_000,1555687846574299 +14188689528137485670_2660_000_2680_000,1555687836599840 +14188689528137485670_2660_000_2680_000,1555687837099812 +14188689528137485670_2660_000_2680_000,1555687848074544 +14188689528137485670_2660_000_2680_000,1555687845075193 +14188689528137485670_2660_000_2680_000,1555687834099910 +14188689528137485670_2660_000_2680_000,1555687845574255 +14188689528137485670_2660_000_2680_000,1555687847074492 +14188689528137485670_2660_000_2680_000,1555687835099800 +14188689528137485670_2660_000_2680_000,1555687843582715 +14188689528137485670_2660_000_2680_000,1555687837599851 +14188689528137485670_2660_000_2680_000,1555687833599780 +14188689528137485670_2660_000_2680_000,1555687846074113 +14188689528137485670_2660_000_2680_000,1555687844079474 +18149616047892103767_2460_000_2480_000,1555706658299969 +18149616047892103767_2460_000_2480_000,1555706646800116 +18149616047892103767_2460_000_2480_000,1555706656800049 +18149616047892103767_2460_000_2480_000,1555706647300089 +18149616047892103767_2460_000_2480_000,1555706645799946 +18149616047892103767_2460_000_2480_000,1555706645299873 +18149616047892103767_2460_000_2480_000,1555706644299834 +18149616047892103767_2460_000_2480_000,1555706654299962 +18149616047892103767_2460_000_2480_000,1555706648799880 +18149616047892103767_2460_000_2480_000,1555706656300141 +18149616047892103767_2460_000_2480_000,1555706644799899 +18149616047892103767_2460_000_2480_000,1555706658800051 +18149616047892103767_2460_000_2480_000,1555706655300035 +18149616047892103767_2460_000_2480_000,1555706654799999 +18149616047892103767_2460_000_2480_000,1555706655800109 +18149616047892103767_2460_000_2480_000,1555706657299969 +18149616047892103767_2460_000_2480_000,1555706646300071 +18149616047892103767_2460_000_2480_000,1555706657799945 +18149616047892103767_2460_000_2480_000,1555706647800020 +18149616047892103767_2460_000_2480_000,1555706648299913 +5026942594071056992_3120_000_3140_000,1555462125499896 +5026942594071056992_3120_000_3140_000,1555462133999526 +5026942594071056992_3120_000_3140_000,1555462131999686 +5026942594071056992_3120_000_3140_000,1555462120999711 +5026942594071056992_3120_000_3140_000,1555462123499771 +5026942594071056992_3120_000_3140_000,1555462132499693 +5026942594071056992_3120_000_3140_000,1555462124499589 +5026942594071056992_3120_000_3140_000,1555462122500198 +5026942594071056992_3120_000_3140_000,1555462123999626 +5026942594071056992_3120_000_3140_000,1555462130999515 +5026942594071056992_3120_000_3140_000,1555462123000001 +5026942594071056992_3120_000_3140_000,1555462121499912 +5026942594071056992_3120_000_3140_000,1555462132999655 +5026942594071056992_3120_000_3140_000,1555462135499500 +5026942594071056992_3120_000_3140_000,1555462124999696 +5026942594071056992_3120_000_3140_000,1555462133499574 +5026942594071056992_3120_000_3140_000,1555462122000279 +5026942594071056992_3120_000_3140_000,1555462134999525 +5026942594071056992_3120_000_3140_000,1555462131499619 +5026942594071056992_3120_000_3140_000,1555462134499515 +11987368976578218644_1340_000_1360_000,1557240254147006 +11987368976578218644_1340_000_1360_000,1557240256647136 +11987368976578218644_1340_000_1360_000,1557240253147019 +11987368976578218644_1340_000_1360_000,1557240264121600 +11987368976578218644_1340_000_1360_000,1557240266622584 +11987368976578218644_1340_000_1360_000,1557240253646981 +11987368976578218644_1340_000_1360_000,1557240263622577 +11987368976578218644_1340_000_1360_000,1557240255647121 +11987368976578218644_1340_000_1360_000,1557240266122577 +11987368976578218644_1340_000_1360_000,1557240252646979 +11987368976578218644_1340_000_1360_000,1557240256147181 +11987368976578218644_1340_000_1360_000,1557240265622400 +11987368976578218644_1340_000_1360_000,1557240263124752 +11987368976578218644_1340_000_1360_000,1557240252147007 +11987368976578218644_1340_000_1360_000,1557240254647011 +11987368976578218644_1340_000_1360_000,1557240264621606 +11987368976578218644_1340_000_1360_000,1557240265121984 +11987368976578218644_1340_000_1360_000,1557240255147121 +11987368976578218644_1340_000_1360_000,1557240262627879 +11987368976578218644_1340_000_1360_000,1557240262131544 +17136775999940024630_4860_000_4880_000,1555381565899350 +17136775999940024630_4860_000_4880_000,1555381569399418 +17136775999940024630_4860_000_4880_000,1555381577399397 +17136775999940024630_4860_000_4880_000,1555381567899452 +17136775999940024630_4860_000_4880_000,1555381579899405 +17136775999940024630_4860_000_4880_000,1555381576399429 +17136775999940024630_4860_000_4880_000,1555381566399384 +17136775999940024630_4860_000_4880_000,1555381569899411 +17136775999940024630_4860_000_4880_000,1555381579399300 +17136775999940024630_4860_000_4880_000,1555381576899420 +17136775999940024630_4860_000_4880_000,1555381565399404 +17136775999940024630_4860_000_4880_000,1555381575399420 +17136775999940024630_4860_000_4880_000,1555381578399393 +17136775999940024630_4860_000_4880_000,1555381567399421 +17136775999940024630_4860_000_4880_000,1555381575899458 +17136775999940024630_4860_000_4880_000,1555381577899394 +17136775999940024630_4860_000_4880_000,1555381568399448 +17136775999940024630_4860_000_4880_000,1555381568899445 +17136775999940024630_4860_000_4880_000,1555381578899304 +10980133015080705026_780_000_800_000,1557159347347517 +10980133015080705026_780_000_800_000,1557159341347548 +10980133015080705026_780_000_800_000,1557159350347179 +10980133015080705026_780_000_800_000,1557159338347170 +10980133015080705026_780_000_800_000,1557159348347894 +10980133015080705026_780_000_800_000,1557159341847592 +10980133015080705026_780_000_800_000,1557159340347306 +10980133015080705026_780_000_800_000,1557159351347307 +10980133015080705026_780_000_800_000,1557159339347070 +10980133015080705026_780_000_800_000,1557159349347437 +10980133015080705026_780_000_800_000,1557159348847749 +10980133015080705026_780_000_800_000,1557159337346937 +10980133015080705026_780_000_800_000,1557159340847461 +10980133015080705026_780_000_800_000,1557159350847321 +10980133015080705026_780_000_800_000,1557159337847132 +10980133015080705026_780_000_800_000,1557159349847214 +10980133015080705026_780_000_800_000,1557159347847829 +10980133015080705026_780_000_800_000,1557159338847114 +10980133015080705026_780_000_800_000,1557159351847230 +10980133015080705026_780_000_800_000,1557159339847156 +17792628511034220885_2360_000_2380_000,1555038976374997 +17792628511034220885_2360_000_2380_000,1555038978374871 +17792628511034220885_2360_000_2380_000,1555038976874968 +17792628511034220885_2360_000_2380_000,1555038975875022 +17792628511034220885_2360_000_2380_000,1555038968860681 +17792628511034220885_2360_000_2380_000,1555038964850579 +17792628511034220885_2360_000_2380_000,1555038974875036 +17792628511034220885_2360_000_2380_000,1555038977374963 +17792628511034220885_2360_000_2380_000,1555038978874913 +17792628511034220885_2360_000_2380_000,1555038966351482 +17792628511034220885_2360_000_2380_000,1555038979375036 +17792628511034220885_2360_000_2380_000,1555038965850871 +17792628511034220885_2360_000_2380_000,1555038977874932 +17792628511034220885_2360_000_2380_000,1555038967353934 +17792628511034220885_2360_000_2380_000,1555038969363655 +17792628511034220885_2360_000_2380_000,1555038965350606 +17792628511034220885_2360_000_2380_000,1555038966852499 +17792628511034220885_2360_000_2380_000,1555038968358038 diff --git a/pointcept/datasets/preprocessing/waymo/preprocess_waymo.py b/pointcept/datasets/preprocessing/waymo/preprocess_waymo.py new file mode 100644 index 0000000000000000000000000000000000000000..aab1de34db01cb0a8c4e8bba9c7184f535c52394 --- /dev/null +++ b/pointcept/datasets/preprocessing/waymo/preprocess_waymo.py @@ -0,0 +1,392 @@ +""" +Preprocessing Script for ScanNet 20/200 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import warnings + +warnings.filterwarnings("ignore", category=DeprecationWarning) + +import os + +os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3" +os.environ["CUDA_VISIBLE_DEVICES"] = "-1" + +import argparse +import numpy as np +import tensorflow.compat.v1 as tf +from pathlib import Path +from waymo_open_dataset.utils import frame_utils +from waymo_open_dataset.utils import transform_utils +from waymo_open_dataset.utils import range_image_utils +from waymo_open_dataset import dataset_pb2 as open_dataset +import glob +import multiprocessing as mp +from concurrent.futures import ProcessPoolExecutor +from itertools import repeat + + +def create_lidar(frame): + """Parse and save the lidar data in psd format. + Args: + frame (:obj:`Frame`): Open dataset frame proto. + """ + ( + range_images, + camera_projections, + segmentation_labels, + range_image_top_pose, + ) = frame_utils.parse_range_image_and_camera_projection(frame) + + points, cp_points, valid_masks = convert_range_image_to_point_cloud( + frame, + range_images, + camera_projections, + range_image_top_pose, + keep_polar_features=True, + ) + points_ri2, cp_points_ri2, valid_masks_ri2 = convert_range_image_to_point_cloud( + frame, + range_images, + camera_projections, + range_image_top_pose, + ri_index=1, + keep_polar_features=True, + ) + + # 3d points in vehicle frame. + points_all = np.concatenate(points, axis=0) + points_all_ri2 = np.concatenate(points_ri2, axis=0) + # point labels. + + points_all = np.concatenate([points_all, points_all_ri2], axis=0) + + velodyne = np.c_[points_all[:, 3:6], points_all[:, 1]] + velodyne = velodyne.reshape((velodyne.shape[0] * velodyne.shape[1])) + + valid_masks = [valid_masks, valid_masks_ri2] + return velodyne, valid_masks + + +def create_label(frame): + ( + range_images, + camera_projections, + segmentation_labels, + range_image_top_pose, + ) = frame_utils.parse_range_image_and_camera_projection(frame) + + point_labels = convert_range_image_to_point_cloud_labels( + frame, range_images, segmentation_labels + ) + point_labels_ri2 = convert_range_image_to_point_cloud_labels( + frame, range_images, segmentation_labels, ri_index=1 + ) + + # point labels. + point_labels_all = np.concatenate(point_labels, axis=0) + point_labels_all_ri2 = np.concatenate(point_labels_ri2, axis=0) + point_labels_all = np.concatenate([point_labels_all, point_labels_all_ri2], axis=0) + + labels = point_labels_all + return labels + + +def convert_range_image_to_cartesian( + frame, range_images, range_image_top_pose, ri_index=0, keep_polar_features=False +): + """Convert range images from polar coordinates to Cartesian coordinates. + + Args: + frame: open dataset frame + range_images: A dict of {laser_name, [range_image_first_return, + range_image_second_return]}. + range_image_top_pose: range image pixel pose for top lidar. + ri_index: 0 for the first return, 1 for the second return. + keep_polar_features: If true, keep the features from the polar range image + (i.e. range, intensity, and elongation) as the first features in the + output range image. + + Returns: + dict of {laser_name, (H, W, D)} range images in Cartesian coordinates. D + will be 3 if keep_polar_features is False (x, y, z) and 6 if + keep_polar_features is True (range, intensity, elongation, x, y, z). + """ + cartesian_range_images = {} + frame_pose = tf.convert_to_tensor( + value=np.reshape(np.array(frame.pose.transform), [4, 4]) + ) + + # [H, W, 6] + range_image_top_pose_tensor = tf.reshape( + tf.convert_to_tensor(value=range_image_top_pose.data), + range_image_top_pose.shape.dims, + ) + # [H, W, 3, 3] + range_image_top_pose_tensor_rotation = transform_utils.get_rotation_matrix( + range_image_top_pose_tensor[..., 0], + range_image_top_pose_tensor[..., 1], + range_image_top_pose_tensor[..., 2], + ) + range_image_top_pose_tensor_translation = range_image_top_pose_tensor[..., 3:] + range_image_top_pose_tensor = transform_utils.get_transform( + range_image_top_pose_tensor_rotation, range_image_top_pose_tensor_translation + ) + + for c in frame.context.laser_calibrations: + range_image = range_images[c.name][ri_index] + if len(c.beam_inclinations) == 0: # pylint: disable=g-explicit-length-test + beam_inclinations = range_image_utils.compute_inclination( + tf.constant([c.beam_inclination_min, c.beam_inclination_max]), + height=range_image.shape.dims[0], + ) + else: + beam_inclinations = tf.constant(c.beam_inclinations) + + beam_inclinations = tf.reverse(beam_inclinations, axis=[-1]) + extrinsic = np.reshape(np.array(c.extrinsic.transform), [4, 4]) + + range_image_tensor = tf.reshape( + tf.convert_to_tensor(value=range_image.data), range_image.shape.dims + ) + pixel_pose_local = None + frame_pose_local = None + if c.name == open_dataset.LaserName.TOP: + pixel_pose_local = range_image_top_pose_tensor + pixel_pose_local = tf.expand_dims(pixel_pose_local, axis=0) + frame_pose_local = tf.expand_dims(frame_pose, axis=0) + range_image_cartesian = range_image_utils.extract_point_cloud_from_range_image( + tf.expand_dims(range_image_tensor[..., 0], axis=0), + tf.expand_dims(extrinsic, axis=0), + tf.expand_dims(tf.convert_to_tensor(value=beam_inclinations), axis=0), + pixel_pose=pixel_pose_local, + frame_pose=frame_pose_local, + ) + + range_image_cartesian = tf.squeeze(range_image_cartesian, axis=0) + + if keep_polar_features: + # If we want to keep the polar coordinate features of range, intensity, + # and elongation, concatenate them to be the initial dimensions of the + # returned Cartesian range image. + range_image_cartesian = tf.concat( + [range_image_tensor[..., 0:3], range_image_cartesian], axis=-1 + ) + + cartesian_range_images[c.name] = range_image_cartesian + + return cartesian_range_images + + +def convert_range_image_to_point_cloud( + frame, + range_images, + camera_projections, + range_image_top_pose, + ri_index=0, + keep_polar_features=False, +): + """Convert range images to point cloud. + + Args: + frame: open dataset frame + range_images: A dict of {laser_name, [range_image_first_return, + range_image_second_return]}. + camera_projections: A dict of {laser_name, + [camera_projection_from_first_return, + camera_projection_from_second_return]}. + range_image_top_pose: range image pixel pose for top lidar. + ri_index: 0 for the first return, 1 for the second return. + keep_polar_features: If true, keep the features from the polar range image + (i.e. range, intensity, and elongation) as the first features in the + output range image. + + Returns: + points: {[N, 3]} list of 3d lidar points of length 5 (number of lidars). + (NOTE: Will be {[N, 6]} if keep_polar_features is true. + cp_points: {[N, 6]} list of camera projections of length 5 + (number of lidars). + """ + calibrations = sorted(frame.context.laser_calibrations, key=lambda c: c.name) + points = [] + cp_points = [] + valid_masks = [] + + cartesian_range_images = convert_range_image_to_cartesian( + frame, range_images, range_image_top_pose, ri_index, keep_polar_features + ) + + for c in calibrations: + range_image = range_images[c.name][ri_index] + range_image_tensor = tf.reshape( + tf.convert_to_tensor(value=range_image.data), range_image.shape.dims + ) + range_image_mask = range_image_tensor[..., 0] > 0 + + range_image_cartesian = cartesian_range_images[c.name] + points_tensor = tf.gather_nd( + range_image_cartesian, tf.compat.v1.where(range_image_mask) + ) + + cp = camera_projections[c.name][ri_index] + cp_tensor = tf.reshape(tf.convert_to_tensor(value=cp.data), cp.shape.dims) + cp_points_tensor = tf.gather_nd(cp_tensor, tf.compat.v1.where(range_image_mask)) + points.append(points_tensor.numpy()) + cp_points.append(cp_points_tensor.numpy()) + valid_masks.append(range_image_mask.numpy()) + + return points, cp_points, valid_masks + + +def convert_range_image_to_point_cloud_labels( + frame, range_images, segmentation_labels, ri_index=0 +): + """Convert segmentation labels from range images to point clouds. + + Args: + frame: open dataset frame + range_images: A dict of {laser_name, [range_image_first_return, + range_image_second_return]}. + segmentation_labels: A dict of {laser_name, [range_image_first_return, + range_image_second_return]}. + ri_index: 0 for the first return, 1 for the second return. + + Returns: + point_labels: {[N, 2]} list of 3d lidar points's segmentation labels. 0 for + points that are not labeled. + """ + calibrations = sorted(frame.context.laser_calibrations, key=lambda c: c.name) + point_labels = [] + for c in calibrations: + range_image = range_images[c.name][ri_index] + range_image_tensor = tf.reshape( + tf.convert_to_tensor(range_image.data), range_image.shape.dims + ) + range_image_mask = range_image_tensor[..., 0] > 0 + + if c.name in segmentation_labels: + sl = segmentation_labels[c.name][ri_index] + sl_tensor = tf.reshape(tf.convert_to_tensor(sl.data), sl.shape.dims) + sl_points_tensor = tf.gather_nd(sl_tensor, tf.where(range_image_mask)) + else: + num_valid_point = tf.math.reduce_sum(tf.cast(range_image_mask, tf.int32)) + sl_points_tensor = tf.zeros([num_valid_point, 2], dtype=tf.int32) + + point_labels.append(sl_points_tensor.numpy()) + return point_labels + + +def handle_process(file_path, output_root, test_frame_list): + file = os.path.basename(file_path) + split = os.path.basename(os.path.dirname(file_path)) + print(f"Parsing {split}/{file}") + save_path = Path(output_root) / split / file.split(".")[0] + + data_group = tf.data.TFRecordDataset(file_path, compression_type="") + for data in data_group: + frame = open_dataset.Frame() + frame.ParseFromString(bytes(data.numpy())) + context_name = frame.context.name + timestamp = str(frame.timestamp_micros) + + if split != "testing": + # for training and validation frame, extract labelled frame + if not frame.lasers[0].ri_return1.segmentation_label_compressed: + continue + else: + # for testing frame, extract frame in test_frame_list + if f"{context_name},{timestamp}" not in test_frame_list: + continue + + os.makedirs(save_path / timestamp, exist_ok=True) + + # extract frame pass above check + point_cloud, valid_masks = create_lidar(frame) + point_cloud = point_cloud.reshape(-1, 4) + coord = point_cloud[:, :3] + strength = np.tanh(point_cloud[:, -1].reshape([-1, 1])) + pose = np.array(frame.pose.transform, np.float32).reshape(4, 4) + mask = np.array(valid_masks, dtype=object) + + np.save(save_path / timestamp / "coord.npy", coord) + np.save(save_path / timestamp / "strength.npy", strength) + np.save(save_path / timestamp / "pose.npy", pose) + + # save mask for reverse prediction + if split != "training": + np.save(save_path / timestamp / "mask.npy", mask) + + # save label + if split != "testing": + # ignore TYPE_UNDEFINED, ignore_index 0 -> -1 + label = create_label(frame)[:, 1].reshape([-1]) - 1 + np.save(save_path / timestamp / "segment.npy", label) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset_root", + required=True, + help="Path to the Waymo dataset", + ) + parser.add_argument( + "--output_root", + required=True, + help="Output path where train/val folders will be located", + ) + parser.add_argument( + "--splits", + required=True, + nargs="+", + choices=["training", "validation", "testing"], + help="Splits need to process ([training, validation, testing]).", + ) + parser.add_argument( + "--num_workers", + default=mp.cpu_count(), + type=int, + help="Num workers for preprocessing.", + ) + config = parser.parse_args() + + # load file list + origin_splits = ["training", "validation", "testing"] + file_list = [] + for split in origin_splits: + file_list.extend( + glob.glob( + os.path.join(os.path.abspath(config.dataset_root), split, "*.tfrecord") + ) + ) + assert len(file_list) == 1150, f"have {len(file_list)} files" + + # Create output directories + for split in config.splits: + os.makedirs(os.path.join(config.output_root, split), exist_ok=True) + + file_list = [ + file + for file in file_list + if os.path.basename(os.path.dirname(file)) in config.splits + ] + + # Load test frame list + test_frame_file = os.path.join( + os.path.dirname(__file__), "3d_semseg_test_set_frames.txt" + ) + test_frame_list = [x.rstrip() for x in (open(test_frame_file, "r").readlines())] + + # Preprocess data. + print("Processing scenes...") + pool = ProcessPoolExecutor(max_workers=config.num_workers) + _ = list( + pool.map( + handle_process, + file_list, + repeat(config.output_root), + repeat(test_frame_list), + ) + ) diff --git a/pointcept/datasets/s3dis.py b/pointcept/datasets/s3dis.py new file mode 100644 index 0000000000000000000000000000000000000000..1cfc176380caaa42d35bbf517ae2f13bf5c86f90 --- /dev/null +++ b/pointcept/datasets/s3dis.py @@ -0,0 +1,18 @@ +""" +S3DIS Dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +from .defaults import DefaultDataset +from .builder import DATASETS + + +@DATASETS.register_module() +class S3DISDataset(DefaultDataset): + def get_data_name(self, idx): + remain, room_name = os.path.split(self.data_list[idx % len(self.data_list)]) + remain, area_name = os.path.split(remain) + return f"{area_name}-{room_name}" diff --git a/pointcept/datasets/scannet.py b/pointcept/datasets/scannet.py new file mode 100644 index 0000000000000000000000000000000000000000..3d92805e2a9f8b343d91f0c556a25b91dfda29fb --- /dev/null +++ b/pointcept/datasets/scannet.py @@ -0,0 +1,117 @@ +""" +ScanNet20 / ScanNet200 / ScanNet Data Efficient Dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import glob +import numpy as np +import torch +from copy import deepcopy +from torch.utils.data import Dataset +from collections.abc import Sequence + +from pointcept.utils.logger import get_root_logger +from pointcept.utils.cache import shared_dict +from .builder import DATASETS +from .defaults import DefaultDataset +from .transform import Compose, TRANSFORMS +from .preprocessing.scannet.meta_data.scannet200_constants import ( + VALID_CLASS_IDS_20, + VALID_CLASS_IDS_200, +) + + +@DATASETS.register_module() +class ScanNetDataset(DefaultDataset): + VALID_ASSETS = [ + "coord", + "color", + "normal", + "segment20", + "instance", + ] + class2id = np.array(VALID_CLASS_IDS_20) + + def __init__( + self, + lr_file=None, + la_file=None, + **kwargs, + ): + self.lr = np.loadtxt(lr_file, dtype=str) if lr_file is not None else None + self.la = torch.load(la_file) if la_file is not None else None + super().__init__(**kwargs) + + def get_data_list(self): + if self.lr is None: + data_list = super().get_data_list() + else: + data_list = [ + os.path.join(self.data_root, "train", name) for name in self.lr + ] + return data_list + + def get_data(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + name = self.get_data_name(idx) + split = self.get_split_name(idx) + if self.cache: + cache_name = f"pointcept-{name}" + return shared_dict(cache_name) + + data_dict = {} + assets = os.listdir(data_path) + for asset in assets: + if not asset.endswith(".npy"): + continue + if asset[:-4] not in self.VALID_ASSETS: + continue + data_dict[asset[:-4]] = np.load(os.path.join(data_path, asset)) + data_dict["name"] = name + data_dict["split"] = split + data_dict["coord"] = data_dict["coord"].astype(np.float32) + data_dict["color"] = data_dict["color"].astype(np.float32) + data_dict["normal"] = data_dict["normal"].astype(np.float32) + if "segment20" in data_dict.keys(): + data_dict["segment"] = ( + data_dict.pop("segment20").reshape([-1]).astype(np.int32) + ) + elif "segment200" in data_dict.keys(): + data_dict["segment"] = ( + data_dict.pop("segment200").reshape([-1]).astype(np.int32) + ) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = ( + data_dict.pop("instance").reshape([-1]).astype(np.int32) + ) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + if self.la: + sampled_index = self.la[self.get_data_name(idx)] + mask = np.ones_like(data_dict["segment"], dtype=bool) + mask[sampled_index] = False + data_dict["segment"][mask] = self.ignore_index + data_dict["sampled_index"] = sampled_index + return data_dict + + +@DATASETS.register_module() +class ScanNet200Dataset(ScanNetDataset): + VALID_ASSETS = [ + "coord", + "color", + "normal", + "segment200", + "instance", + ] + class2id = np.array(VALID_CLASS_IDS_200) diff --git a/pointcept/datasets/scannet_pair.py b/pointcept/datasets/scannet_pair.py new file mode 100644 index 0000000000000000000000000000000000000000..b7fdf199aa90b113bb4e8c7643e5549f62d0c51a --- /dev/null +++ b/pointcept/datasets/scannet_pair.py @@ -0,0 +1,89 @@ +""" +ScanNet Pair Dataset (Frame-level contrastive view) + +Refer PointContrast + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import glob +import numpy as np +import torch +from copy import deepcopy +from torch.utils.data import Dataset + +from pointcept.utils.logger import get_root_logger +from .builder import DATASETS +from .transform import Compose, TRANSFORMS + + +@DATASETS.register_module() +class ScanNetPairDataset(Dataset): + def __init__( + self, + data_root="data/scannet_pair", + overlap_threshold=0.3, + view1_transform=None, + view2_transform=None, + loop=1, + **kwargs + ): + super(ScanNetPairDataset, self).__init__() + self.data_root = data_root + self.overlap_threshold = overlap_threshold + self.view1_transform = Compose(view1_transform) + self.view2_transform = Compose(view2_transform) + self.loop = loop + self.data_list = self.get_data_list() + logger = get_root_logger() + logger.info("Totally {} x {} samples.".format(len(self.data_list), self.loop)) + + def get_data_list(self): + data_list = [] + overlap_list = glob.glob( + os.path.join(self.data_root, "*", "pcd", "overlap.txt") + ) + for overlap_file in overlap_list: + with open(overlap_file) as f: + overlap = f.readlines() + overlap = [pair.strip().split() for pair in overlap] + data_list.extend( + [ + pair[:2] + for pair in overlap + if float(pair[2]) > self.overlap_threshold + ] + ) + return data_list + + def get_data(self, idx): + pair = self.data_list[idx % len(self.data_list)] + view1_dict = torch.load(self.data_root + pair[0]) + view2_dict = torch.load(self.data_root + pair[1]) + return view1_dict, view2_dict + + def get_data_name(self, idx): + return os.path.basename(self.data_list[idx % len(self.data_list)]).split(".")[0] + + def prepare_train_data(self, idx): + # load data + view1_dict, view2_dict = self.get_data(idx) + view1_dict = self.view1_transform(view1_dict) + view2_dict = self.view2_transform(view2_dict) + data_dict = dict() + for key, value in view1_dict.items(): + data_dict["view1_" + key] = value + for key, value in view2_dict.items(): + data_dict["view2_" + key] = value + return data_dict + + def prepare_test_data(self, idx): + raise NotImplementedError + + def __getitem__(self, idx): + return self.prepare_train_data(idx) + + def __len__(self): + return len(self.data_list) * self.loop diff --git a/pointcept/datasets/scannetpp.py b/pointcept/datasets/scannetpp.py new file mode 100644 index 0000000000000000000000000000000000000000..53e7dcaa91e8f813fcd53db5ad77cffd5c80141e --- /dev/null +++ b/pointcept/datasets/scannetpp.py @@ -0,0 +1,82 @@ +""" +ScanNet++ dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import numpy as np +import glob + +from pointcept.utils.cache import shared_dict + +from .builder import DATASETS +from .defaults import DefaultDataset + + +@DATASETS.register_module() +class ScanNetPPDataset(DefaultDataset): + VALID_ASSETS = [ + "coord", + "color", + "normal", + "superpoint", + "segment", + "instance", + ] + + def __init__( + self, + multilabel=False, + **kwargs, + ): + super().__init__(**kwargs) + self.multilabel = multilabel + + def get_data(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + name = self.get_data_name(idx) + if self.cache: + cache_name = f"pointcept-{name}" + return shared_dict(cache_name) + + data_dict = {} + assets = os.listdir(data_path) + for asset in assets: + if not asset.endswith(".npy"): + continue + if asset[:-4] not in self.VALID_ASSETS: + continue + data_dict[asset[:-4]] = np.load(os.path.join(data_path, asset)) + data_dict["name"] = name + + if "coord" in data_dict.keys(): + data_dict["coord"] = data_dict["coord"].astype(np.float32) + + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"].astype(np.float32) + + if "normal" in data_dict.keys(): + data_dict["normal"] = data_dict["normal"].astype(np.float32) + + if "superpoint" in data_dict.keys(): + data_dict["superpoint"] = data_dict["superpoint"].astype(np.int32) + + if not self.multilabel: + if "segment" in data_dict.keys(): + data_dict["segment"] = data_dict["segment"][:, 0].astype(np.int32) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = data_dict["instance"][:, 0].astype(np.int32) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + else: + raise NotImplementedError + return data_dict diff --git a/pointcept/datasets/scanobjectnn.py b/pointcept/datasets/scanobjectnn.py new file mode 100644 index 0000000000000000000000000000000000000000..32906c9659c9450530827010b6e9cb68c5cd0228 --- /dev/null +++ b/pointcept/datasets/scanobjectnn.py @@ -0,0 +1,355 @@ +""" +ScanObjectNN Dataset + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import glob +import h5py +import numpy as np +import torch +from collections.abc import Sequence +import open3d as o3d +import copy + +from pointcept.utils.logger import get_root_logger + +from .builder import DATASETS +from .defaults import DefaultDataset +from .transform import Compose + + +@DATASETS.register_module() +class ScanObjectNNRawDataset(DefaultDataset): + """ + ScanObjectNN Dataset loaded from a standard folder structure. + """ + + def __init__(self, if_img=True, crop_h=518, crop_w=518, patch_size=14, **kwargs): + self.crop_h = crop_h + self.crop_w = crop_w + self.patch_size = patch_size + self.patch_h = crop_h // patch_size + self.patch_w = crop_w // patch_size + self.if_img = if_img + super().__init__(**kwargs) + + def get_data_list(self): + """ + Overrides the parent method. + Scans the data directory (or directories) to find all relevant .bin files + for the specified split(s). + """ + if isinstance(self.split, str): + split_list = [self.split] + elif isinstance(self.split, Sequence): + split_list = self.split + else: + raise NotImplementedError + + data_list = [] + logger = get_root_logger() + + for split in split_list: + split_path = os.path.join(self.data_root, split) + if not os.path.isdir(split_path): + raise FileNotFoundError(f"Split directory not found at: {split_path}") + + logger.info(f"Scanning for .bin files in {split_path}") + # Use glob to find all .bin files within the class subdirectories + all_bin_files = glob.glob(os.path.join(split_path, "*", "*.bin")) + + # Filter out auxiliary files, keeping only the main object files + filtered_files = [ + path + for path in all_bin_files + if not path.endswith("_part.bin") and not path.endswith("_indices.bin") + ] + + if not filtered_files: + logger.warning( + f"No valid .bin files found in {split_path}. Please check the file structure." + ) + + data_list.extend(filtered_files) + + return sorted(data_list) # Sort for reproducibility + + def get_data(self, idx): + """ + Overrides the parent method. + Loads and parses a single .bin file based on its path. + The logic for parsing the binary file format remains the same. + """ + data_path = self.data_list[idx % len(self.data_list)] + + try: + raw_data = np.fromfile(data_path, dtype=np.float32) + except IOError as e: + logger = get_root_logger() + logger.error(f"Could not read file {data_path}: {e}") + # Skip to the next sample if the current one is unreadable + return self.get_data((idx + 1) % len(self.data_list)) + + if raw_data.size == 0: + logger = get_root_logger() + logger.warning(f"File is empty, skipping: {data_path}") + return self.get_data((idx + 1) % len(self.data_list)) + + # First float is the number of points + num_points = int(raw_data[0]) + point_data_flat = raw_data[1:] + + # Validate that the file is not corrupt + expected_size = num_points * 11 + if point_data_flat.size != expected_size: + logger = get_root_logger() + logger.warning( + f"Data corruption detected in {data_path}: " + f"Expected {expected_size} floats for {num_points} points, but found {point_data_flat.size}. Skipping." + ) + return self.get_data((idx + 1) % len(self.data_list)) + + # Reshape into a (num_points, 11) array + point_cloud = point_data_flat.reshape(num_points, 11) + rot_mat = np.array( + [ + [1.0, 0.0, 0.0], + [0.0, 0.0, -1.0], + [0.0, 1.0, 0.0], + ], + dtype=np.float32, + ) + rotated_coord = point_cloud[:, :3].astype(np.float32) @ rot_mat.T + rotated_normal = point_cloud[:, 3:6].astype(np.float32) @ rot_mat.T + if self.if_img: + imgs_tensor = torch.empty( + (0, 3, self.patch_h * self.patch_size, self.patch_w * self.patch_size) + ) + img_num = np.array([imgs_tensor.shape[0]], dtype=np.int32) + correspondence_infos = np.ones( + (point_cloud.shape[0], imgs_tensor.shape[0], 2), + dtype=np.float32, + # (data_dict["coord"].shape[0], len(correspondences_path), 2), dtype=np.int32 + ) * (-1) + # Pack the data into the dictionary format expected by the framework + # Attributes order: x, y, z, nx, ny, nz, r, g, b, instance_label, semantic_label + data_dict = { + "coord": rotated_coord.astype(np.float32), + "normal": rotated_normal.astype(np.float32), + "color": point_cloud[:, 6:9].astype(np.float32), + "instance": point_cloud[:, 9].astype(np.int64), + "segment": point_cloud[:, 10].astype(np.int64), + "name": self.get_data_name(idx), + "split": self.get_split_name(idx), + "images": imgs_tensor, + "img_num": img_num, + "correspondence": correspondence_infos, + } + else: + data_dict = { + "coord": rotated_coord.astype(np.float32), + "normal": rotated_normal.astype(np.float32), + "color": point_cloud[:, 6:9].astype(np.float32), + "instance": point_cloud[:, 9].astype(np.int64), + "segment": point_cloud[:, 10].astype(np.int64), + "name": self.get_data_name(idx), + "split": self.get_split_name(idx), + } + return data_dict + + def get_data_name(self, idx): + """ + Overrides the parent method. + Returns the filename without extension as the data name. + Example: '005_00001' + """ + path = self.data_list[idx % len(self.data_list)] + return os.path.splitext(os.path.basename(path))[0] + + def get_split_name(self, idx): + """ + Overrides the parent method. + Correctly extracts the split name (e.g., 'train') from the file path. + """ + path = self.data_list[idx % len(self.data_list)] + return os.path.basename(os.path.dirname(os.path.dirname(path))) + + +@DATASETS.register_module() +class ScanObjectNNDataset(DefaultDataset): + """ + ScanObjectNN Dataset. + Loads data from HDF5 files into memory. + """ + + def __init__( + self, + class_names, + if_color=True, + if_normal=True, + split="train", + data_root="data/dataset", + transform=None, + test_mode=False, + test_cfg=None, + cache=False, + ignore_index=-1, + loop=1, + ): + self.if_color = if_color + self.if_normal = if_normal + self.class_names = class_names + self.data_root = data_root + self.split = split + self.transform = Compose(transform) + self.cache = cache + self.ignore_index = ignore_index + self.loop = ( + loop if not test_mode else 1 + ) # force make loop = 1 while in test mode + self.test_mode = test_mode + self.test_cfg = test_cfg if test_mode else None + + if test_mode: + self.post_transform = Compose(self.test_cfg.post_transform) + self.aug_transform = [Compose(aug) for aug in self.test_cfg.aug_transform] + + self.data_list = self.get_data_list() + logger = get_root_logger() + logger.info( + "Totally {} x {} samples in {} {} set.".format( + len(self.data_list), self.loop, os.path.basename(self.data_root), split + ) + ) + + @staticmethod + def get_normals(center, coords): + Cs = np.repeat(center.reshape((1, -1)), coords.shape[0], axis=0) + view_dirs = coords - Cs + view_dirs = view_dirs / ( + np.linalg.norm(view_dirs, axis=-1, keepdims=True) + 1e-6 + ) + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coords) + pcd.estimate_normals() + normals = np.asarray(pcd.normals) + dot_product = np.sum(normals * view_dirs, axis=-1) + flip_mask = dot_product < 0 + normals[flip_mask] = -normals[flip_mask] + normals = normals / np.linalg.norm(normals, axis=-1, keepdims=True) + return normals + + def get_data_list(self): + """ + Called by the parent constructor. + Here, we load the HDF5 file into memory and return a list of sample indices. + """ + if isinstance(self.split, str): + split_list = [self.split] + elif isinstance(self.split, Sequence): + split_list = self.split + else: + raise NotImplementedError + + self.points = [] + self.labels = [] + for split in split_list: + if split == "train": + h5_path = os.path.join(self.data_root, "training_objectdataset.h5") + elif split == "test": + h5_path = os.path.join(self.data_root, "test_objectdataset.h5") + else: + raise NotImplementedError(f"Split {self.split} not supported.") + + with h5py.File(h5_path, "r") as h5: + self.points.append(np.array(h5["data"]).astype(np.float32)) + # Use np.int64 for labels, as it's the default for PyTorch's CrossEntropyLoss + self.labels.append(np.array(h5["label"]).astype(np.int64)) + self.points = np.concatenate(self.points, axis=0) + self.labels = np.concatenate(self.labels, axis=0) + # The data_list is now a list of indices, from 0 to num_samples-1. + return list(range(self.points.shape[0])) + + def get_data(self, idx): + """ + Overrides the parent method. + Instead of loading from a file path, we get the data from the pre-loaded numpy arrays. + """ + data_idx = self.data_list[idx % len(self.data_list)] + coord = self.points[data_idx].copy() + if self.if_color: + color = np.zeros_like(coord) + if self.if_normal: + normal = self.get_normals(np.mean(coord, axis=0), coord) + label = self.labels[data_idx] + category = np.array([label]) + + data_dict = { + "coord": coord, + "color": color, + "normal": normal, + "category": category, + "name": self.get_data_name(idx), + } + + return data_dict + + def get_data_name(self, idx): + """ + Overrides the parent method. + Returns a unique name for the sample at the given index. + """ + real_idx = idx % len(self.data_list) + return f"{self.split}-{real_idx}" + + def get_split_name(self, idx): + return self.split + + def prepare_test_data(self, idx): + assert idx < len(self.data_list) + data_dict = self.get_data(idx) + category = data_dict.pop("category") + data_dict = self.transform(data_dict) + data_dict_list = [] + for aug in self.aug_transform: + data_dict_list.append(aug(copy.deepcopy(data_dict))) + for i in range(len(data_dict_list)): + data_dict_list[i] = self.post_transform(data_dict_list[i]) + data_dict = dict( + voting_list=data_dict_list, + category=category, + name=self.get_data_name(idx), + ) + return data_dict + + +@DATASETS.register_module() +class ScanObjectNNHardestDataset(ScanObjectNNDataset): + """ + ScanObjectNN "Hardest" variant. + Inherits from our refactored ScanObjectNN and only overrides the file loading logic. + """ + + def get_data_list(self): + """ + Overrides the parent method to load from the augmented HDF5 files. + """ + if self.split == "train": + h5_path = os.path.join( + self.data_root, "training_objectdataset_augmentedrot_scale75.h5" + ) + elif self.split == "test": + h5_path = os.path.join( + self.data_root, "test_objectdataset_augmentedrot_scale75.h5" + ) + else: + raise NotImplementedError(f"Split {self.split} not supported.") + + with h5py.File(h5_path, "r") as h5: + self.points = np.array(h5["data"]).astype(np.float32) + self.labels = np.array(h5["label"]).astype(np.int64) + + return list(range(self.points.shape[0])) diff --git a/pointcept/datasets/semantic_kitti.py b/pointcept/datasets/semantic_kitti.py new file mode 100644 index 0000000000000000000000000000000000000000..955a224ea6f3fec1737c80a696ce3ad7bc1ee43b --- /dev/null +++ b/pointcept/datasets/semantic_kitti.py @@ -0,0 +1,630 @@ +""" +Semantic KITTI dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import numpy as np +from PIL import Image +import torch +import open3d as o3d +import random + +from .builder import DATASETS +from .defaults import DefaultDataset, DefaultImagePointDataset + +os.environ["OMP_NUM_THREADS"] = "1" + + +@DATASETS.register_module() +class SemanticKITTIDataset(DefaultDataset): + def __init__(self, ignore_index=-1, **kwargs): + self.ignore_index = ignore_index + self.learning_map = self.get_learning_map(ignore_index) + self.learning_map_inv = self.get_learning_map_inv(ignore_index) + super().__init__(ignore_index=ignore_index, **kwargs) + + def get_data_list(self): + split2seq = dict( + train=[0, 1, 2, 3, 4, 5, 6, 7, 9, 10], + val=[8], + test=[11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21], + ) + if isinstance(self.split, str): + seq_list = split2seq[self.split] + elif isinstance(self.split, list): + seq_list = [] + for split in self.split: + seq_list += split2seq[split] + else: + raise NotImplementedError + + data_list = [] + for seq in seq_list: + seq = str(seq).zfill(2) + seq_folder = os.path.join(self.data_root, "dataset", "sequences", seq) + seq_files = sorted(os.listdir(os.path.join(seq_folder, "velodyne"))) + data_list += [ + os.path.join(seq_folder, "velodyne", file) for file in seq_files + ] + return data_list + + def get_data(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + with open(data_path, "rb") as b: + scan = np.fromfile(b, dtype=np.float32).reshape(-1, 4) + coord = scan[:, :3] + strength = scan[:, -1].reshape([-1, 1]) + + label_file = data_path.replace("velodyne", "labels").replace(".bin", ".label") + if os.path.exists(label_file): + with open(label_file, "rb") as a: + segment = np.fromfile(a, dtype=np.int32).reshape(-1) + segment = np.vectorize(self.learning_map.__getitem__)( + segment & 0xFFFF + ).astype(np.int32) + else: + segment = np.zeros(scan.shape[0]).astype(np.int32) + data_dict = dict( + coord=coord, + strength=strength, + segment=segment, + name=self.get_data_name(idx), + ) + return data_dict + + def get_data_name(self, idx): + file_path = self.data_list[idx % len(self.data_list)] + dir_path, file_name = os.path.split(file_path) + sequence_name = os.path.basename(os.path.dirname(dir_path)) + frame_name = os.path.splitext(file_name)[0] + data_name = f"{sequence_name}_{frame_name}" + return data_name + + @staticmethod + def get_learning_map(ignore_index): + learning_map = { + 0: ignore_index, # "unlabeled" + 1: ignore_index, # "outlier" mapped to "unlabeled" --------------------------mapped + 10: 0, # "car" + 11: 1, # "bicycle" + 13: 4, # "bus" mapped to "other-vehicle" --------------------------mapped + 15: 2, # "motorcycle" + 16: 4, # "on-rails" mapped to "other-vehicle" ---------------------mapped + 18: 3, # "truck" + 20: 4, # "other-vehicle" + 30: 5, # "person" + 31: 6, # "bicyclist" + 32: 7, # "motorcyclist" + 40: 8, # "road" + 44: 9, # "parking" + 48: 10, # "sidewalk" + 49: 11, # "other-ground" + 50: 12, # "building" + 51: 13, # "fence" + 52: ignore_index, # "other-structure" mapped to "unlabeled" ------------------mapped + 60: 8, # "lane-marking" to "road" ---------------------------------mapped + 70: 14, # "vegetation" + 71: 15, # "trunk" + 72: 16, # "terrain" + 80: 17, # "pole" + 81: 18, # "traffic-sign" + 99: ignore_index, # "other-object" to "unlabeled" ----------------------------mapped + 252: 0, # "moving-car" to "car" ------------------------------------mapped + 253: 6, # "moving-bicyclist" to "bicyclist" ------------------------mapped + 254: 5, # "moving-person" to "person" ------------------------------mapped + 255: 7, # "moving-motorcyclist" to "motorcyclist" ------------------mapped + 256: 4, # "moving-on-rails" mapped to "other-vehicle" --------------mapped + 257: 4, # "moving-bus" mapped to "other-vehicle" -------------------mapped + 258: 3, # "moving-truck" to "truck" --------------------------------mapped + 259: 4, # "moving-other"-vehicle to "other-vehicle" ----------------mapped + } + return learning_map + + @staticmethod + def get_learning_map_inv(ignore_index): + learning_map_inv = { + ignore_index: ignore_index, # "unlabeled" + 0: 10, # "car" + 1: 11, # "bicycle" + 2: 15, # "motorcycle" + 3: 18, # "truck" + 4: 20, # "other-vehicle" + 5: 30, # "person" + 6: 31, # "bicyclist" + 7: 32, # "motorcyclist" + 8: 40, # "road" + 9: 44, # "parking" + 10: 48, # "sidewalk" + 11: 49, # "other-ground" + 12: 50, # "building" + 13: 51, # "fence" + 14: 70, # "vegetation" + 15: 71, # "trunk" + 16: 72, # "terrain" + 17: 80, # "pole" + 18: 81, # "traffic-sign" + } + return learning_map_inv + + +@DATASETS.register_module() +class SemanticKITTIColorNormalDataset(SemanticKITTIDataset): + @staticmethod + def estimate_normals(points, center=np.array([0, 0, 0])): + normals = points - center[None, :] + norms = np.linalg.norm(normals, axis=1, keepdims=True) + normals = normals / norms + return normals + + def get_data(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + with open(data_path, "rb") as b: + scan = np.fromfile(b, dtype=np.float32).reshape(-1, 4) + coord = scan[:, :3] + normal = self.estimate_normals(coord) + label_file = data_path.replace("velodyne", "labels").replace(".bin", ".label") + if os.path.exists(label_file): + with open(label_file, "rb") as a: + segment = np.fromfile(a, dtype=np.int32).reshape(-1) + segment = np.vectorize(self.learning_map.__getitem__)( + segment & 0xFFFF + ).astype(np.int32) + else: + segment = np.zeros(scan.shape[0]).astype(np.int32) + data_dict = dict( + coord=coord, + color=np.zeros_like(coord), # placeholder for color + normal=np.zeros_like(coord), # placeholder for normal + segment=segment, + name=self.get_data_name(idx), + ) + return data_dict + + +@DATASETS.register_module() +class SemanticKITTIImagePointDataset(DefaultImagePointDataset): + CAMERA_TYPES = [2, 3] + + def __init__( + self, + if_img=False, + if_sweep=False, + sweeps=3, + sweep_gap=5, + ignore_index=-1, + **kwargs, + ): + self.if_img = if_img + self.ignore_index = ignore_index + self.learning_map = self.get_learning_map(ignore_index) + self.learning_map_inv = self.get_learning_map_inv(ignore_index) + if if_sweep: + self.timestamp = tuple(range(0, sweeps * sweep_gap, sweep_gap)) + else: + self.timestamp = (0,) + super().__init__(ignore_index=ignore_index, if_img=if_img, **kwargs) + + @staticmethod + def project_lidar_to_image_with_color( + lidar_points, # shape: (N, 3) or (N, 4) + image, # shape: (H, W, 3), uint8 RGB + cam_intrinsic, # shape: (3, 3) + lidar_to_cam, # shape: (4, 4) + lidar_colors, + ): + """ + Projects LiDAR points to the image, fetches pixel color and pixel coordinates. + Returns: + filtered_points: (M, 3) - 3D points in camera frame that project onto the image. + colors: (M, 3) - RGB colors at projected 2D locations. + uv_coords: (M, 2) - Integer pixel coordinates (u, v) on the image. + mask: (N,) - (optional) Boolean mask indicating which lidar points are used. + """ + lidar_uv_coords = np.full( + (lidar_points.shape[0], 2), -1, dtype=int + ) # Default to (-1, -1) + lidar_points_coord = lidar_points[:, :3] + ones = np.ones((lidar_points_coord.shape[0], 1)) + lidar_hom = np.concatenate([lidar_points_coord, ones], axis=1) # (N, 4) + + points_cam = (lidar_to_cam @ lidar_hom.T).T # (N, 4) + + valid = points_cam[:, 2] > 0 + points_cam = points_cam[valid] + points_cam_homogeneous = np.hstack( + (points_cam, np.ones((points_cam.shape[0], 1))) + ) + + pts_2d = (cam_intrinsic @ points_cam_homogeneous.T).T # (N, 4) + pts_2d = pts_2d[:, :2] / pts_2d[:, 2:3] # (N, 2) - pixel (u, v) + + H, W = image.shape[:2] + u, v = pts_2d[:, 0], pts_2d[:, 1] + inside = (u >= 0) & (u < W) & (v >= 0) & (v < H) + + u = u[inside].astype(int) + v = v[inside].astype(int) + + mask = np.zeros(lidar_points.shape[0], dtype=bool) + mask[np.where(valid)[0][inside]] = True + + lidar_colors[mask] = image[v, u, :] + lidar_uv_coords[mask] = np.stack([u, v], axis=1) # (M, 2) + + return lidar_colors, lidar_uv_coords, mask + + @staticmethod + def parse_calib_file(filepath): + calib_data = {} + with open(filepath, "r") as f: + file_content = f.read() + + for line in file_content.strip().split("\n"): + if ":" not in line: + continue + key, value_str = line.split(":", 1) + key = key.strip() + values = np.fromstring(value_str, dtype=np.float64, sep=" ") + if values.size == 12: + matrix = values.reshape(3, 4) + elif values.size == 9: + matrix = values.reshape(3, 3) + else: + matrix = values + calib_data[key] = matrix + + return calib_data + + @staticmethod + def align_pose(coord, pose, target_pose): + coord = np.hstack((coord, np.ones_like(coord[:, :1]))) + pose_align = np.matmul(np.linalg.inv(target_pose), pose) + coord = (pose_align @ coord.T).T[:, :3] + return coord + + @staticmethod + def align_normal(normal, pose, target_pose): + pose_align = np.matmul(np.linalg.inv(target_pose), pose) + return (pose_align[:3, :3] @ normal.T).T + + @staticmethod + def get_pose(poses_file_path, frame_index, Tr): + Tr = np.vstack((Tr, np.array([0, 0, 0, 1]))) + Tr_inv = np.linalg.inv(Tr) + with open(poses_file_path, "r") as f: + lines = f.readlines() + + if not frame_index < len(lines): + print(frame_index, len(lines)) + assert frame_index < len(lines) + line = lines[frame_index].strip().split() + values = [float(v) for v in line] + pose_matrix_3x4 = np.array(values).reshape(3, 4) + pose_matrix_4x4 = np.vstack((pose_matrix_3x4, np.array([0, 0, 0, 1]))) + pose_matrix_4x4 = np.matmul(Tr_inv, np.matmul(pose_matrix_4x4, Tr)) + return pose_matrix_4x4 + + def get_data_list(self): + split2seq = dict( + train=[0, 1, 2, 3, 4, 5, 6, 7, 9, 10], + val=[8], + test=[11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21], + ) + if isinstance(self.split, str): + seq_list = split2seq[self.split] + elif isinstance(self.split, list): + seq_list = [] + for split in self.split: + seq_list += split2seq[split] + else: + raise NotImplementedError + seq_list = sorted(seq_list) + + data_list = [] + split_list = {"train": [], "val": [], "test": []} + for seq in seq_list: + seq2 = str(seq).zfill(2) + seq_folder = os.path.join(self.data_root, "dataset", "sequences", seq2) + seq_files = sorted(os.listdir(os.path.join(seq_folder, "velodyne"))) + data_list += [ + os.path.join(seq_folder, "velodyne", file) for file in seq_files + ] + if seq in split2seq["train"]: + split_list["train"].extend( + [os.path.join(seq_folder, "velodyne", file) for file in seq_files] + ) + elif seq in split2seq["val"]: + split_list["val"].extend( + [os.path.join(seq_folder, "velodyne", file) for file in seq_files] + ) + elif seq in split2seq["test"]: + split_list["test"].extend( + [os.path.join(seq_folder, "velodyne", file) for file in seq_files] + ) + _, self.sequence_offset, self.sequence_index = np.unique( + [os.path.dirname(os.path.dirname(data)) for data in data_list], + return_index=True, + return_inverse=True, + ) + self.sequence_offset = np.append(self.sequence_offset, len(data_list)) + return data_list, split_list + + def get_single_frame(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + with open(data_path, "rb") as b: + scan = np.fromfile(b, dtype=np.float32).reshape(-1, 4) + coord = scan[:, :3] + strength = scan[:, -1].reshape([-1, 1]) + + label_file = data_path.replace("velodyne", "labels").replace(".bin", ".label") + if os.path.exists(label_file): + with open(label_file, "rb") as a: + segment = np.fromfile(a, dtype=np.int32).reshape(-1) + segment = np.vectorize(self.learning_map.__getitem__)( + segment & 0xFFFF + ).astype(np.int32) + else: + segment = np.zeros(scan.shape[0]).astype(np.int32) + + imgs = [] + cam_correspondences = [] + correspondence_start = 0 + color = np.zeros((coord.shape[0], 3), dtype=int) # Default to black + + calibration_path = os.path.join( + os.path.dirname(os.path.dirname(data_path)), "calib.txt" + ) + calib_info = self.parse_calib_file(calibration_path) + Tr = calib_info["Tr"] + for id, cam_name in enumerate(self.CAMERA_TYPES): + image_path = data_path.replace("velodyne", f"image_{cam_name}").replace( + ".bin", ".png" + ) + P = calib_info[f"P{cam_name}"] + cam_image = Image.open(image_path) + cam_image_np = np.array(cam_image) + color, correspondence_info, _ = self.project_lidar_to_image_with_color( + coord, cam_image_np, P, Tr, color + ) + correspondence_point_id = ( + np.array(range(correspondence_info.shape[0])).reshape((-1, 1)) + + correspondence_start + ) + correspondence_info = np.hstack( + [correspondence_info, correspondence_point_id] + ) + # if np.random.rand() 0: + imgs_list = torch.stack(imgs) + img_DLC["images"] = imgs_list.float() + else: + img_DLC["images"] = torch.empty( + ( + 0, + 3, + self.patch_h * self.patch_size, + self.patch_w * self.patch_size, + ) + ) + img_DLC["img_num"] = np.array([img_DLC["images"].shape[0]], dtype=np.int32) + + correspondence_infos = np.ones( + (coord.shape[0], len(cam_correspondences), 2), dtype=np.int32 + ) * (-1) + for id, correspondence_info in enumerate(cam_correspondences): + correspondence_info = self.resize_correspondence_info( + correspondence_info, + (self.patch_h * self.patch_size, self.patch_w * self.patch_size), + (img_height, img_width), + (left, top, right, bottom), + self.patch_size, + ) + correspondence_infos[correspondence_info[:, -1], id, :] = ( + correspondence_info[:, :-1] + ) + img_DLC["correspondence"] = correspondence_infos + + color = color.astype(np.float32) + data_dict = dict( + coord=coord + np.array([0, 0, 1.73]), + color=color, + normal=normal, + strength=strength, + segment=segment, + name=self.get_data_name(idx), + ) + data_dict.update(img_DLC) + data_dict["Tr"] = Tr + return data_dict + + def get_data(self, idx): + idx = idx % len(self.data_list) + if self.timestamp == (0,): + data_dict = self.get_single_frame(idx) + return data_dict + + sequence_index = self.sequence_index[idx] + lower, upper = self.sequence_offset[[sequence_index, sequence_index + 1]] + assert lower <= idx and idx < upper + valid_timestamps = [0] + [ + timestamp + for timestamp in self.timestamp[1:] + if lower <= timestamp + idx < upper + ] + imgs_idx = random.sample(valid_timestamps, 1)[0] + major_frame = self.get_single_frame(idx) + poses_file_path = os.path.dirname(os.path.dirname(self.data_list[idx])) + poses_file_path = os.path.join(poses_file_path, "poses.txt") + Tr = major_frame.pop("Tr") + target_pose = self.get_pose(poses_file_path, idx - lower, Tr) + name = major_frame.pop("name") + if self.if_img: + if imgs_idx == 0: + major_frame["correspondence"] = [major_frame["correspondence"]] + else: + major_frame["correspondence"] = [ + -np.ones_like(major_frame["correspondence"]) + ] + for key in major_frame.keys(): + if key in self.PC_VALID_ASSETS: + major_frame[key] = [major_frame[key]] + + for timestamp in valid_timestamps[1:]: + refer_idx = timestamp + idx + refer_frame = self.get_single_frame(refer_idx) + refer_frame.pop("name") + Tr = refer_frame.pop("Tr") + pose = self.get_pose(poses_file_path, idx - lower + timestamp, Tr) + refer_frame["coord"] = self.align_pose( + refer_frame["coord"], pose, target_pose + ) + refer_frame["normal"] = self.align_normal( + refer_frame["normal"], pose, target_pose + ) + for key in major_frame.keys(): + if key in self.PC_VALID_ASSETS: + major_frame[key].append(refer_frame[key]) + + if self.if_img: + if imgs_idx == timestamp: + major_frame["img_num"] = refer_frame["img_num"] + major_frame["images"] = refer_frame["images"] + major_frame["correspondence"].append(refer_frame["correspondence"]) + else: + major_frame["correspondence"].append( + -np.ones_like(refer_frame["correspondence"]) + ) + frame_pcd_offset = [ + frame_coord.shape[0] for frame_coord in major_frame["coord"] + ] + frame_pcd_offset = np.cumsum(frame_pcd_offset) + + for key in major_frame.keys(): + if key in self.PC_VALID_ASSETS + ["correspondence"]: + major_frame[key] = np.concatenate(major_frame[key], axis=0) + major_frame["name"] = name + major_frame["frame_pcd_offset"] = frame_pcd_offset + return major_frame + + @staticmethod + def get_normals(cam_center, coords): + Cs = np.repeat(cam_center.reshape((1, -1)), coords.shape[0], axis=0) + view_dirs = coords - Cs + view_dirs = view_dirs / np.linalg.norm(view_dirs, axis=-1, keepdims=True) + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coords) + pcd.estimate_normals() + normals = np.asarray(pcd.normals) + # normals = estimate_normals(coords,center = cam_center) + dot_product = np.sum(normals * view_dirs, axis=-1) + flip_mask = dot_product > 0 + normals[flip_mask] = -normals[flip_mask] + + # Normalize normals a nd m + normals = normals / np.linalg.norm(normals, axis=-1, keepdims=True) + return normals + + # or using the vector from LiDAR to points, which is faster for data loading + # @staticmethod + # def get_normals(center, points): + # normals = points - center[None, :] + # norms = np.linalg.norm(normals, axis=1, keepdims=True) + # normals = normals / norms + # return normals + + def get_data_name(self, idx): + file_path = self.data_list[idx % len(self.data_list)] + dir_path, file_name = os.path.split(file_path) + sequence_name = os.path.basename(os.path.dirname(dir_path)) + frame_name = os.path.splitext(file_name)[0] + data_name = f"{sequence_name}_{frame_name}" + return data_name + + @staticmethod + def get_learning_map(ignore_index): + learning_map = { + 0: ignore_index, # "unlabeled" + 1: ignore_index, # "outlier" mapped to "unlabeled" --------------------------mapped + 10: 0, # "car" + 11: 1, # "bicycle" + 13: 4, # "bus" mapped to "other-vehicle" --------------------------mapped + 15: 2, # "motorcycle" + 16: 4, # "on-rails" mapped to "other-vehicle" ---------------------mapped + 18: 3, # "truck" + 20: 4, # "other-vehicle" + 30: 5, # "person" + 31: 6, # "bicyclist" + 32: 7, # "motorcyclist" + 40: 8, # "road" + 44: 9, # "parking" + 48: 10, # "sidewalk" + 49: 11, # "other-ground" + 50: 12, # "building" + 51: 13, # "fence" + 52: ignore_index, # "other-structure" mapped to "unlabeled" ------------------mapped + 60: 8, # "lane-marking" to "road" ---------------------------------mapped + 70: 14, # "vegetation" + 71: 15, # "trunk" + 72: 16, # "terrain" + 80: 17, # "pole" + 81: 18, # "traffic-sign" + 99: ignore_index, # "other-object" to "unlabeled" ----------------------------mapped + 252: 0, # "moving-car" to "car" ------------------------------------mapped + 253: 6, # "moving-bicyclist" to "bicyclist" ------------------------mapped + 254: 5, # "moving-person" to "person" ------------------------------mapped + 255: 7, # "moving-motorcyclist" to "motorcyclist" ------------------mapped + 256: 4, # "moving-on-rails" mapped to "other-vehicle" --------------mapped + 257: 4, # "moving-bus" mapped to "other-vehicle" -------------------mapped + 258: 3, # "moving-truck" to "truck" --------------------------------mapped + 259: 4, # "moving-other"-vehicle to "other-vehicle" ----------------mapped + } + return learning_map + + @staticmethod + def get_learning_map_inv(ignore_index): + learning_map_inv = { + ignore_index: ignore_index, # "unlabeled" + 0: 10, # "car" + 1: 11, # "bicycle" + 2: 15, # "motorcycle" + 3: 18, # "truck" + 4: 20, # "other-vehicle" + 5: 30, # "person" + 6: 31, # "bicyclist" + 7: 32, # "motorcyclist" + 8: 40, # "road" + 9: 44, # "parking" + 10: 48, # "sidewalk" + 11: 49, # "other-ground" + 12: 50, # "building" + 13: 51, # "fence" + 14: 70, # "vegetation" + 15: 71, # "trunk" + 16: 72, # "terrain" + 17: 80, # "pole" + 18: 81, # "traffic-sign" + } + return learning_map_inv diff --git a/pointcept/datasets/shapenet_part.py b/pointcept/datasets/shapenet_part.py new file mode 100644 index 0000000000000000000000000000000000000000..fbd8341937c46a59ee194dd5ddbac671dbe2fc62 --- /dev/null +++ b/pointcept/datasets/shapenet_part.py @@ -0,0 +1,159 @@ +""" +ShapeNet Part Dataset (Unmaintained) + +get processed shapenet part dataset +at "https://shapenet.cs.stanford.edu/media/shapenetcore_partanno_segmentation_benchmark_v0_normal.zip" + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import json +import torch +import numpy as np +from copy import deepcopy + +from pointcept.utils.logger import get_root_logger +from .builder import DATASETS +from .transform import Compose, TRANSFORMS +from .defaults import DefaultDataset + + +@DATASETS.register_module() +class ShapeNetPartDataset(DefaultDataset): + def __init__( + self, + if_color=False, + split="train", + data_root="data/shapenetcore_partanno_segmentation_benchmark_v0_normal", + transform=None, + test_mode=False, + test_cfg=None, + loop=1, + ): + super(ShapeNetPartDataset, self).__init__() + self.data_root = data_root + self.split = split + self.if_color = if_color + self.transform = Compose(transform) + self.loop = ( + loop if not test_mode else 1 + ) # force make loop = 1 while in test mode + self.test_mode = test_mode + self.test_cfg = test_cfg if test_mode else None + self.cache = {} + + # load categories file + self.categories = [] + self.category2part = { + "Airplane": [0, 1, 2, 3], + "Bag": [4, 5], + "Cap": [6, 7], + "Car": [8, 9, 10, 11], + "Chair": [12, 13, 14, 15], + "Earphone": [16, 17, 18], + "Guitar": [19, 20, 21], + "Knife": [22, 23], + "Lamp": [24, 25, 26, 27], + "Laptop": [28, 29], + "Motorbike": [30, 31, 32, 33, 34, 35], + "Mug": [36, 37], + "Pistol": [38, 39, 40], + "Rocket": [41, 42, 43], + "Skateboard": [44, 45, 46], + "Table": [47, 48, 49], + } + self.token2category = {} + with open(os.path.join(self.data_root, "synsetoffset2category.txt"), "r") as f: + for line in f: + ls = line.strip().split() + self.token2category[ls[1]] = len(self.categories) + self.categories.append(ls[0]) + + if test_mode: + self.test_voxelize = TRANSFORMS.build(self.test_cfg.voxelize) + self.test_crop = ( + TRANSFORMS.build(self.test_cfg.crop) if self.test_cfg.crop else None + ) + self.post_transform = Compose(self.test_cfg.post_transform) + self.aug_transform = [Compose(aug) for aug in self.test_cfg.aug_transform] + + # load data list + if isinstance(self.split, str): + self.data_list = self.load_data_list(self.split) + elif isinstance(self.split, list): + self.data_list = [] + for s in self.split: + self.data_list += self.load_data_list(s) + else: + raise NotImplementedError + + logger = get_root_logger() + logger.info( + "Totally {} x {} samples in {} set.".format( + len(self.data_list), self.loop, split + ) + ) + + def load_data_list(self, split): + split_file = os.path.join( + self.data_root, + "train_test_split", + "shuffled_{}_file_list.json".format(split), + ) + if not os.path.isfile(split_file): + raise (RuntimeError("Split file do not exist: " + split_file + "\n")) + with open(split_file, "r") as f: + # drop "shape_data/" and append ".txt" + data_list = [ + os.path.join(self.data_root, data[11:] + ".txt") + for data in json.load(f) + ] + return data_list + + def get_data(self, idx): + data_idx = idx % len(self.data_list) + if data_idx in self.cache: + coord, normal, segment, cls_token = self.cache[data_idx] + else: + data = np.loadtxt(self.data_list[data_idx]).astype(np.float32) + cls_token = self.token2category[ + os.path.basename(os.path.dirname(self.data_list[data_idx])) + ] + coord, normal, segment = ( + data[:, :3], + data[:, 3:6], + data[:, 6].astype(np.int32), + ) + self.cache[data_idx] = (coord, normal, segment, cls_token) + name = self.get_data_name(idx) + + if self.if_color: + color = np.zeros_like(coord) + data_dict = dict( + coord=coord, + color=color, + normal=normal, + segment=segment, + cls_token=cls_token, + name=name, + ) + else: + data_dict = dict(coord=coord, normal=normal, cls_token=cls_token, name=name) + return data_dict + + def get_data_name(self, idx): + data_idx = idx % len(self.data_list) + basename = os.path.basename(self.data_list[data_idx]).split(".")[0] + dirname = os.path.basename(os.path.dirname(self.data_list[data_idx])) + return dirname + "_" + basename + + def __getitem__(self, idx): + if self.test_mode: + return self.prepare_test_data(idx) + else: + return self.prepare_train_data(idx) + + def __len__(self): + return len(self.data_list) * self.loop diff --git a/pointcept/datasets/structure3d.py b/pointcept/datasets/structure3d.py new file mode 100644 index 0000000000000000000000000000000000000000..81649e3d83275f0419546f37da810e7508b327e0 --- /dev/null +++ b/pointcept/datasets/structure3d.py @@ -0,0 +1,38 @@ +""" +Structured3D Datasets + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import glob +from collections.abc import Sequence + +from .defaults import DefaultDataset +from .builder import DATASETS + + +@DATASETS.register_module() +class Structured3DDataset(DefaultDataset): + def get_data_list(self): + if isinstance(self.split, str): + data_list = glob.glob( + os.path.join(self.data_root, self.split, "scene_*/room_*") + ) + elif isinstance(self.split, Sequence): + data_list = [] + for split in self.split: + data_list += glob.glob( + os.path.join(self.data_root, split, "scene_*/room_*") + ) + else: + raise NotImplementedError + return data_list + + def get_data_name(self, idx): + file_path = self.data_list[idx % len(self.data_list)] + dir_path, room_name = os.path.split(file_path) + scene_name = os.path.basename(dir_path) + data_name = f"{scene_name}_{room_name}" + return data_name diff --git a/pointcept/datasets/transform.py b/pointcept/datasets/transform.py new file mode 100644 index 0000000000000000000000000000000000000000..19f8d94f9cb0c2f68c625eaa35f1287df63fa0ad --- /dev/null +++ b/pointcept/datasets/transform.py @@ -0,0 +1,1556 @@ +""" +3D point cloud augmentation + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import copy +import random +import numbers +import scipy +import scipy.ndimage +import scipy.interpolate +import numpy as np +import torch +from torchvision import transforms +from collections.abc import Sequence, Mapping +from pointcept.utils.registry import Registry + +TRANSFORMS = Registry("transforms") + + +def index_operator(data_dict, index, duplicate=False): + # index selection operator for keys in "index_valid_keys" + # custom these keys by "Update" transform in config + if "index_valid_keys" not in data_dict: + data_dict["index_valid_keys"] = [ + "coord", + "color", + "normal", + "superpoint", + "strength", + "segment", + "instance", + ] + if not duplicate: + for key in data_dict["index_valid_keys"]: + if key in data_dict: + data_dict[key] = data_dict[key][index] + return data_dict + else: + data_dict_ = dict() + for key in data_dict.keys(): + if key in data_dict["index_valid_keys"]: + data_dict_[key] = data_dict[key][index] + elif key == "index_valid_keys": + data_dict_[key] = copy.copy(data_dict[key]) + else: + data_dict_[key] = data_dict[key] + return data_dict_ + + +@TRANSFORMS.register_module() +class Collect(object): + def __init__(self, keys, offset_keys_dict=None, **kwargs): + """ + e.g. Collect(keys=[coord], feat_keys=[coord, color]) + """ + if offset_keys_dict is None: + offset_keys_dict = dict(offset="coord") + self.keys = keys + self.offset_keys = offset_keys_dict + self.kwargs = kwargs + + def __call__(self, data_dict): + data = dict() + data_dict["offset"] = torch.cumsum( + torch.tensor([data.shape[0] for data in data_dict["coord"]]), dim=0 + ) + if isinstance(self.keys, str): + self.keys = [self.keys] + for key in self.keys: + data[key] = data_dict[key] + for key, value in self.offset_keys.items(): + data[key] = torch.tensor([data_dict[value].shape[0]]) + for name, keys in self.kwargs.items(): + name = name.replace("_keys", "") + assert isinstance(keys, Sequence) + data[name] = torch.cat([data_dict[key].float() for key in keys], dim=1) + return data + + +@TRANSFORMS.register_module() +class Copy(object): + def __init__(self, keys_dict=None): + if keys_dict is None: + keys_dict = dict(coord="origin_coord", segment="origin_segment") + self.keys_dict = keys_dict + + def __call__(self, data_dict): + for key, value in self.keys_dict.items(): + if isinstance(data_dict[key], np.ndarray): + data_dict[value] = data_dict[key].copy() + elif isinstance(data_dict[key], torch.Tensor): + data_dict[value] = data_dict[key].clone().detach() + else: + data_dict[value] = copy.deepcopy(data_dict[key]) + return data_dict + + +@TRANSFORMS.register_module() +class Update(object): + def __init__(self, keys_dict=None): + if keys_dict is None: + keys_dict = dict() + self.keys_dict = keys_dict + + def __call__(self, data_dict): + for key, value in self.keys_dict.items(): + data_dict[key] = value + return data_dict + + +@TRANSFORMS.register_module() +class ToTensor(object): + def __call__(self, data): + if isinstance(data, torch.Tensor): + return data + elif isinstance(data, str): + # note that str is also a kind of sequence, judgement should before sequence + return data + elif isinstance(data, int): + return torch.LongTensor([data]) + elif isinstance(data, float): + return torch.FloatTensor([data]) + elif isinstance(data, np.ndarray) and np.issubdtype(data.dtype, bool): + return torch.from_numpy(data) + elif isinstance(data, np.ndarray) and np.issubdtype(data.dtype, np.integer): + return torch.from_numpy(data).long() + elif isinstance(data, np.ndarray) and np.issubdtype(data.dtype, np.floating): + return torch.from_numpy(data).float() + elif isinstance(data, Mapping): + result = {sub_key: self(item) for sub_key, item in data.items()} + return result + elif isinstance(data, Sequence): + result = [self(item) for item in data] + return result + else: + raise TypeError(f"type {type(data)} cannot be converted to tensor.") + + +@TRANSFORMS.register_module() +class NormalizeColor(object): + def __call__(self, data_dict): + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"] / 255 + return data_dict + + +@TRANSFORMS.register_module() +class NormalizeCoord(object): + def __call__(self, data_dict): + if "coord" in data_dict.keys(): + # modified from pointnet2 + centroid = np.mean(data_dict["coord"], axis=0) + data_dict["coord"] -= centroid + m = np.max(np.sqrt(np.sum(data_dict["coord"] ** 2, axis=1))) + data_dict["coord"] = data_dict["coord"] / m + return data_dict + + +@TRANSFORMS.register_module() +class PositiveShift(object): + def __call__(self, data_dict): + if "coord" in data_dict.keys(): + coord_min = np.min(data_dict["coord"], 0) + data_dict["coord"] -= coord_min + return data_dict + + +@TRANSFORMS.register_module() +class CenterShift(object): + def __init__(self, apply_z=True): + self.apply_z = apply_z + + def __call__(self, data_dict): + if "coord" in data_dict.keys(): + x_min, y_min, z_min = data_dict["coord"].min(axis=0) + x_max, y_max, _ = data_dict["coord"].max(axis=0) + if self.apply_z: + shift = [(x_min + x_max) / 2, (y_min + y_max) / 2, z_min] + else: + shift = [(x_min + x_max) / 2, (y_min + y_max) / 2, 0] + data_dict["coord"] -= shift + return data_dict + + +@TRANSFORMS.register_module() +class RandomShift(object): + def __init__(self, shift=((-0.2, 0.2), (-0.2, 0.2), (0, 0))): + self.shift = shift + + def __call__(self, data_dict): + if "coord" in data_dict.keys(): + shift_x = np.random.uniform(self.shift[0][0], self.shift[0][1]) + shift_y = np.random.uniform(self.shift[1][0], self.shift[1][1]) + shift_z = np.random.uniform(self.shift[2][0], self.shift[2][1]) + data_dict["coord"] += [shift_x, shift_y, shift_z] + return data_dict + + +@TRANSFORMS.register_module() +class PointClip(object): + def __init__(self, point_cloud_range=(-80, -80, -3, 80, 80, 1)): + self.point_cloud_range = point_cloud_range + + def __call__(self, data_dict): + if "coord" in data_dict.keys(): + data_dict["coord"] = np.clip( + data_dict["coord"], + a_min=self.point_cloud_range[:3], + a_max=self.point_cloud_range[3:], + ) + return data_dict + + +@TRANSFORMS.register_module() +class RandomDropout(object): + def __init__(self, dropout_ratio=0.2, dropout_application_ratio=0.5): + """ + upright_axis: axis index among x,y,z, i.e. 2 for z + """ + self.dropout_ratio = dropout_ratio + self.dropout_application_ratio = dropout_application_ratio + + def __call__(self, data_dict): + if random.random() < self.dropout_application_ratio: + n = len(data_dict["coord"]) + idx = np.random.choice(n, int(n * (1 - self.dropout_ratio)), replace=False) + if "sampled_index" in data_dict: + # for ScanNet data efficient, we need to make sure labeled point is sampled. + idx = np.unique(np.append(idx, data_dict["sampled_index"])) + mask = np.zeros_like(data_dict["segment"]).astype(bool) + mask[data_dict["sampled_index"]] = True + data_dict["sampled_index"] = np.where(mask[idx])[0] + data_dict = index_operator(data_dict, idx) + return data_dict + + +@TRANSFORMS.register_module() +class RandomRotate(object): + def __init__( + self, + angle=None, + center=None, + axis="z", + always_apply=False, + p=0.5, + ): + self.angle = [-1, 1] if angle is None else angle + self.axis = axis + self.always_apply = always_apply + self.p = p if not self.always_apply else 1 + self.center = center + + def __call__(self, data_dict): + if random.random() > self.p: + return data_dict + angle = np.random.uniform(self.angle[0], self.angle[1]) * np.pi + rot_cos, rot_sin = np.cos(angle), np.sin(angle) + if self.axis == "x": + rot_t = np.array([[1, 0, 0], [0, rot_cos, -rot_sin], [0, rot_sin, rot_cos]]) + elif self.axis == "y": + rot_t = np.array([[rot_cos, 0, rot_sin], [0, 1, 0], [-rot_sin, 0, rot_cos]]) + elif self.axis == "z": + rot_t = np.array([[rot_cos, -rot_sin, 0], [rot_sin, rot_cos, 0], [0, 0, 1]]) + else: + raise NotImplementedError + if "coord" in data_dict.keys(): + if self.center is None: + x_min, y_min, z_min = data_dict["coord"].min(axis=0) + x_max, y_max, z_max = data_dict["coord"].max(axis=0) + center = [(x_min + x_max) / 2, (y_min + y_max) / 2, (z_min + z_max) / 2] + else: + center = self.center + data_dict["coord"] -= center + data_dict["coord"] = np.dot(data_dict["coord"], np.transpose(rot_t)) + data_dict["coord"] += center + if "normal" in data_dict.keys(): + data_dict["normal"] = np.dot(data_dict["normal"], np.transpose(rot_t)) + return data_dict + + +@TRANSFORMS.register_module() +class RandomRotateTargetAngle(object): + def __init__( + self, angle=(1 / 2, 1, 3 / 2), center=None, axis="z", always_apply=False, p=0.75 + ): + self.angle = angle + self.axis = axis + self.always_apply = always_apply + self.p = p if not self.always_apply else 1 + self.center = center + + def __call__(self, data_dict): + if random.random() > self.p: + return data_dict + angle = np.random.choice(self.angle) * np.pi + rot_cos, rot_sin = np.cos(angle), np.sin(angle) + if self.axis == "x": + rot_t = np.array([[1, 0, 0], [0, rot_cos, -rot_sin], [0, rot_sin, rot_cos]]) + elif self.axis == "y": + rot_t = np.array([[rot_cos, 0, rot_sin], [0, 1, 0], [-rot_sin, 0, rot_cos]]) + elif self.axis == "z": + rot_t = np.array([[rot_cos, -rot_sin, 0], [rot_sin, rot_cos, 0], [0, 0, 1]]) + else: + raise NotImplementedError + if "coord" in data_dict.keys(): + if self.center is None: + x_min, y_min, z_min = data_dict["coord"].min(axis=0) + x_max, y_max, z_max = data_dict["coord"].max(axis=0) + center = [(x_min + x_max) / 2, (y_min + y_max) / 2, (z_min + z_max) / 2] + else: + center = self.center + data_dict["coord"] -= center + data_dict["coord"] = np.dot(data_dict["coord"], np.transpose(rot_t)) + data_dict["coord"] += center + if "normal" in data_dict.keys(): + data_dict["normal"] = np.dot(data_dict["normal"], np.transpose(rot_t)) + return data_dict + + +@TRANSFORMS.register_module() +class RandomScale(object): + def __init__(self, scale=None, anisotropic=False): + self.scale = scale if scale is not None else [0.95, 1.05] + self.anisotropic = anisotropic + + def __call__(self, data_dict): + if "coord" in data_dict.keys(): + scale = np.random.uniform( + self.scale[0], self.scale[1], 3 if self.anisotropic else 1 + ) + data_dict["coord"] *= scale + return data_dict + + +@TRANSFORMS.register_module() +class RandomFlip(object): + def __init__(self, p=0.5): + self.p = p + + def __call__(self, data_dict): + if np.random.rand() < self.p: + if "coord" in data_dict.keys(): + data_dict["coord"][:, 0] = -data_dict["coord"][:, 0] + if "normal" in data_dict.keys(): + data_dict["normal"][:, 0] = -data_dict["normal"][:, 0] + if np.random.rand() < self.p: + if "coord" in data_dict.keys(): + data_dict["coord"][:, 1] = -data_dict["coord"][:, 1] + if "normal" in data_dict.keys(): + data_dict["normal"][:, 1] = -data_dict["normal"][:, 1] + return data_dict + + +@TRANSFORMS.register_module() +class RandomJitter(object): + def __init__(self, sigma=0.01, clip=0.05): + assert clip > 0 + self.sigma = sigma + self.clip = clip + + def __call__(self, data_dict): + if "coord" in data_dict.keys(): + jitter = np.clip( + self.sigma * np.random.randn(data_dict["coord"].shape[0], 3), + -self.clip, + self.clip, + ) + data_dict["coord"] += jitter + return data_dict + + +@TRANSFORMS.register_module() +class ClipGaussianJitter(object): + def __init__(self, scalar=0.02, store_jitter=False): + self.scalar = scalar + self.mean = np.mean(3) + self.cov = np.identity(3) + self.quantile = 1.96 + self.store_jitter = store_jitter + + def __call__(self, data_dict): + if "coord" in data_dict.keys(): + jitter = np.random.multivariate_normal( + self.mean, self.cov, data_dict["coord"].shape[0] + ) + jitter = self.scalar * np.clip(jitter / 1.96, -1, 1) + data_dict["coord"] += jitter + if self.store_jitter: + data_dict["jitter"] = jitter + return data_dict + + +@TRANSFORMS.register_module() +class ChromaticAutoContrast(object): + def __init__(self, p=0.2, blend_factor=None): + self.p = p + self.blend_factor = blend_factor + + def __call__(self, data_dict): + if "color" in data_dict.keys() and np.random.rand() < self.p: + lo = np.min(data_dict["color"], 0, keepdims=True) + hi = np.max(data_dict["color"], 0, keepdims=True) + diff = hi - lo + if not np.any(diff > 0): + return data_dict + scale = np.divide( + 255, + diff, + out=np.ones_like(diff, dtype=data_dict["color"].dtype), + where=diff > 0, + ) + contrast_feat = (data_dict["color"][:, :3] - lo) * scale + blend_factor = ( + np.random.rand() if self.blend_factor is None else self.blend_factor + ) + data_dict["color"][:, :3] = (1 - blend_factor) * data_dict["color"][ + :, :3 + ] + blend_factor * contrast_feat + return data_dict + + +@TRANSFORMS.register_module() +class ChromaticTranslation(object): + def __init__(self, p=0.95, ratio=0.05): + self.p = p + self.ratio = ratio + + def __call__(self, data_dict): + if "color" in data_dict.keys() and np.random.rand() < self.p: + tr = (np.random.rand(1, 3) - 0.5) * 255 * 2 * self.ratio + data_dict["color"][:, :3] = np.clip(tr + data_dict["color"][:, :3], 0, 255) + return data_dict + + +@TRANSFORMS.register_module() +class ChromaticJitter(object): + def __init__(self, p=0.95, std=0.005): + self.p = p + self.std = std + + def __call__(self, data_dict): + if "color" in data_dict.keys() and np.random.rand() < self.p: + noise = np.random.randn(data_dict["color"].shape[0], 3) + noise *= self.std * 255 + data_dict["color"][:, :3] = np.clip( + noise + data_dict["color"][:, :3], 0, 255 + ) + return data_dict + + +@TRANSFORMS.register_module() +class RandomColorGrayScale(object): + def __init__(self, p): + self.p = p + + @staticmethod + def rgb_to_grayscale(color, num_output_channels=1): + if color.shape[-1] < 3: + raise TypeError( + "Input color should have at least 3 dimensions, but found {}".format( + color.shape[-1] + ) + ) + + if num_output_channels not in (1, 3): + raise ValueError("num_output_channels should be either 1 or 3") + + r, g, b = color[..., 0], color[..., 1], color[..., 2] + gray = (0.2989 * r + 0.587 * g + 0.114 * b).astype(color.dtype) + gray = np.expand_dims(gray, axis=-1) + + if num_output_channels == 3: + gray = np.broadcast_to(gray, color.shape) + + return gray + + def __call__(self, data_dict): + if np.random.rand() < self.p: + data_dict["color"] = self.rgb_to_grayscale(data_dict["color"], 3) + return data_dict + + +@TRANSFORMS.register_module() +class RandomColorJitter(object): + """ + Random Color Jitter for 3D point cloud (refer torchvision) + """ + + def __init__(self, brightness=0, contrast=0, saturation=0, hue=0, p=0.95): + self.brightness = self._check_input(brightness, "brightness") + self.contrast = self._check_input(contrast, "contrast") + self.saturation = self._check_input(saturation, "saturation") + self.hue = self._check_input( + hue, "hue", center=0, bound=(-0.5, 0.5), clip_first_on_zero=False + ) + self.p = p + + @staticmethod + def _check_input( + value, name, center=1, bound=(0, float("inf")), clip_first_on_zero=True + ): + if isinstance(value, numbers.Number): + if value < 0: + raise ValueError( + "If {} is a single number, it must be non negative.".format(name) + ) + value = [center - float(value), center + float(value)] + if clip_first_on_zero: + value[0] = max(value[0], 0.0) + elif isinstance(value, (tuple, list)) and len(value) == 2: + if not bound[0] <= value[0] <= value[1] <= bound[1]: + raise ValueError("{} values should be between {}".format(name, bound)) + else: + raise TypeError( + "{} should be a single number or a list/tuple with length 2.".format( + name + ) + ) + + # if value is 0 or (1., 1.) for brightness/contrast/saturation + # or (0., 0.) for hue, do nothing + if value[0] == value[1] == center: + value = None + return value + + @staticmethod + def blend(color1, color2, ratio): + ratio = float(ratio) + bound = 255.0 + return ( + (ratio * color1 + (1.0 - ratio) * color2) + .clip(0, bound) + .astype(color1.dtype) + ) + + @staticmethod + def rgb2hsv(rgb): + r, g, b = rgb[..., 0], rgb[..., 1], rgb[..., 2] + maxc = np.max(rgb, axis=-1) + minc = np.min(rgb, axis=-1) + eqc = maxc == minc + cr = maxc - minc + s = cr / (np.ones_like(maxc) * eqc + maxc * (1 - eqc)) + cr_divisor = np.ones_like(maxc) * eqc + cr * (1 - eqc) + rc = (maxc - r) / cr_divisor + gc = (maxc - g) / cr_divisor + bc = (maxc - b) / cr_divisor + + hr = (maxc == r) * (bc - gc) + hg = ((maxc == g) & (maxc != r)) * (2.0 + rc - bc) + hb = ((maxc != g) & (maxc != r)) * (4.0 + gc - rc) + h = hr + hg + hb + h = (h / 6.0 + 1.0) % 1.0 + return np.stack((h, s, maxc), axis=-1) + + @staticmethod + def hsv2rgb(hsv): + h, s, v = hsv[..., 0], hsv[..., 1], hsv[..., 2] + i = np.floor(h * 6.0) + f = (h * 6.0) - i + i = i.astype(np.int32) + + p = np.clip((v * (1.0 - s)), 0.0, 1.0) + q = np.clip((v * (1.0 - s * f)), 0.0, 1.0) + t = np.clip((v * (1.0 - s * (1.0 - f))), 0.0, 1.0) + i = i % 6 + mask = np.expand_dims(i, axis=-1) == np.arange(6) + + a1 = np.stack((v, q, p, p, t, v), axis=-1) + a2 = np.stack((t, v, v, q, p, p), axis=-1) + a3 = np.stack((p, p, t, v, v, q), axis=-1) + a4 = np.stack((a1, a2, a3), axis=-1) + + return np.einsum("...na, ...nab -> ...nb", mask.astype(hsv.dtype), a4) + + def adjust_brightness(self, color, brightness_factor): + if brightness_factor < 0: + raise ValueError( + "brightness_factor ({}) is not non-negative.".format(brightness_factor) + ) + + return self.blend(color, np.zeros_like(color), brightness_factor) + + def adjust_contrast(self, color, contrast_factor): + if contrast_factor < 0: + raise ValueError( + "contrast_factor ({}) is not non-negative.".format(contrast_factor) + ) + mean = np.mean(RandomColorGrayScale.rgb_to_grayscale(color)) + return self.blend(color, mean, contrast_factor) + + def adjust_saturation(self, color, saturation_factor): + if saturation_factor < 0: + raise ValueError( + "saturation_factor ({}) is not non-negative.".format(saturation_factor) + ) + gray = RandomColorGrayScale.rgb_to_grayscale(color) + return self.blend(color, gray, saturation_factor) + + def adjust_hue(self, color, hue_factor): + if not (-0.5 <= hue_factor <= 0.5): + raise ValueError( + "hue_factor ({}) is not in [-0.5, 0.5].".format(hue_factor) + ) + orig_dtype = color.dtype + hsv = self.rgb2hsv(color / 255.0) + h, s, v = hsv[..., 0], hsv[..., 1], hsv[..., 2] + h = (h + hue_factor) % 1.0 + hsv = np.stack((h, s, v), axis=-1) + color_hue_adj = (self.hsv2rgb(hsv) * 255.0).astype(orig_dtype) + return color_hue_adj + + @staticmethod + def get_params(brightness, contrast, saturation, hue): + fn_idx = torch.randperm(4) + b = ( + None + if brightness is None + else np.random.uniform(brightness[0], brightness[1]) + ) + c = None if contrast is None else np.random.uniform(contrast[0], contrast[1]) + s = ( + None + if saturation is None + else np.random.uniform(saturation[0], saturation[1]) + ) + h = None if hue is None else np.random.uniform(hue[0], hue[1]) + return fn_idx, b, c, s, h + + def __call__(self, data_dict): + ( + fn_idx, + brightness_factor, + contrast_factor, + saturation_factor, + hue_factor, + ) = self.get_params(self.brightness, self.contrast, self.saturation, self.hue) + + for fn_id in fn_idx: + if ( + fn_id == 0 + and brightness_factor is not None + and np.random.rand() < self.p + ): + data_dict["color"] = self.adjust_brightness( + data_dict["color"], brightness_factor + ) + elif ( + fn_id == 1 and contrast_factor is not None and np.random.rand() < self.p + ): + data_dict["color"] = self.adjust_contrast( + data_dict["color"], contrast_factor + ) + elif ( + fn_id == 2 + and saturation_factor is not None + and np.random.rand() < self.p + ): + data_dict["color"] = self.adjust_saturation( + data_dict["color"], saturation_factor + ) + elif fn_id == 3 and hue_factor is not None and np.random.rand() < self.p: + data_dict["color"] = self.adjust_hue(data_dict["color"], hue_factor) + return data_dict + + +@TRANSFORMS.register_module() +class HueSaturationTranslation(object): + @staticmethod + def rgb_to_hsv(rgb): + # Translated from source of colorsys.rgb_to_hsv + # r,g,b should be a numpy arrays with values between 0 and 255 + # rgb_to_hsv returns an array of floats between 0.0 and 1.0. + rgb = rgb.astype("float") + hsv = np.zeros_like(rgb) + # in case an RGBA array was passed, just copy the A channel + hsv[..., 3:] = rgb[..., 3:] + r, g, b = rgb[..., 0], rgb[..., 1], rgb[..., 2] + maxc = np.max(rgb[..., :3], axis=-1) + minc = np.min(rgb[..., :3], axis=-1) + hsv[..., 2] = maxc + mask = maxc != minc + hsv[mask, 1] = (maxc - minc)[mask] / maxc[mask] + rc = np.zeros_like(r) + gc = np.zeros_like(g) + bc = np.zeros_like(b) + rc[mask] = (maxc - r)[mask] / (maxc - minc)[mask] + gc[mask] = (maxc - g)[mask] / (maxc - minc)[mask] + bc[mask] = (maxc - b)[mask] / (maxc - minc)[mask] + hsv[..., 0] = np.select( + [r == maxc, g == maxc], [bc - gc, 2.0 + rc - bc], default=4.0 + gc - rc + ) + hsv[..., 0] = (hsv[..., 0] / 6.0) % 1.0 + return hsv + + @staticmethod + def hsv_to_rgb(hsv): + # Translated from source of colorsys.hsv_to_rgb + # h,s should be a numpy arrays with values between 0.0 and 1.0 + # v should be a numpy array with values between 0.0 and 255.0 + # hsv_to_rgb returns an array of uints between 0 and 255. + rgb = np.empty_like(hsv) + rgb[..., 3:] = hsv[..., 3:] + h, s, v = hsv[..., 0], hsv[..., 1], hsv[..., 2] + i = (h * 6.0).astype("uint8") + f = (h * 6.0) - i + p = v * (1.0 - s) + q = v * (1.0 - s * f) + t = v * (1.0 - s * (1.0 - f)) + i = i % 6 + conditions = [s == 0.0, i == 1, i == 2, i == 3, i == 4, i == 5] + rgb[..., 0] = np.select(conditions, [v, q, p, p, t, v], default=v) + rgb[..., 1] = np.select(conditions, [v, v, v, q, p, p], default=t) + rgb[..., 2] = np.select(conditions, [v, p, t, v, v, q], default=p) + return rgb.astype("uint8") + + def __init__(self, hue_max=0.5, saturation_max=0.2): + self.hue_max = hue_max + self.saturation_max = saturation_max + + def __call__(self, data_dict): + if "color" in data_dict.keys(): + # Assume color[:, :3] is rgb + hsv = HueSaturationTranslation.rgb_to_hsv(data_dict["color"][:, :3]) + hue_val = (np.random.rand() - 0.5) * 2 * self.hue_max + sat_ratio = 1 + (np.random.rand() - 0.5) * 2 * self.saturation_max + hsv[..., 0] = np.remainder(hue_val + hsv[..., 0] + 1, 1) + hsv[..., 1] = np.clip(sat_ratio * hsv[..., 1], 0, 1) + data_dict["color"][:, :3] = np.clip( + HueSaturationTranslation.hsv_to_rgb(hsv), 0, 255 + ) + return data_dict + + +@TRANSFORMS.register_module() +class RandomDropColor(object): + def __init__(self, drop_ratio=0.2, drop_application_ratio=0.5): + """ + upright_axis: axis index among x,y,z, i.e. 2 for z + """ + self.drop_ratio = drop_ratio + self.drop_application_ratio = drop_application_ratio + self.drop_value = 0.0 + + def __call__(self, data_dict): + if ( + "color" in data_dict.keys() + and random.random() < self.drop_application_ratio + ): + n = len(data_dict["color"]) + idx = np.random.choice(n, int(n * self.drop_ratio), replace=False) + data_dict["color"][idx] = self.drop_value + return data_dict + + +@TRANSFORMS.register_module() +class RandomDropNormal(object): + def __init__(self, drop_ratio=0.2, drop_application_ratio=0.5): + self.drop_ratio = drop_ratio + self.drop_application_ratio = drop_application_ratio + self.drop_value = 0.0 + + def __call__(self, data_dict): + if ( + "normal" in data_dict.keys() + and random.random() < self.drop_application_ratio + ): + n = len(data_dict["normal"]) + num_to_drop = int(n * self.drop_ratio) + idx = np.random.choice(n, num_to_drop, replace=False) + data_dict["normal"][idx] = self.drop_value + return data_dict + + +@TRANSFORMS.register_module() +class ElasticDistortion(object): + def __init__(self, distortion_params=None): + self.distortion_params = ( + [[0.2, 0.4], [0.8, 1.6]] if distortion_params is None else distortion_params + ) + + @staticmethod + def elastic_distortion(coords, granularity, magnitude): + """ + Apply elastic distortion on sparse coordinate space. + pointcloud: numpy array of (number of points, at least 3 spatial dims) + granularity: size of the noise grid (in same scale[m/cm] as the voxel grid) + magnitude: noise multiplier + """ + blurx = np.ones((3, 1, 1, 1)).astype("float32") / 3 + blury = np.ones((1, 3, 1, 1)).astype("float32") / 3 + blurz = np.ones((1, 1, 3, 1)).astype("float32") / 3 + coords_min = coords.min(0) + + # Create Gaussian noise tensor of the size given by granularity. + noise_dim = ((coords - coords_min).max(0) // granularity).astype(int) + 3 + noise = np.random.randn(*noise_dim, 3).astype(np.float32) + + # Smoothing. + for _ in range(2): + noise = scipy.ndimage.filters.convolve( + noise, blurx, mode="constant", cval=0 + ) + noise = scipy.ndimage.filters.convolve( + noise, blury, mode="constant", cval=0 + ) + noise = scipy.ndimage.filters.convolve( + noise, blurz, mode="constant", cval=0 + ) + + # Trilinear interpolate noise filters for each spatial dimensions. + ax = [ + np.linspace(d_min, d_max, d) + for d_min, d_max, d in zip( + coords_min - granularity, + coords_min + granularity * (noise_dim - 2), + noise_dim, + ) + ] + interp = scipy.interpolate.RegularGridInterpolator( + ax, noise, bounds_error=False, fill_value=0 + ) + coords += interp(coords) * magnitude + return coords + + def __call__(self, data_dict): + if "coord" in data_dict.keys() and self.distortion_params is not None: + if random.random() < 0.95: + for granularity, magnitude in self.distortion_params: + data_dict["coord"] = self.elastic_distortion( + data_dict["coord"], granularity, magnitude + ) + return data_dict + + +@TRANSFORMS.register_module() +class GridSample(object): + def __init__( + self, + grid_size=0.05, + hash_type="fnv", + mode="train", + return_inverse=False, + return_grid_coord=False, + return_min_coord=False, + return_displacement=False, + project_displacement=False, + ): + self.grid_size = grid_size + self.hash = self.fnv_hash_vec if hash_type == "fnv" else self.ravel_hash_vec + assert mode in ["train", "test"] + self.mode = mode + self.return_inverse = return_inverse + self.return_grid_coord = return_grid_coord + self.return_min_coord = return_min_coord + self.return_displacement = return_displacement + self.project_displacement = project_displacement + + def __call__(self, data_dict): + assert "coord" in data_dict.keys() + scaled_coord = data_dict["coord"] / np.array(self.grid_size) + grid_coord = np.floor(scaled_coord).astype(int) + min_coord = grid_coord.min(0) + grid_coord -= min_coord + scaled_coord -= min_coord + min_coord = min_coord * np.array(self.grid_size) + key = self.hash(grid_coord) + idx_sort = np.argsort(key) + key_sort = key[idx_sort] + _, inverse, count = np.unique(key_sort, return_inverse=True, return_counts=True) + if self.mode == "train": # train mode + idx_select = ( + np.cumsum(np.insert(count, 0, 0)[0:-1]) + + np.random.randint(0, count.max(), count.size) % count + ) + idx_unique = idx_sort[idx_select] + if "sampled_index" in data_dict: + # for ScanNet data efficient, we need to make sure labeled point is sampled. + idx_unique = np.unique( + np.append(idx_unique, data_dict["sampled_index"]) + ) + mask = np.zeros_like(data_dict["segment"]).astype(bool) + mask[data_dict["sampled_index"]] = True + data_dict["sampled_index"] = np.where(mask[idx_unique])[0] + data_dict = index_operator(data_dict, idx_unique) + if "frame_pcd_offset" in data_dict: + data_dict["frame_pcd_offset"] = self._recompute_offsets( + data_dict["frame_pcd_offset"], idx_unique + ) + if self.return_inverse: + data_dict["inverse"] = np.zeros_like(inverse) + data_dict["inverse"][idx_sort] = inverse + if self.return_grid_coord: + data_dict["grid_coord"] = grid_coord[idx_unique] + if "grid_coord" not in data_dict["index_valid_keys"]: + data_dict["index_valid_keys"] = list(data_dict["index_valid_keys"]) + data_dict["index_valid_keys"].append("grid_coord") + if self.return_min_coord: + data_dict["min_coord"] = min_coord.reshape([1, 3]) + if self.return_displacement: + displacement = ( + scaled_coord - grid_coord - 0.5 + ) # [0, 1] -> [-0.5, 0.5] displacement to center + if self.project_displacement: + displacement = np.sum( + displacement * data_dict["normal"], axis=-1, keepdims=True + ) + data_dict["displacement"] = displacement[idx_unique] + if "displacement" not in data_dict["index_valid_keys"]: + data_dict["index_valid_keys"].append("displacement") + return data_dict + + elif self.mode == "test": # test mode + data_part_list = [] + for i in range(count.max()): + idx_select = np.cumsum(np.insert(count, 0, 0)[0:-1]) + i % count + idx_part = idx_sort[idx_select] + data_part = index_operator(data_dict, idx_part, duplicate=True) + data_part["index"] = idx_part + if "frame_pcd_offset" in data_dict: + data_part["frame_pcd_offset"] = self._recompute_offsets( + data_dict["frame_pcd_offset"], idx_part + ) + if self.return_inverse: + data_part["inverse"] = np.zeros_like(inverse) + data_part["inverse"][idx_sort] = inverse + if self.return_grid_coord: + data_part["grid_coord"] = grid_coord[idx_part] + if "grid_coord" not in data_part["index_valid_keys"]: + data_part["index_valid_keys"] = list( + data_part["index_valid_keys"] + ) + data_part["index_valid_keys"].append("grid_coord") + if self.return_min_coord: + data_part["min_coord"] = min_coord.reshape([1, 3]) + if self.return_displacement: + displacement = ( + scaled_coord - grid_coord - 0.5 + ) # [0, 1] -> [-0.5, 0.5] displacement to center + if self.project_displacement: + displacement = np.sum( + displacement * data_dict["normal"], axis=-1, keepdims=True + ) + data_part["displacement"] = displacement[idx_part] + if "displacement" not in data_part["index_valid_keys"]: + data_part["index_valid_keys"].append("displacement") + data_part_list.append(data_part) + return data_part_list + else: + raise NotImplementedError + + @staticmethod + def _recompute_offsets(origin_offsets, sampled_indices): + """ + Args: + origin_offsets (np.ndarray) [N1, N1+N2, ...] + sampled_indices (np.ndarray) + + Returns: + np.ndarray: [C1, C1+C2, ...]。 + """ + if len(origin_offsets) == 0: + return np.array([], dtype=origin_offsets.dtype) + + standard_format_offsets = np.insert(origin_offsets, 0, 0) + frame_assignment = np.searchsorted( + standard_format_offsets, sampled_indices, side="right" + ) + new_offset_dict = {} + for frame_id in range(1, len(origin_offsets) + 1): + new_offset_dict[frame_id - 1] = np.where(frame_assignment == frame_id)[0] + return new_offset_dict + + @staticmethod + def ravel_hash_vec(arr): + """ + Ravel the coordinates after subtracting the min coordinates. + """ + assert arr.ndim == 2 + arr = arr.copy() + arr -= arr.min(0) + arr = arr.astype(np.uint64, copy=False) + arr_max = arr.max(0).astype(np.uint64) + 1 + + keys = np.zeros(arr.shape[0], dtype=np.uint64) + # Fortran style indexing + for j in range(arr.shape[1] - 1): + keys += arr[:, j] + keys *= arr_max[j + 1] + keys += arr[:, -1] + return keys + + @staticmethod + def fnv_hash_vec(arr): + """ + FNV64-1A + """ + assert arr.ndim == 2 + # Floor first for negative coordinates + arr = arr.copy() + arr = arr.astype(np.uint64, copy=False) + hashed_arr = np.uint64(14695981039346656037) * np.ones( + arr.shape[0], dtype=np.uint64 + ) + for j in range(arr.shape[1]): + hashed_arr *= np.uint64(1099511628211) + hashed_arr = np.bitwise_xor(hashed_arr, arr[:, j]) + return hashed_arr + + +@TRANSFORMS.register_module() +class SphereCrop(object): + def __init__(self, point_max=80000, sample_rate=None, mode="random"): + self.point_max = point_max + self.sample_rate = sample_rate + assert mode in ["random", "center", "all", "given"] + self.mode = mode + + def __call__(self, data_dict): + point_max = ( + int(self.sample_rate * data_dict["coord"].shape[0]) + if self.sample_rate is not None + else self.point_max + ) + + assert "coord" in data_dict.keys() + if data_dict["coord"].shape[0] > point_max: + if self.mode == "random": + center = data_dict["coord"][ + np.random.randint(data_dict["coord"].shape[0]) + ] + elif self.mode == "center": + center = data_dict["coord"][data_dict["coord"].shape[0] // 2] + elif self.mode == "given": + given_index = data_dict["correspondence"].reshape( + data_dict["correspondence"].shape[0], -1 + ) + given_index = np.all( + given_index != np.ones_like(given_index[0]) * -1, axis=1 + ) + given_coord = data_dict["coord"][given_index] + if given_coord.shape[0] == 0: + center = data_dict["coord"][ + np.random.randint(data_dict["coord"].shape[0]) + ] + else: + center = np.mean(given_coord, axis=0) + else: + raise NotImplementedError + idx_crop = np.argsort(np.sum(np.square(data_dict["coord"] - center), 1))[ + :point_max + ] + data_dict = index_operator(data_dict, idx_crop) + return data_dict + + +@TRANSFORMS.register_module() +class ShufflePoint(object): + def __call__(self, data_dict): + assert "coord" in data_dict.keys() + shuffle_index = np.arange(data_dict["coord"].shape[0]) + np.random.shuffle(shuffle_index) + data_dict = index_operator(data_dict, shuffle_index) + return data_dict + + +@TRANSFORMS.register_module() +class CropBoundary(object): + def __call__(self, data_dict): + assert "segment" in data_dict + segment = data_dict["segment"].flatten() + mask = (segment != 0) * (segment != 1) + data_dict = index_operator(data_dict, mask) + return data_dict + + +@TRANSFORMS.register_module() +class ContrastiveViewsGenerator(object): + def __init__( + self, + view_keys=("coord", "color", "normal", "origin_coord"), + view_trans_cfg=None, + ): + self.view_keys = view_keys + self.view_trans = Compose(view_trans_cfg) + + def __call__(self, data_dict): + view1_dict = dict() + view2_dict = dict() + for key in self.view_keys: + view1_dict[key] = data_dict[key].copy() + view2_dict[key] = data_dict[key].copy() + view1_dict = self.view_trans(view1_dict) + view2_dict = self.view_trans(view2_dict) + for key, value in view1_dict.items(): + data_dict["view1_" + key] = value + for key, value in view2_dict.items(): + data_dict["view2_" + key] = value + return data_dict + + +@TRANSFORMS.register_module() +class MultiViewGenerator(object): + def __init__( + self, + global_view_num=2, + global_view_scale=(0.4, 1.0), + local_view_num=4, + local_view_scale=(0.1, 0.4), + global_shared_transform=None, + global_transform=None, + local_transform=None, + max_size=65536, + enc2d_max_size=102400, + enc2d_scale=(0.8, 1), + center_height_scale=(0, 1), + shared_global_view=False, + view_keys=("coord", "origin_coord", "color", "normal", "correspondence"), + static_view_keys=("name", "img_num"), + if_frame_selected=False, + ): + self.global_view_num = global_view_num + self.global_view_scale = global_view_scale + self.local_view_num = local_view_num + self.local_view_scale = local_view_scale + self.global_shared_transform = Compose(global_shared_transform) + self.global_transform = Compose(global_transform) + self.local_transform = Compose(local_transform) + self.max_size = max_size + self.enc2d_max_size = enc2d_max_size + self.enc2d_scale = enc2d_scale + self.center_height_scale = center_height_scale + self.shared_global_view = shared_global_view + self.view_keys = view_keys + self.static_view_keys = static_view_keys + self.if_frame_selected = if_frame_selected + assert "coord" in view_keys + + def get_view( + self, point, center, scale, if_enc2d=False, frame_num=1, if_frame_selected=False + ): + coord = point["coord"] + max_size = min(self.max_size, coord.shape[0]) + enc2d_max_size = min(self.enc2d_max_size, coord.shape[0]) + size = 0 + for _ in range(10): + if if_enc2d: + size = enc2d_max_size + else: + size = int(np.random.uniform(*scale) * max_size) + if size > 0: + break + if size == 0: + size = max(10, scale[-1] * max_size) + assert size > 0 + index = np.argsort(np.sum(np.square(coord - center), axis=-1))[:size] + origin_index = copy.deepcopy(index) + if if_frame_selected and "frame_pcd_offset" in point: + input_frame_num = len(point["frame_pcd_offset"]) + for _ in range(10): + if input_frame_num - frame_num == 0: + frame_id = 0 + else: + frame_id = np.random.randint(0, input_frame_num) + single_frame_index = point["frame_pcd_offset"][frame_id] + index = list(set(origin_index) & set(single_frame_index)) + if len(index) > 0: + break + if len(index) == 0: + permutation = random.sample(range(input_frame_num), input_frame_num) + for idx in permutation: + single_frame_index = point["frame_pcd_offset"][idx] + index = list(set(origin_index) & set(single_frame_index)) + if len(index) > 0: + break + assert len(index) > 0 + view = dict(index=index) + for key in point.keys(): + if key in self.view_keys: + view[key] = point[key][index] + if key in self.static_view_keys: + view[key] = point[key] + if "index_valid_keys" in point.keys(): + # inherit index_valid_keys from point + view["index_valid_keys"] = point["index_valid_keys"] + return view + + @staticmethod + def match_point_image(major_view, data_dict): + major_correspondence = major_view["correspondence"].transpose(1, 0, 2) + correspondence = data_dict["correspondence"].transpose(1, 0, 2) + is_all_neg1 = np.any(major_correspondence != np.array([-1, -1]), axis=(1, 2)) + indices = np.where(is_all_neg1)[0] + img_dict = { + "images": data_dict["images"][indices], + "img_num": indices.shape[0], + "major_correspondence": major_correspondence[indices].transpose(1, 0, 2), + "correspondence": correspondence[indices].transpose(1, 0, 2), + } + return img_dict + + def __call__(self, data_dict): + coord = data_dict["coord"] + point = self.global_shared_transform(copy.deepcopy(data_dict)) + z_min = coord[:, 2].min() + z_max = coord[:, 2].max() + z_min_ = z_min + (z_max - z_min) * self.center_height_scale[0] + z_max_ = z_min + (z_max - z_min) * self.center_height_scale[1] + if "correspondence" not in data_dict.keys(): + center_mask = np.logical_and(coord[:, 2] >= z_min_, coord[:, 2] <= z_max_) + major_center = coord[np.random.choice(np.where(center_mask)[0])] + major_view = self.get_view(point, major_center, self.global_view_scale) + else: + given_index = data_dict["correspondence"].reshape( + data_dict["correspondence"].shape[0], -1 + ) + given_index = np.all( + given_index != np.ones_like(given_index[0]) * -1, axis=1 + ) + given_coord = data_dict["coord"][given_index] + if given_coord.shape[0] == 0: + center_mask = np.logical_and( + coord[:, 2] >= z_min_, coord[:, 2] <= z_max_ + ) + major_center = coord[np.random.choice(np.where(center_mask)[0])] + else: + major_center = np.mean(given_coord, axis=0) + major_view = self.get_view( + point, major_center, self.global_view_scale, if_enc2d=True + ) + img_dict = self.match_point_image(major_view, data_dict) + major_view["correspondence"] = img_dict["major_correspondence"] + data_dict["correspondence"] = img_dict["correspondence"] + point["correspondence"] = img_dict["correspondence"] + data_dict["img_num"] = img_dict["img_num"] + data_dict["images"] = img_dict["images"] + major_coord = major_view["coord"] + + # get global views: restrict the center of left global view within the major global view + if not self.shared_global_view: + global_views = [ + self.get_view( + point=point, + center=major_coord[np.random.randint(major_coord.shape[0])], + scale=self.global_view_scale, + ) + for _ in range(self.global_view_num - 1) + ] + else: + global_views = [ + {key: value.copy() for key, value in major_view.items()} + for _ in range(self.global_view_num - 1) + ] + + global_views = [major_view] + global_views + + # get local views: restrict the center of local view within the major global view + cover_mask = np.zeros_like(major_view["index"], dtype=bool) + local_views = [] + for i in range(self.local_view_num): + if sum(~cover_mask) == 0: + # reset cover mask if all points are sampled + cover_mask[:] = False + local_view = self.get_view( + point=data_dict, + center=major_coord[np.random.choice(np.where(~cover_mask)[0])], + scale=self.local_view_scale, + if_frame_selected=self.if_frame_selected, + ) + local_views.append(local_view) + cover_mask[np.isin(major_view["index"], local_view["index"])] = True + + # augmentation and concat + view_dict = {} + for global_view in global_views: + global_view.pop("index") + global_view = self.global_transform(global_view) + for key in self.view_keys: + if f"global_{key}" in view_dict.keys(): + view_dict[f"global_{key}"].append(global_view[key]) + else: + view_dict[f"global_{key}"] = [global_view[key]] + view_dict["global_offset"] = np.cumsum( + [data.shape[0] for data in view_dict["global_coord"]] + ) + for local_view in local_views: + local_view.pop("index") + local_view = self.local_transform(local_view) + for key in self.view_keys: + if f"local_{key}" in view_dict.keys(): + view_dict[f"local_{key}"].append(local_view[key]) + else: + view_dict[f"local_{key}"] = [local_view[key]] + view_dict["local_offset"] = np.cumsum( + [data.shape[0] for data in view_dict["local_coord"]] + ) + + for key in view_dict.keys(): + if "offset" not in key: + if key in self.static_view_keys: + view_dict[key] = view_dict[key] + else: + view_dict[key] = np.concatenate(view_dict[key], axis=0) + data_dict.update(view_dict) + return data_dict + + +@TRANSFORMS.register_module() +class InstanceParser(object): + def __init__(self, segment_ignore_index=(-1, 0, 1), instance_ignore_index=-1): + self.segment_ignore_index = segment_ignore_index + self.instance_ignore_index = instance_ignore_index + + def __call__(self, data_dict): + coord = data_dict["coord"] + segment = data_dict["segment"] + instance = data_dict["instance"] + mask = ~np.in1d(segment, self.segment_ignore_index) + # mapping ignored instance to ignore index + instance[~mask] = self.instance_ignore_index + # reorder left instance + unique, inverse = np.unique(instance[mask], return_inverse=True) + instance_num = len(unique) + instance[mask] = inverse + # init instance information + centroid = np.ones((coord.shape[0], 3)) * self.instance_ignore_index + bbox = np.ones((instance_num, 8)) * self.instance_ignore_index + vacancy = [ + index for index in self.segment_ignore_index if index >= 0 + ] # vacate class index + + for instance_id in range(instance_num): + mask_ = instance == instance_id + coord_ = coord[mask_] + bbox_min = coord_.min(0) + bbox_max = coord_.max(0) + bbox_centroid = coord_.mean(0) + bbox_center = (bbox_max + bbox_min) / 2 + bbox_size = bbox_max - bbox_min + bbox_theta = np.zeros(1, dtype=coord_.dtype) + bbox_class = np.array([segment[mask_][0]], dtype=coord_.dtype) + # shift class index to fill vacate class index caused by segment ignore index + bbox_class -= np.greater(bbox_class, vacancy).sum() + + centroid[mask_] = bbox_centroid + bbox[instance_id] = np.concatenate( + [bbox_center, bbox_size, bbox_theta, bbox_class] + ) # 3 + 3 + 1 + 1 = 8 + data_dict["instance"] = instance + data_dict["instance_centroid"] = centroid + data_dict["bbox"] = bbox + return data_dict + + +class Compose(object): + def __init__(self, cfg=None): + self.cfg = cfg if cfg is not None else [] + self.transforms = [] + for t_cfg in self.cfg: + self.transforms.append(TRANSFORMS.build(t_cfg)) + + def __call__(self, data_dict): + for t in self.transforms: + data_dict = t(data_dict) + return data_dict + + +@TRANSFORMS.register_module() +class ImgToTensor(object): + def __init__(self): + self.totensor = transforms.ToTensor() + + def __call__(self, img): + return self.totensor(img) + + +@TRANSFORMS.register_module() +class ImgGaussianBlur(object): + """ + Apply Gaussian Blur to the PIL image. + """ + + def __init__( + self, *, p: float = 0.5, radius_min: float = 0.1, radius_max: float = 2.0 + ): + # NOTE: torchvision is applying 1 - probability to return the original image + self.p = p + self.transform = transforms.GaussianBlur( + kernel_size=9, sigma=(radius_min, radius_max) + ) + super().__init__() + + def __call__(self, img): + if np.random.rand() < self.p: + img = self.transform(img) + return img + + +@TRANSFORMS.register_module() +class ImgChromaticJitter(object): + def __init__(self, p=0.95, std=0.005): + self.p = p + self.std = std + + def __call__(self, img): + if np.random.rand() < self.p: + noise = torch.rand(3) + noise *= self.std + noise = noise[:, None, None].expand_as(img) + img += noise + img = torch.clip(img, 0, 1) + return img + + +@TRANSFORMS.register_module() +class ImgPixelContrast(object): + def __init__(self, threshold, p=0.2): + super().__init__() + self.p = p + self.threshold = threshold + + def __call__(self, img): + if np.random.rand() < self.p: + n, h, w = img.shape[0], img.shape[2], img.shape[3] + num_pixels = int(self.threshold * h * w * n) + indices = torch.randint(0, n * h * w, (num_pixels,)) + img = img.permute(0, 2, 3, 1).reshape(-1, 3) + img[indices, :] = 255.0 - img[indices, :] + img = img.reshape(n, h, w, 3).permute(0, 3, 1, 2) + return img + + +IMAGENET_DEFAULT_MEAN = (0.485, 0.456, 0.406) +IMAGENET_DEFAULT_STD = (0.229, 0.224, 0.225) + + +@TRANSFORMS.register_module() +class Imgnormalize(object): + def __init__(self, mean=IMAGENET_DEFAULT_MEAN, std=IMAGENET_DEFAULT_STD): + super().__init__() + self.normalize = transforms.Normalize(mean=mean, std=std) + + def __call__(self, img): + return self.normalize(img) + + +@TRANSFORMS.register_module() +class ImgRandomHorizontalFlip(object): + def __init__(self, p=0.5): + super().__init__() + self.p = p + self.imgrandomhorizontalflip = transforms.RandomHorizontalFlip(p=p) + + def __call__(self, img): + return self.imgrandomhorizontalflip(img) + + +@TRANSFORMS.register_module() +class ImgRandomResizedCrop(object): + def __init__(self, size, scale, interpolation): + super().__init__() + self.imgrandomresizedcrop = transforms.RandomResizedCrop( + size=size, scale=scale, interpolation=interpolation + ) + + def __call__(self, img): + return self.imgrandomresizedcrop(img) + + +@TRANSFORMS.register_module() +class ImgRandomColorJitter(object): + def __init__(self, brightness=0.4, contrast=0.4, saturation=0.2, hue=0.1, p=0.8): + colorjitter = transforms.ColorJitter( + brightness=brightness, contrast=contrast, saturation=saturation, hue=hue + ) + super().__init__() + self.p = p + self.colorjitter = colorjitter + + def __call__(self, img): + return self.colorjitter(img) + + +@TRANSFORMS.register_module() +class ImgRandomGrayscale(object): + def __init__(self, p=0.1): + super().__init__() + self.p = p + self.imgrandomgrayscale = transforms.RandomGrayscale(p=p) + + def __call__(self, img): + return self.imgrandomgrayscale(img) + + +@TRANSFORMS.register_module() +class ImgRandomSolarize(object): + def __init__(self, threshold, p=0.1): + super().__init__() + self.p = p + self.imgrandomsolarize = transforms.RandomSolarize(threshold=threshold, p=p) + + def __call__(self, img): + return self.imgrandomsolarize(img) + + +@TRANSFORMS.register_module() +class ImgAugmentation(object): + def __init__( + self, + imgtransforms, + crop_h=518, + crop_w=518, + patch_h=37, + patch_w=37, + patch_size=14, + ): + self.transforms = [] + self.transforms_cfg = imgtransforms + for t_cfg in self.transforms_cfg: + self.transforms.append(TRANSFORMS.build(t_cfg)) + self.crop_h = crop_h + self.crop_w = crop_w + self.patch_h = patch_h + self.patch_w = patch_w + self.patch_size = patch_size + self.crop_start = [ + random.randint(0, patch_h * patch_size - crop_h), + random.randint(0, patch_w * patch_size - crop_w), + ] + + def __call__(self, point): + point["images"] = transforms.functional.crop( + point["images"], + top=self.crop_start[0], + left=self.crop_start[1], + height=self.crop_h, + width=self.crop_w, + ) + for id, t in enumerate(self.transforms): + point["images"] = t(point["images"]) + correspondence = point["correspondence"] + correspondence_shape = correspondence.shape + correspondence = correspondence.reshape(-1, 2) + mask = ( + (self.crop_start[0] <= correspondence[:, 0]) + & (correspondence[:, 0] < self.crop_start[0] + self.crop_h) + & (self.crop_start[1] <= correspondence[:, 1]) + & (correspondence[:, 1] < self.crop_start[1] + self.crop_w) + ) + correspondence[~mask] = np.array([-1, -1]) + correspondence[mask] -= np.array(self.crop_start) + point["correspondence"] = correspondence.reshape(correspondence_shape) + return point diff --git a/pointcept/datasets/utils.py b/pointcept/datasets/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..9393ffd3bfca4cc97a8afc3ef6e3c5565d42edbd --- /dev/null +++ b/pointcept/datasets/utils.py @@ -0,0 +1,295 @@ +""" +Utils for Datasets + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import random +from collections.abc import Mapping, Sequence +import numpy as np +import torch +from torch.utils.data.dataloader import default_collate +import torch.nn.functional as F + +from torch_scatter import scatter_min +from pointcept.models.utils import offset2batch + + +def collate_fn(batch): + """ + collate function for point cloud which support dict and list, + 'coord' is necessary to determine 'offset' + """ + if not isinstance(batch, Sequence): + raise TypeError(f"{batch.dtype} is not supported.") + + if isinstance(batch[0], torch.Tensor): + return torch.cat(list(batch)) + elif isinstance(batch[0], str): + # str is also a kind of Sequence, judgement should before Sequence + return list(batch) + elif isinstance(batch[0], list): + batch = [torch.tensor(data) for data in batch] + return torch.cat(batch) + elif isinstance(batch[0], Sequence): + for data in batch: + data.append(torch.tensor([data[0].shape[0]])) + batch = [collate_fn(samples) for samples in zip(*batch)] + batch[-1] = torch.cumsum(batch[-1], dim=0).int() + return batch + elif isinstance(batch[0], Mapping): + if "img_num" in batch[0].keys(): + max_img_num = max([d["img_num"] for d in batch]) + batch = { + key: ( + ( + collate_fn([d[key] for d in batch]) + if "offset" not in key + # offset -> bincount -> concat bincount-> concat offset + else torch.cumsum( + collate_fn( + [d[key].diff(prepend=torch.tensor([0])) for d in batch] + ), + dim=0, + ) + ) + if "correspondence" not in key + else collate_fn( + [ + F.pad( + d[key].permute(0, 2, 1), + (0, max_img_num - d[key].shape[1]), + value=-1, + ).permute(0, 2, 1) + for d in batch + ] + ) + ) + for key in batch[0] + } + return batch + else: + return default_collate(batch) + + +def pairwise_concatenate(tensors_list, dim=0): + new_list = [] + n = len(tensors_list) + for i in range(0, n - 1, 2): + tensor1 = tensors_list[i] + tensor2 = tensors_list[i + 1] + concatenated_tensor = torch.cat([tensor1, tensor2], dim=dim) + new_list.append(concatenated_tensor) + if n % 2 != 0: + new_list.append(tensors_list[-1]) + return new_list + + +def regroup_batch(batch, N, original_offsets, data_keys): + num_segments = len(original_offsets) + grouped_segments = {key: [[] for _ in range(N)] for key in data_keys} + start_idx = 0 + for i in range(num_segments): + end_idx = original_offsets[i] + group_idx = i % N + for key in data_keys: + segment = batch[key][start_idx:end_idx] + grouped_segments[key][group_idx].append(segment) + start_idx = end_idx + + segment_lengths = original_offsets[1:] - original_offsets[:-1] + segment_lengths = torch.cat([original_offsets[:1], segment_lengths]) + new_lengths_order = [] + split_lengths_order = [] + for i in range(N): + segment_lengths_i = segment_lengths[i::N] + n = segment_lengths_i.shape[0] + num_pairs = n // 2 + paired_part = segment_lengths_i[: num_pairs * 2] + reshaped_part = paired_part.view(num_pairs, 2, *segment_lengths_i.shape[1:]) + sums = torch.sum(reshaped_part, dim=1) + first_parts = reshaped_part[:, 0] + if n % 2 != 0: + last_element = segment_lengths_i[-1] + last_element = last_element.unsqueeze(0) + final_result = torch.cat([sums, last_element], dim=0) + else: + final_result = sums + new_lengths_order.append(final_result) + split_lengths_order.append(first_parts) + new_lengths_order = torch.stack(new_lengths_order, dim=1) + new_lengths_order = new_lengths_order.flatten() + new_offsets = torch.cumsum(new_lengths_order, dim=0) + split_lengths_order = torch.stack(split_lengths_order, dim=1).flatten() + + new_batch = {} + new_batch_imgs = [] + new_batch_img_num = 0 + img_num_offset = torch.cat( + [torch.tensor([0]), torch.cumsum(batch["img_num"], dim=0)] + ) + for key in data_keys: + final_segments_in_order = [] + for i in range(N): + grouped_segments[key][i] = pairwise_concatenate( + grouped_segments[key][i], dim=0 + ) + for i in range(len(grouped_segments[key][0])): + for j in range(N): + final_segments_in_order.append(grouped_segments[key][j][i]) + new_batch[key] = torch.vstack(final_segments_in_order) + if "correspondence" in key: + current_start = 0 + N0, v, n_dim = new_batch[key].shape + v2 = v * 2 + batch_correspondence_mix = -torch.ones( + (N0, v2, n_dim), + dtype=new_batch[key].dtype, + device=new_batch[key].device, + ) + for k, end in enumerate(new_offsets): + len_part1 = split_lengths_order[k] + split_point = current_start + len_part1 + if split_point > current_start: + mask1 = torch.any( + new_batch[key][current_start:split_point] + != torch.tensor([-1, -1]), + dim=2, + ) + valid_index1 = torch.where(mask1) + if len(valid_index1[1]) == 0: + count1 = 0 + else: + count1 = max(valid_index1[1]) + batch_correspondence_mix[current_start:split_point, 0:count1] = ( + new_batch[key][current_start:split_point, 0:count1] + ) + if k % N == 0 and N == 2: + new_batch_imgs.append( + batch["images"][ + img_num_offset[k // N * 2] : img_num_offset[k // N * 2] + + count1 + ] + ) + if end > split_point: + mask2 = torch.any( + new_batch[key][split_point:end] != torch.tensor([-1, -1]), dim=2 + ) + valid_index2 = torch.where(mask2) + if len(valid_index2[1]) == 0: + count2 = 0 + else: + count2 = max(valid_index2[1]) + batch_correspondence_mix[ + split_point:end, count1 : count1 + count2 + ] = new_batch[key][split_point:end, 0:count2] + if k % N == 0 and N == 2: + new_batch_imgs.append( + batch["images"][ + img_num_offset[k // N * 2 + 1] : img_num_offset[ + k // N * 2 + 1 + ] + + count2 + ] + ) + current_start = end + + if N == 2: + new_batch_img_num = torch.tensor([i.shape[0] for i in new_batch_imgs]) + new_batch_imgs = torch.vstack(new_batch_imgs) + else: + new_batch_img_num = None + new_batch_imgs = None + new_batch[key] = batch_correspondence_mix + return new_batch, new_offsets, new_batch_imgs, new_batch_img_num + + +def point_collate_fn(batch, mix_prob=0): + assert isinstance( + batch[0], Mapping + ) # currently, only support input_dict, rather than input_list + batch = collate_fn(batch) + if random.random() < mix_prob: + valid_keys = [ + "coord", + "grid_coord", + "origin_coord", + "color", + "normal", + "feat", + "correspondence", + ] + if "instance" in batch.keys(): + offset = batch["offset"] + start = 0 + num_instance = 0 + for i in range(len(offset)): + if i % 2 == 0: + num_instance = max(batch["instance"][start : offset[i]]) + if i % 2 != 0: + mask = batch["instance"][start : offset[i]] != -1 + batch["instance"][start : offset[i]] += num_instance * mask + start = offset[i] + offset_assets = [asset for asset in batch.keys() if "offset" in asset] + for offset_asset in offset_assets: + batch[offset_asset] = torch.cat( + [batch[offset_asset][1:-1:2], batch[offset_asset][-1].unsqueeze(0)], + dim=0, + ) + + # Recompute grid_coord after mixing, because each scene's grid_coord was + # independently shifted before mixing and is no longer consistent with + # the merged coord. Only done when grid_size is available (e.g. LitePT + # configs); other configs are unaffected. + if "grid_coord" in batch and "grid_size" in batch: + batch_idx = offset2batch(batch["offset"]) + scaled_coord = batch["coord"] / batch["grid_size"][0] + grid_coord = torch.floor(scaled_coord).to(torch.int64) + min_coord, _ = scatter_min(grid_coord, batch_idx, dim=0) + batch["grid_coord"] = grid_coord - min_coord[batch_idx] + offset_assets = [asset for asset in batch.keys() if "_offset" in asset] + for offset_asset in offset_assets: + offset_prefix = offset_asset.split("_")[0] + valid_keys_with_prefix = [ + offset_prefix + "_" + valid_key for valid_key in valid_keys + ] + valid_keys_with_prefix = [ + valid_key_with_prefix + for valid_key_with_prefix in valid_keys_with_prefix + if valid_key_with_prefix in batch.keys() + ] + if "global" in offset_asset: + N = 2 + elif "local" in offset_asset: + N = 4 + updated_batch, new_offset, imgs, img_num = regroup_batch( + batch, N, batch[offset_asset], valid_keys_with_prefix + ) + batch[offset_asset] = new_offset + batch.update(updated_batch) + if "global" in offset_asset: + batch["images"] = imgs + batch["img_num"] = img_num + + if "img_num" in batch.keys(): + n = batch["img_num"].shape[0] + num_pairs = n // 2 + len_pairs = num_pairs * 2 + pairs_tensor = batch["img_num"][:len_pairs] + + if num_pairs == 0: + pass + else: + summed_pairs = pairs_tensor.view(-1, 2).sum(dim=1) + if n % 2 != 0: + last_element = batch["img_num"][-1:] + result = torch.cat((summed_pairs, last_element)) + else: + result = summed_pairs + batch["img_num"] = result + return batch + + +def gaussian_kernel(dist2: np.array, a: float = 1, c: float = 5): + return a * np.exp(-dist2 / (2 * c**2)) diff --git a/pointcept/datasets/waymo.py b/pointcept/datasets/waymo.py new file mode 100644 index 0000000000000000000000000000000000000000..2fd7505c8fa84acb781cf47319427bfdc44501d0 --- /dev/null +++ b/pointcept/datasets/waymo.py @@ -0,0 +1,368 @@ +""" +Waymo dataset + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import numpy as np +import glob +import random + +from .builder import DATASETS +from .defaults import DefaultDataset, DefaultImagePointDataset + + +def _append_timing_embedding(strength, timestamp): + if strength.ndim == 1: + strength = strength.reshape(-1, 1) + timing = np.full((strength.shape[0], 1), timestamp, dtype=strength.dtype) + return np.hstack((strength, timing)) + + +@DATASETS.register_module() +class WaymoDataset(DefaultDataset): + def __init__( + self, + timestamp=(0,), + reference_label=True, + timing_embedding=False, + **kwargs, + ): + super().__init__(**kwargs) + assert timestamp[0] == 0 + self.timestamp = timestamp + self.reference_label = reference_label + self.timing_embedding = timing_embedding + self.data_list = sorted(self.data_list) + _, self.sequence_offset, self.sequence_index = np.unique( + [os.path.dirname(data) for data in self.data_list], + return_index=True, + return_inverse=True, + ) + self.sequence_offset = np.append(self.sequence_offset, len(self.data_list)) + + def get_data_list(self): + if isinstance(self.split, str): + self.split = [self.split] + data_list = [] + for split in self.split: + data_list += glob.glob(os.path.join(self.data_root, split, "*", "*")) + return data_list + + @staticmethod + def align_pose(coord, pose, target_pose): + coord = np.hstack((coord, np.ones_like(coord[:, :1]))) + pose_align = np.matmul(np.linalg.inv(target_pose), pose) + coord = (pose_align @ coord.T).T[:, :3] + return coord + + @staticmethod + def align_normal(normal, pose, target_pose): + pose_align = np.matmul(np.linalg.inv(target_pose), pose) + return (pose_align[:3, :3] @ normal.T).T + + def get_single_frame(self, idx): + return super().get_data(idx) + + def get_data(self, idx): + idx = idx % len(self.data_list) + if self.timestamp == (0,): + data_dict = self.get_single_frame(idx) + if self.timing_embedding: + data_dict["strength"] = _append_timing_embedding( + data_dict["strength"], 0 + ) + return data_dict + + sequence_index = self.sequence_index[idx] + lower, upper = self.sequence_offset[[sequence_index, sequence_index + 1]] + major_frame = self.get_single_frame(idx) + name = major_frame.pop("name") + target_pose = major_frame.pop("pose") + if self.timing_embedding: + major_frame["strength"] = _append_timing_embedding( + major_frame["strength"], 0 + ) + for key in major_frame.keys(): + major_frame[key] = [major_frame[key]] + + for timestamp in self.timestamp[1:]: + refer_idx = timestamp + idx + if refer_idx < lower or upper <= refer_idx: + continue + refer_frame = self.get_single_frame(refer_idx) + refer_frame.pop("name") + pose = refer_frame.pop("pose") + refer_frame["coord"] = self.align_pose( + refer_frame["coord"], pose, target_pose + ) + if "normal" in refer_frame: + refer_frame["normal"] = self.align_normal( + refer_frame["normal"], pose, target_pose + ) + if not self.reference_label: + refer_frame["segment"] = ( + np.ones_like(refer_frame["segment"]) * self.ignore_index + ) + + if self.timing_embedding: + refer_frame["strength"] = _append_timing_embedding( + refer_frame["strength"], timestamp + ) + + for key in major_frame.keys(): + major_frame[key].append(refer_frame[key]) + for key in major_frame.keys(): + if isinstance(major_frame[key][0], str): + continue + major_frame[key] = np.concatenate(major_frame[key], axis=0) + major_frame["name"] = name + return major_frame + + def get_data_name(self, idx): + file_path = self.data_list[idx % len(self.data_list)] + sequence_path, frame_name = os.path.split(file_path) + sequence_name = os.path.basename(sequence_path) + data_name = f"{sequence_name}_{frame_name}" + return data_name + + +@DATASETS.register_module() +class WaymoColorNormalDataset(WaymoDataset): + @staticmethod + def estimate_normals(points, center=np.array([0, 0, 0])): + normals = points - center[None, :] + norms = np.linalg.norm(normals, axis=1, keepdims=True) + normals = normals / norms + return normals + + def get_single_frame(self, idx): + data_path = self.data_list[idx % len(self.data_list)] + name = self.get_data_name(idx) + split = self.get_split_name(idx) + data_dict = {} + assets = os.listdir(data_path) + for asset in assets: + if not asset.endswith(".npy"): + continue + if asset[:-4] not in self.VALID_ASSETS: + continue + data_dict[asset[:-4]] = np.load(os.path.join(data_path, asset)) + data_dict["name"] = name + data_dict["split"] = split + data_dict["color"] = np.zeros_like(data_dict["coord"]) # placeholder for color + data_dict["normal"] = np.zeros_like(data_dict["coord"]) + + if "coord" in data_dict.keys(): + data_dict["coord"] = data_dict["coord"].astype(np.float32) + + if "color" in data_dict.keys(): + data_dict["color"] = data_dict["color"].astype(np.float32) + + if "normal" in data_dict.keys(): + data_dict["normal"] = data_dict["normal"].astype(np.float32) + + if "segment" in data_dict.keys(): + data_dict["segment"] = data_dict["segment"].reshape([-1]).astype(np.int32) + else: + data_dict["segment"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + + if "instance" in data_dict.keys(): + data_dict["instance"] = data_dict["instance"].reshape([-1]).astype(np.int32) + else: + data_dict["instance"] = ( + np.ones(data_dict["coord"].shape[0], dtype=np.int32) * -1 + ) + return data_dict + + def get_data(self, idx): + idx = idx % len(self.data_list) + if self.timestamp == (0,): + data_dict = self.get_single_frame(idx) + if self.timing_embedding: + data_dict["strength"] = _append_timing_embedding( + data_dict["strength"], 0 + ) + return data_dict + + sequence_index = self.sequence_index[idx] + lower, upper = self.sequence_offset[[sequence_index, sequence_index + 1]] + major_frame = self.get_single_frame(idx) + name = major_frame.pop("name") + target_pose = major_frame.pop("pose") + if self.timing_embedding: + major_frame["strength"] = _append_timing_embedding( + major_frame["strength"], 0 + ) + for key in major_frame.keys(): + major_frame[key] = [major_frame[key]] + + for timestamp in self.timestamp[1:]: + refer_idx = timestamp + idx + if refer_idx < lower or upper <= refer_idx: + continue + refer_frame = self.get_single_frame(refer_idx) + refer_frame.pop("name") + pose = refer_frame.pop("pose") + refer_frame["coord"] = self.align_pose( + refer_frame["coord"], pose, target_pose + ) + if "normal" in refer_frame: + refer_frame["normal"] = self.align_normal( + refer_frame["normal"], pose, target_pose + ) + if not self.reference_label: + refer_frame["segment"] = ( + np.ones_like(refer_frame["segment"]) * self.ignore_index + ) + + if self.timing_embedding: + refer_frame["strength"] = _append_timing_embedding( + refer_frame["strength"], timestamp + ) + + for key in major_frame.keys(): + major_frame[key].append(refer_frame[key]) + for key in major_frame.keys(): + if isinstance(major_frame[key][0], str): + major_frame[key] = major_frame[key][0] + continue + major_frame[key] = np.concatenate(major_frame[key], axis=0) + major_frame["name"] = name + return major_frame + + +@DATASETS.register_module() +class WaymoImagePointDataset(DefaultImagePointDataset): + def __init__( + self, + if_img=False, + if_sweep=False, + reference_label=True, + timing_embedding=False, + sweeps=3, + sweep_gap=5, + **kwargs, + ): + super().__init__(**kwargs) + self.if_img = if_img + self.if_sweep = if_sweep + if if_sweep: + self.timestamp = tuple(range(0, sweeps * sweep_gap, sweep_gap)) + else: + self.timestamp = (0,) + self.reference_label = reference_label + self.timing_embedding = timing_embedding + self.sweeps = sweeps + self.data_name = sorted(self.data_name) + _, self.sequence_offset, self.sequence_index = np.unique( + [data.split("_with_camera_labels_")[0] for data in self.data_name], + return_index=True, + return_inverse=True, + ) + self.sequence_offset = np.append(self.sequence_offset, len(self.data_name)) + + @staticmethod + def align_pose(coord, pose, target_pose): + coord = np.hstack((coord, np.ones_like(coord[:, :1]))) + pose_align = np.matmul(np.linalg.inv(target_pose), pose) + coord = (pose_align @ coord.T).T[:, :3] + return coord + + @staticmethod + def align_normal(normal, pose, target_pose): + pose_align = np.matmul(np.linalg.inv(target_pose), pose) + return (pose_align[:3, :3] @ normal.T).T + + def get_single_frame(self, idx): + return super().get_data(idx) + + def get_data(self, idx): + idx = idx % len(self.data_list) + if self.timestamp == (0,): + data_dict = self.get_single_frame(idx) + if self.timing_embedding: + data_dict["strength"] = _append_timing_embedding( + data_dict["strength"], 0 + ) + return data_dict + + sequence_index = self.sequence_index[idx] + lower, upper = self.sequence_offset[[sequence_index, sequence_index + 1]] + valid_timestamps = [0] + [ + timestamp + for timestamp in self.timestamp[1:] + if lower <= timestamp + idx < upper + ] + imgs_idx = random.sample(valid_timestamps, 1)[0] + major_frame = self.get_single_frame(idx) + name = major_frame.pop("name") + target_pose = major_frame.pop("pose") + if self.timing_embedding: + major_frame["strength"] = _append_timing_embedding( + major_frame["strength"], 0 + ) + if not self.if_img: + pass + elif imgs_idx == 0: + major_frame["correspondence"] = [major_frame["correspondence"]] + else: + major_frame.pop("img_num") + major_frame.pop("images") + major_frame["correspondence"] = [ + -np.ones_like(major_frame["correspondence"]) + ] + for key in major_frame.keys(): + if key in self.PC_VALID_ASSETS: + major_frame[key] = [major_frame[key]] + + for timestamp in valid_timestamps[1:]: + refer_idx = timestamp + idx + refer_frame = self.get_single_frame(refer_idx) + refer_frame.pop("name") + pose = refer_frame.pop("pose") + refer_frame["coord"] = self.align_pose( + refer_frame["coord"], pose, target_pose + ) + if "normal" in refer_frame: + refer_frame["normal"] = self.align_normal( + refer_frame["normal"], pose, target_pose + ) + if not self.reference_label: + refer_frame["segment"] = ( + np.ones_like(refer_frame["segment"]) * self.ignore_index + ) + + if self.timing_embedding: + refer_frame["strength"] = _append_timing_embedding( + refer_frame["strength"], timestamp + ) + + for key in major_frame.keys(): + if key in self.PC_VALID_ASSETS: + major_frame[key].append(refer_frame[key]) + + if self.if_img: + if imgs_idx == timestamp: + major_frame["img_num"] = refer_frame["img_num"] + major_frame["images"] = refer_frame["images"] + major_frame["correspondence"].append(refer_frame["correspondence"]) + else: + major_frame["correspondence"].append( + -np.ones_like(refer_frame["correspondence"]) + ) + frame_pcd_offset = [ + frame_coord.shape[0] for frame_coord in major_frame["coord"] + ] + frame_pcd_offset = np.cumsum(frame_pcd_offset) + + for key in major_frame.keys(): + if key in self.PC_VALID_ASSETS + ["correspondence"]: + major_frame[key] = np.concatenate(major_frame[key], axis=0) + major_frame["name"] = name + major_frame["frame_pcd_offset"] = frame_pcd_offset + return major_frame diff --git a/pointcept/engines/__init__.py b/pointcept/engines/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/pointcept/engines/defaults.py b/pointcept/engines/defaults.py new file mode 100644 index 0000000000000000000000000000000000000000..680cbd3ffdf4712a192c51fdfc8d5be25bd28ed4 --- /dev/null +++ b/pointcept/engines/defaults.py @@ -0,0 +1,151 @@ +""" +Default training/testing logic + +modified from detectron2(https://github.com/facebookresearch/detectron2) + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import sys +import argparse +import multiprocessing as mp +from torch.nn.parallel import DistributedDataParallel + + +import pointcept.utils.comm as comm +from pointcept.utils.env import get_random_seed, set_seed +from pointcept.utils.config import Config, DictAction + + +def create_ddp_model(model, *, fp16_compression=False, **kwargs): + """ + Create a DistributedDataParallel model if there are >1 processes. + Args: + model: a torch.nn.Module + fp16_compression: add fp16 compression hooks to the ddp object. + See more at https://pytorch.org/docs/stable/ddp_comm_hooks.html#torch.distributed.algorithms.ddp_comm_hooks.default_hooks.fp16_compress_hook + kwargs: other arguments of :module:`torch.nn.parallel.DistributedDataParallel`. + """ + if comm.get_world_size() == 1: + return model + # kwargs['find_unused_parameters'] = True + if "device_ids" not in kwargs: + kwargs["device_ids"] = [comm.get_local_rank()] + if "output_device" not in kwargs: + kwargs["output_device"] = [comm.get_local_rank()] + ddp = DistributedDataParallel(model, **kwargs) + if fp16_compression: + from torch.distributed.algorithms.ddp_comm_hooks import default as comm_hooks + + ddp.register_comm_hook(state=None, hook=comm_hooks.fp16_compress_hook) + return ddp + + +def worker_init_fn(worker_id, num_workers, rank, seed): + """Worker init func for dataloader. + + The seed of each worker equals to num_worker * rank + worker_id + user_seed + + Args: + worker_id (int): Worker id. + num_workers (int): Number of workers. + rank (int): The rank of current process. + seed (int): The random seed to use. + """ + + worker_seed = None if seed is None else num_workers * rank + worker_id + seed + set_seed(worker_seed) + + +def default_argument_parser(epilog=None): + parser = argparse.ArgumentParser( + epilog=epilog or f""" + Examples: + Run on single machine: + $ {sys.argv[0]} --num-gpus 8 --config-file cfg.yaml + Change some config options: + $ {sys.argv[0]} --config-file cfg.yaml MODEL.WEIGHTS /path/to/weight.pth SOLVER.BASE_LR 0.001 + Run on multiple machines: + (machine0)$ {sys.argv[0]} --machine-rank 0 --num-machines 2 --dist-url [--other-flags] + (machine1)$ {sys.argv[0]} --machine-rank 1 --num-machines 2 --dist-url [--other-flags] + """, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "--config-file", default="", metavar="FILE", help="path to config file" + ) + parser.add_argument( + "--num-gpus", type=int, default=1, help="number of gpus *per machine*" + ) + parser.add_argument( + "--num-machines", type=int, default=1, help="total number of machines" + ) + parser.add_argument( + "--machine-rank", + type=int, + default=0, + help="the rank of this machine (unique per machine)", + ) + # PyTorch still may leave orphan processes in multi-gpu training. + # Therefore we use a deterministic way to obtain port, + # so that users are aware of orphan processes by seeing the port occupied. + # port = 2 ** 15 + 2 ** 14 + hash(os.getuid() if sys.platform != "win32" else 1) % 2 ** 14 + parser.add_argument( + "--dist-url", + # default="tcp://127.0.0.1:{}".format(port), + default="auto", + help="initialization URL for pytorch distributed backend. See " + "https://pytorch.org/docs/stable/distributed.html for details.", + ) + parser.add_argument( + "--options", nargs="+", action=DictAction, help="custom options" + ) + return parser + + +def default_config_parser(file_path, options): + # config name protocol: dataset_name/model_name-exp_name + if os.path.isfile(file_path): + cfg = Config.fromfile(file_path) + else: + sep = file_path.find("-") + cfg = Config.fromfile(os.path.join(file_path[:sep], file_path[sep + 1 :])) + + if options is not None: + cfg.merge_from_dict(options) + + if cfg.seed is None: + cfg.seed = get_random_seed() + + cfg.data.train.loop = cfg.epoch // cfg.eval_epoch + + os.makedirs(os.path.join(cfg.save_path, "model"), exist_ok=True) + if not cfg.resume: + cfg.dump(os.path.join(cfg.save_path, "config.py")) + return cfg + + +def default_setup(cfg): + # scalar by world size + world_size = comm.get_world_size() + cfg.num_worker = cfg.num_worker if cfg.num_worker is not None else mp.cpu_count() + cfg.num_worker_per_gpu = cfg.num_worker // world_size + assert cfg.batch_size % world_size == 0 + assert cfg.batch_size_val is None or cfg.batch_size_val % world_size == 0 + assert cfg.batch_size_test is None or cfg.batch_size_test % world_size == 0 + cfg.batch_size_per_gpu = cfg.batch_size // world_size + cfg.batch_size_val_per_gpu = ( + cfg.batch_size_val // world_size if cfg.batch_size_val is not None else 1 + ) + cfg.batch_size_test_per_gpu = ( + cfg.batch_size_test // world_size if cfg.batch_size_test is not None else 1 + ) + # update data loop + assert cfg.epoch % cfg.eval_epoch == 0 + # settle random seed + rank = comm.get_rank() + seed = None if cfg.seed is None else cfg.seed + rank * cfg.num_worker_per_gpu + set_seed(seed) + return cfg diff --git a/pointcept/engines/hooks/__init__.py b/pointcept/engines/hooks/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..781c888115b5ee091a7fb9adf0be2bf7f5b2be26 --- /dev/null +++ b/pointcept/engines/hooks/__init__.py @@ -0,0 +1,7 @@ +# Trimmed for inference-only use: this delivery never builds a Trainer, so +# the training-time hooks in .misc/.evaluator (which pull in wandb/pointops, +# unneeded here) aren't imported. Only HookBase is needed as a base class by +# pointcept.models.modules.PointModel. +from .default import * + +from .builder import build_hooks diff --git a/pointcept/engines/hooks/builder.py b/pointcept/engines/hooks/builder.py new file mode 100644 index 0000000000000000000000000000000000000000..8a5827a1b9539355a3e6eb21935fd486bbec507d --- /dev/null +++ b/pointcept/engines/hooks/builder.py @@ -0,0 +1,17 @@ +""" +Hook Builder + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from pointcept.utils.registry import Registry + +HOOKS = Registry("hooks") + + +def build_hooks(cfg): + hooks = [] + for hook_cfg in cfg: + hooks.append(HOOKS.build(hook_cfg)) + return hooks diff --git a/pointcept/engines/hooks/default.py b/pointcept/engines/hooks/default.py new file mode 100644 index 0000000000000000000000000000000000000000..47f2aa199f8d54b540153b2362899c0e27fdd8fa --- /dev/null +++ b/pointcept/engines/hooks/default.py @@ -0,0 +1,66 @@ +""" +Default Hook + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import pointcept.utils.comm as comm +import weakref +from .builder import HOOKS + + +class HookBase: + """ + Base class for hooks that can be registered with :class:`TrainerBase`. + """ + + trainer = None # A weak reference to the trainer object. + + def before_train(self): + pass + + def before_epoch(self): + pass + + def before_step(self): + pass + + def after_step(self): + pass + + def after_epoch(self): + pass + + def after_train(self): + pass + + +@HOOKS.register_module() +class ModelHook(HookBase): + def before_train(self): + if comm.get_world_size() > 1 and isinstance( + self.trainer.model.module, HookBase + ): + self.model = weakref.proxy(self.trainer.model.module) + elif isinstance(self.trainer.model, HookBase): + self.model = weakref.proxy(self.trainer.model) + else: + self.model = HookBase() + self.model.trainer = self.trainer + self.model.before_train() + + def before_epoch(self): + self.model.before_epoch() + + def before_step(self): + self.model.before_step() + + def after_step(self): + self.model.after_step() + + def after_epoch(self): + self.model.after_epoch() + + def after_train(self): + self.model.after_train() diff --git a/pointcept/engines/hooks/evaluator.py b/pointcept/engines/hooks/evaluator.py new file mode 100644 index 0000000000000000000000000000000000000000..db2f2c1b3e865512e06f1ffd91e7c1ae78bd0b6d --- /dev/null +++ b/pointcept/engines/hooks/evaluator.py @@ -0,0 +1,997 @@ +""" +Evaluate Hook + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import numpy as np +import torch +import torch.distributed as dist +import pointops +import wandb +from uuid import uuid4 + +import pointcept.utils.comm as comm +from pointcept.utils.misc import intersection_and_union_gpu + +from .default import HookBase +from .builder import HOOKS + + +@HOOKS.register_module() +class ClsEvaluator(HookBase): + def after_epoch(self): + if self.trainer.cfg.evaluate: + self.eval() + + def before_train(self): + if self.trainer.writer is not None and self.trainer.cfg.enable_wandb: + wandb.define_metric("val/*", step_metric="Epoch") + + def eval(self): + self.trainer.logger.info( + ">>>>>>>>>>>>>>>> Start Evaluation (Classification) >>>>>>>>>>>>>>>>" + ) + self.trainer.model.eval() + for i, input_dict in enumerate(self.trainer.val_loader): + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + with torch.no_grad(): + output_dict = self.trainer.model(input_dict) + output = output_dict["cls_logits"] + loss = output_dict["loss"] + pred = output.max(1)[1] + label = input_dict["category"] + intersection, union, target = intersection_and_union_gpu( + pred, + label, + self.trainer.cfg.data.num_classes, + self.trainer.cfg.data.ignore_index, + ) + if comm.get_world_size() > 1: + dist.all_reduce(intersection), dist.all_reduce(union), dist.all_reduce( + target + ) + intersection, union, target = ( + intersection.cpu().numpy(), + union.cpu().numpy(), + target.cpu().numpy(), + ) + self.trainer.storage.put_scalar("val_intersection", intersection) + self.trainer.storage.put_scalar("val_union", union) + self.trainer.storage.put_scalar("val_target", target) + self.trainer.storage.put_scalar("val_loss", loss.item()) + self.trainer.logger.info( + "Test: [{iter}/{max_iter}] " + "Loss {loss:.4f} ".format( + iter=i + 1, max_iter=len(self.trainer.val_loader), loss=loss.item() + ) + ) + + # Strict Metric Calculation & Ordering + loss_avg = self.trainer.storage.history("val_loss").avg + intersection = self.trainer.storage.history("val_intersection").total # TP + union = self.trainer.storage.history("val_union").total # TP + FP + FN + target = self.trainer.storage.history("val_target").total # TP + FN (GT) + + # Derived count: TP + FP = (TP + FP + FN) + TP - (TP + FN) + area_output = union + intersection - target + epsilon = 1e-10 + + # Per-Class Calculation + iou_class = intersection / (union + epsilon) + acc_class = intersection / (target + epsilon) # Recall + precision_class = intersection / (area_output + epsilon) + f1_class = ( + 2 * (precision_class * acc_class) / (precision_class + acc_class + epsilon) + ) + + # Global Calculation + m_iou = np.mean(iou_class) + m_acc = np.mean(acc_class) # Macro Recall + m_precision = np.mean(precision_class) # Macro Precision + macro_f1 = np.mean(f1_class) # Macro F1 + + total_tp = sum(intersection) + total_target = sum(target) + all_acc = total_tp / (total_target + epsilon) # Overall Accuracy + + # Log Output - Global (Order: mIoU, macroF1, mAcc, mPrecision, allAcc) + self.trainer.logger.info( + "Val result: mIoU/macroF1/mAcc/mPrecision/allAcc " + "{:.4f}/{:.4f}/{:.4f}/{:.4f}/{:.4f}.".format( + m_iou, macro_f1, m_acc, m_precision, all_acc + ) + ) + + # Log Output - Per Class (Order: iou, acc, precision, f1) + for i in range(self.trainer.cfg.data.num_classes): + self.trainer.logger.info( + "Class_{idx}-{name} Result: iou/acc/precision/f1 " + "{iou:.4f}/{acc:.4f}/{prec:.4f}/{f1:.4f}".format( + idx=i, + name=self.trainer.cfg.data.names[i], + iou=iou_class[i], + acc=acc_class[i], + prec=precision_class[i], + f1=f1_class[i], + ) + ) + + current_epoch = self.trainer.epoch + 1 + if self.trainer.writer is not None: + self.trainer.writer.add_scalar("val/loss", loss_avg, current_epoch) + self.trainer.writer.add_scalar("val/mIoU", m_iou, current_epoch) + self.trainer.writer.add_scalar("val/macroF1", macro_f1, current_epoch) + self.trainer.writer.add_scalar("val/mAcc", m_acc, current_epoch) + self.trainer.writer.add_scalar("val/mPrecision", m_precision, current_epoch) + self.trainer.writer.add_scalar("val/allAcc", all_acc, current_epoch) + + if self.trainer.cfg.enable_wandb: + wandb.log( + { + "Epoch": current_epoch, + "val/loss": loss_avg, + "val/mIoU": m_iou, + "val/macroF1": macro_f1, + "val/mAcc": m_acc, + "val/mPrecision": m_precision, + "val/allAcc": all_acc, + }, + step=wandb.run.step, + ) + self.trainer.logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + self.trainer.comm_info["current_metric_value"] = all_acc + self.trainer.comm_info["current_metric_name"] = "allAcc" + + def after_train(self): + self.trainer.logger.info( + "Best {}: {:.4f}".format("allAcc", self.trainer.best_metric_value) + ) + + +@HOOKS.register_module() +class SemSegEvaluator(HookBase): + def __init__(self, write_cls_metrics=False): + self.write_cls_metrics = write_cls_metrics + + def before_train(self): + if self.trainer.writer is not None and self.trainer.cfg.enable_wandb: + wandb.define_metric("val/*", step_metric="Epoch") + + def after_epoch(self): + if self.trainer.cfg.evaluate: + self.eval() + + def eval(self): + self.trainer.logger.info( + ">>>>>>>>>>>>>>>> Start Evaluation (Segmentation) >>>>>>>>>>>>>>>>" + ) + self.trainer.model.eval() + for i, input_dict in enumerate(self.trainer.val_loader): + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + with torch.no_grad(): + output_dict = self.trainer.model(input_dict) + output = output_dict["seg_logits"] + loss = output_dict["loss"] + pred = output.max(1)[1] + segment = input_dict["segment"] + + # Handle inverse mapping if needed + if "inverse" in input_dict.keys(): + assert "origin_segment" in input_dict.keys() + pred = pred[input_dict["inverse"]] + segment = input_dict["origin_segment"] + elif "origin_coord" in input_dict.keys(): + assert "origin_segment" in input_dict.keys() + idx, _ = pointops.knn_query( + 1, + input_dict["coord"].float(), + input_dict["offset"].int(), + input_dict["origin_coord"].float(), + input_dict["origin_offset"].int(), + ) + pred = pred[idx.flatten().long()] + segment = input_dict["origin_segment"] + + intersection, union, target = intersection_and_union_gpu( + pred, + segment, + self.trainer.cfg.data.num_classes, + self.trainer.cfg.data.ignore_index, + ) + if comm.get_world_size() > 1: + dist.all_reduce(intersection), dist.all_reduce(union), dist.all_reduce( + target + ) + intersection, union, target = ( + intersection.cpu().numpy(), + union.cpu().numpy(), + target.cpu().numpy(), + ) + + # Here there is no need to sync since sync happened in dist.all_reduce + self.trainer.storage.put_scalar("val_intersection", intersection) + self.trainer.storage.put_scalar("val_union", union) + self.trainer.storage.put_scalar("val_target", target) + self.trainer.storage.put_scalar("val_loss", loss.item()) + + info = "Test: [{iter}/{max_iter}] ".format( + iter=i + 1, max_iter=len(self.trainer.val_loader) + ) + if "origin_coord" in input_dict.keys(): + info = "Interp. " + info + self.trainer.logger.info(info + "Loss {loss:.4f} ".format(loss=loss.item())) + + loss_avg = self.trainer.storage.history("val_loss").avg + intersection = self.trainer.storage.history("val_intersection").total + union = self.trainer.storage.history("val_union").total + target = self.trainer.storage.history("val_target").total + + epsilon = 1e-10 + area_output = union + intersection - target + + # Per-Class Calculation + iou_class = intersection / (union + epsilon) + acc_class = intersection / (target + epsilon) # Recall + precision_class = intersection / (area_output + epsilon) # Precision + f1_class = ( + 2 * (precision_class * acc_class) / (precision_class + acc_class + epsilon) + ) + + # Global Calculation + m_iou = np.mean(iou_class) + m_acc = np.mean(acc_class) + m_precision = np.mean(precision_class) + macro_f1 = np.mean(f1_class) + + total_tp = sum(intersection) + total_target = sum(target) + all_acc = total_tp / (total_target + epsilon) + + # Log Output - Global (Order: mIoU, macroF1, mAcc, mPrecision, allAcc) + self.trainer.logger.info( + "Val result: mIoU/macroF1/mAcc/mPrecision/allAcc " + "{:.4f}/{:.4f}/{:.4f}/{:.4f}/{:.4f}.".format( + m_iou, macro_f1, m_acc, m_precision, all_acc + ) + ) + + # Log Output - Per Class (Order: iou, acc, precision, f1) + for i in range(self.trainer.cfg.data.num_classes): + self.trainer.logger.info( + "Class_{idx}-{name} Metrics: IoU={iou:.4f}, accuracy={acc:.4f}, " + "precision={prec:.4f}, F1={f1:.4f}".format( + idx=i, + name=self.trainer.cfg.data.names[i], + iou=iou_class[i], + acc=acc_class[i], + prec=precision_class[i], + f1=f1_class[i], + ) + ) + + current_epoch = self.trainer.epoch + 1 + if self.trainer.writer is not None: + self.trainer.writer.add_scalar("val/loss", loss_avg, current_epoch) + self.trainer.writer.add_scalar("val/mIoU", m_iou, current_epoch) + self.trainer.writer.add_scalar("val/macroF1", macro_f1, current_epoch) + self.trainer.writer.add_scalar("val/mAcc", m_acc, current_epoch) + self.trainer.writer.add_scalar("val/mPrecision", m_precision, current_epoch) + self.trainer.writer.add_scalar("val/allAcc", all_acc, current_epoch) + + if self.trainer.cfg.enable_wandb: + wandb.log( + { + "val/loss": loss_avg, + "val/mIoU": m_iou, + "val/macroF1": macro_f1, + "val/mAcc": m_acc, + "val/mPrecision": m_precision, + "val/allAcc": all_acc, + "Epoch": current_epoch, + }, + step=wandb.run.step, + ) + + if self.write_cls_metrics: + for i in range(self.trainer.cfg.data.num_classes): + name = self.trainer.cfg.data.names[i] + self.trainer.writer.add_scalar( + f"val/cls_{i}-{name}_IoU", iou_class[i], current_epoch + ) + self.trainer.writer.add_scalar( + f"val/cls_{i}-{name}_F1", f1_class[i], current_epoch + ) + + if self.trainer.cfg.enable_wandb: + wandb.log( + { + f"val/cls_{i}-{name}_IoU": iou_class[i], + f"val/cls_{i}-{name}_F1": f1_class[i], + "Epoch": current_epoch, + }, + step=wandb.run.step, + ) + + self.trainer.logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + self.trainer.comm_info["current_metric_value"] = m_iou # save for saver + self.trainer.comm_info["current_metric_name"] = "mIoU" # save for saver + + def after_train(self): + self.trainer.logger.info( + "Best {}: {:.4f}".format("mIoU", self.trainer.best_metric_value) + ) + + +@HOOKS.register_module() +class InsSegEvaluator(HookBase): + def __init__(self, segment_ignore_index=(-1,), instance_ignore_index=-1): + self.segment_ignore_index = segment_ignore_index + self.instance_ignore_index = instance_ignore_index + + self.valid_class_names = None # update in before train + self.overlaps = np.append(np.arange(0.5, 0.95, 0.05), 0.25) + self.min_region_sizes = 100 + self.distance_threshes = float("inf") + self.distance_confs = -float("inf") + + def before_train(self): + self.valid_class_names = [ + self.trainer.cfg.data.names[i] + for i in range(self.trainer.cfg.data.num_classes) + if i not in self.segment_ignore_index + ] + + def after_epoch(self): + if self.trainer.cfg.evaluate: + self.eval() + + def associate_instances(self, pred, segment, instance): + segment = segment.cpu().numpy() + instance = instance.cpu().numpy() + void_mask = np.in1d(segment, self.segment_ignore_index) + + assert ( + pred["pred_classes"].shape[0] + == pred["pred_scores"].shape[0] + == pred["pred_masks"].shape[0] + ) + assert pred["pred_masks"].shape[1] == segment.shape[0] == instance.shape[0] + # get gt instances + gt_instances = dict() + for i in range(self.trainer.cfg.data.num_classes): + if i not in self.segment_ignore_index: + gt_instances[self.trainer.cfg.data.names[i]] = [] + instance_ids, idx, counts = np.unique( + instance, return_index=True, return_counts=True + ) + segment_ids = segment[idx] + for i in range(len(instance_ids)): + if instance_ids[i] == self.instance_ignore_index: + continue + if segment_ids[i] in self.segment_ignore_index: + continue + gt_inst = dict() + gt_inst["instance_id"] = instance_ids[i] + gt_inst["segment_id"] = segment_ids[i] + gt_inst["dist_conf"] = 0.0 + gt_inst["med_dist"] = -1.0 + gt_inst["vert_count"] = counts[i] + gt_inst["matched_pred"] = [] + gt_instances[self.trainer.cfg.data.names[segment_ids[i]]].append(gt_inst) + + # get pred instances and associate with gt + pred_instances = dict() + for i in range(self.trainer.cfg.data.num_classes): + if i not in self.segment_ignore_index: + pred_instances[self.trainer.cfg.data.names[i]] = [] + instance_id = 0 + for i in range(len(pred["pred_classes"])): + if pred["pred_classes"][i] in self.segment_ignore_index: + continue + pred_inst = dict() + pred_inst["uuid"] = uuid4() + pred_inst["instance_id"] = instance_id + pred_inst["segment_id"] = pred["pred_classes"][i] + pred_inst["confidence"] = pred["pred_scores"][i] + pred_inst["mask"] = np.not_equal(pred["pred_masks"][i], 0) + pred_inst["vert_count"] = np.count_nonzero(pred_inst["mask"]) + pred_inst["void_intersection"] = np.count_nonzero( + np.logical_and(void_mask, pred_inst["mask"]) + ) + if pred_inst["vert_count"] < self.min_region_sizes: + continue # skip if empty + segment_name = self.trainer.cfg.data.names[pred_inst["segment_id"]] + matched_gt = [] + for gt_idx, gt_inst in enumerate(gt_instances[segment_name]): + intersection = np.count_nonzero( + np.logical_and( + instance == gt_inst["instance_id"], pred_inst["mask"] + ) + ) + if intersection > 0: + gt_inst_ = gt_inst.copy() + pred_inst_ = pred_inst.copy() + gt_inst_["intersection"] = intersection + pred_inst_["intersection"] = intersection + matched_gt.append(gt_inst_) + gt_inst["matched_pred"].append(pred_inst_) + pred_inst["matched_gt"] = matched_gt + pred_instances[segment_name].append(pred_inst) + instance_id += 1 + return gt_instances, pred_instances + + def evaluate_matches(self, scenes): + overlaps = self.overlaps + min_region_sizes = [self.min_region_sizes] + dist_threshes = [self.distance_threshes] + dist_confs = [self.distance_confs] + + # results: class x overlap + ap_table = np.zeros( + (len(dist_threshes), len(self.valid_class_names), len(overlaps)), float + ) + for di, (min_region_size, distance_thresh, distance_conf) in enumerate( + zip(min_region_sizes, dist_threshes, dist_confs) + ): + for oi, overlap_th in enumerate(overlaps): + pred_visited = {} + for scene in scenes: + for _ in scene["pred"]: + for label_name in self.valid_class_names: + for p in scene["pred"][label_name]: + if "uuid" in p: + pred_visited[p["uuid"]] = False + for li, label_name in enumerate(self.valid_class_names): + y_true = np.empty(0) + y_score = np.empty(0) + hard_false_negatives = 0 + has_gt = False + has_pred = False + for scene in scenes: + pred_instances = scene["pred"][label_name] + gt_instances = scene["gt"][label_name] + # filter groups in ground truth + gt_instances = [ + gt + for gt in gt_instances + if gt["vert_count"] >= min_region_size + and gt["med_dist"] <= distance_thresh + and gt["dist_conf"] >= distance_conf + ] + if gt_instances: + has_gt = True + if pred_instances: + has_pred = True + + cur_true = np.ones(len(gt_instances)) + cur_score = np.ones(len(gt_instances)) * (-float("inf")) + cur_match = np.zeros(len(gt_instances), dtype=bool) + # collect matches + for gti, gt in enumerate(gt_instances): + found_match = False + for pred in gt["matched_pred"]: + # greedy assignments + if pred_visited[pred["uuid"]]: + continue + overlap = float(pred["intersection"]) / ( + gt["vert_count"] + + pred["vert_count"] + - pred["intersection"] + ) + if overlap > overlap_th: + confidence = pred["confidence"] + # if already have a prediction for this gt, + # the prediction with the lower score is automatically a false positive + if cur_match[gti]: + max_score = max(cur_score[gti], confidence) + min_score = min(cur_score[gti], confidence) + cur_score[gti] = max_score + # append false positive + cur_true = np.append(cur_true, 0) + cur_score = np.append(cur_score, min_score) + cur_match = np.append(cur_match, True) + # otherwise set score + else: + found_match = True + cur_match[gti] = True + cur_score[gti] = confidence + pred_visited[pred["uuid"]] = True + if not found_match: + hard_false_negatives += 1 + # remove non-matched ground truth instances + cur_true = cur_true[cur_match] + cur_score = cur_score[cur_match] + + # collect non-matched predictions as false positive + for pred in pred_instances: + found_gt = False + for gt in pred["matched_gt"]: + overlap = float(gt["intersection"]) / ( + gt["vert_count"] + + pred["vert_count"] + - gt["intersection"] + ) + if overlap > overlap_th: + found_gt = True + break + if not found_gt: + num_ignore = pred["void_intersection"] + for gt in pred["matched_gt"]: + if gt["segment_id"] in self.segment_ignore_index: + num_ignore += gt["intersection"] + # small ground truth instances + if ( + gt["vert_count"] < min_region_size + or gt["med_dist"] > distance_thresh + or gt["dist_conf"] < distance_conf + ): + num_ignore += gt["intersection"] + proportion_ignore = ( + float(num_ignore) / pred["vert_count"] + ) + # if not ignored append false positive + if proportion_ignore <= overlap_th: + cur_true = np.append(cur_true, 0) + confidence = pred["confidence"] + cur_score = np.append(cur_score, confidence) + + # append to overall results + y_true = np.append(y_true, cur_true) + y_score = np.append(y_score, cur_score) + + # compute average precision + if has_gt and has_pred: + # compute precision recall curve first + + # sorting and cumsum + score_arg_sort = np.argsort(y_score) + y_score_sorted = y_score[score_arg_sort] + y_true_sorted = y_true[score_arg_sort] + y_true_sorted_cumsum = np.cumsum(y_true_sorted) + + # unique thresholds + thresholds, unique_indices = np.unique( + y_score_sorted, return_index=True + ) + num_prec_recall = len(unique_indices) + 1 + + # prepare precision recall + num_examples = len(y_score_sorted) + # https://github.com/ScanNet/ScanNet/pull/26 + # all predictions are non-matched but also all of them are ignored and not counted as FP + # y_true_sorted_cumsum is empty + # num_true_examples = y_true_sorted_cumsum[-1] + num_true_examples = ( + y_true_sorted_cumsum[-1] + if len(y_true_sorted_cumsum) > 0 + else 0 + ) + precision = np.zeros(num_prec_recall) + recall = np.zeros(num_prec_recall) + + # deal with the first point + y_true_sorted_cumsum = np.append(y_true_sorted_cumsum, 0) + # deal with remaining + for idx_res, idx_scores in enumerate(unique_indices): + cumsum = y_true_sorted_cumsum[idx_scores - 1] + tp = num_true_examples - cumsum + fp = num_examples - idx_scores - tp + fn = cumsum + hard_false_negatives + p = float(tp) / (tp + fp) + r = float(tp) / (tp + fn) + precision[idx_res] = p + recall[idx_res] = r + + # first point in curve is artificial + precision[-1] = 1.0 + recall[-1] = 0.0 + + # compute average of precision-recall curve + recall_for_conv = np.copy(recall) + recall_for_conv = np.append(recall_for_conv[0], recall_for_conv) + recall_for_conv = np.append(recall_for_conv, 0.0) + + stepWidths = np.convolve( + recall_for_conv, [-0.5, 0, 0.5], "valid" + ) + # integrate is now simply a dot product + ap_current = np.dot(precision, stepWidths) + + elif has_gt: + ap_current = 0.0 + else: + ap_current = float("nan") + ap_table[di, li, oi] = ap_current + d_inf = 0 + o50 = np.where(np.isclose(self.overlaps, 0.5)) + o25 = np.where(np.isclose(self.overlaps, 0.25)) + oAllBut25 = np.where(np.logical_not(np.isclose(self.overlaps, 0.25))) + ap_scores = dict() + ap_scores["all_ap"] = np.nanmean(ap_table[d_inf, :, oAllBut25]) + ap_scores["all_ap_50%"] = np.nanmean(ap_table[d_inf, :, o50]) + ap_scores["all_ap_25%"] = np.nanmean(ap_table[d_inf, :, o25]) + ap_scores["classes"] = {} + for li, label_name in enumerate(self.valid_class_names): + ap_scores["classes"][label_name] = {} + ap_scores["classes"][label_name]["ap"] = np.average( + ap_table[d_inf, li, oAllBut25] + ) + ap_scores["classes"][label_name]["ap50%"] = np.average( + ap_table[d_inf, li, o50] + ) + ap_scores["classes"][label_name]["ap25%"] = np.average( + ap_table[d_inf, li, o25] + ) + return ap_scores + + def eval(self): + self.trainer.logger.info(">>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>") + self.trainer.model.eval() + scenes = {} + for i, input_dict in enumerate(self.trainer.val_loader): + assert ( + len(input_dict["offset"]) == 1 + ) # currently only support bs 1 for each GPU + data_name = input_dict.pop("name")[0] + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + with torch.no_grad(): + output_dict = self.trainer.model(input_dict) + + loss = output_dict["loss"] + + segment = input_dict["segment"] + instance = input_dict["instance"] + # map to origin + if "origin_coord" in input_dict.keys(): + idx, _ = pointops.knn_query( + 1, + input_dict["coord"].float(), + input_dict["offset"].int(), + input_dict["origin_coord"].float(), + input_dict["origin_offset"].int(), + ) + idx = idx.cpu().flatten().long() + output_dict["pred_masks"] = output_dict["pred_masks"][:, idx] + segment = input_dict["origin_segment"] + instance = input_dict["origin_instance"] + + gt_instances, pred_instance = self.associate_instances( + output_dict, segment, instance + ) + scenes[data_name] = dict(gt=gt_instances, pred=pred_instance) + + self.trainer.storage.put_scalar("val_loss", loss.item()) + self.trainer.logger.info( + "Test: [{iter}/{max_iter}] " + "Loss {loss:.4f} ".format( + iter=i + 1, max_iter=len(self.trainer.val_loader), loss=loss.item() + ) + ) + + loss_avg = self.trainer.storage.history("val_loss").avg + comm.synchronize() + scenes_sync = comm.gather(scenes, dst=0) + + if comm.is_main_process(): + scenes = {} + for _ in range(len(scenes_sync)): + r = scenes_sync.pop() + scenes.update(r) + del r + scenes = list(scenes.values()) + ap_scores = self.evaluate_matches(scenes) + all_ap = ap_scores["all_ap"] + all_ap_50 = ap_scores["all_ap_50%"] + all_ap_25 = ap_scores["all_ap_25%"] + self.trainer.logger.info( + "Val result: mAP/AP50/AP25 {:.4f}/{:.4f}/{:.4f}.".format( + all_ap, all_ap_50, all_ap_25 + ) + ) + for i, label_name in enumerate(self.valid_class_names): + ap = ap_scores["classes"][label_name]["ap"] + ap_50 = ap_scores["classes"][label_name]["ap50%"] + ap_25 = ap_scores["classes"][label_name]["ap25%"] + self.trainer.logger.info( + "Class_{idx}-{name} Result: AP/AP50/AP25 {AP:.4f}/{AP50:.4f}/{AP25:.4f}".format( + idx=i, name=label_name, AP=ap, AP50=ap_50, AP25=ap_25 + ) + ) + current_epoch = self.trainer.epoch + 1 + if self.trainer.writer is not None: + self.trainer.writer.add_scalar("val/loss", loss_avg, current_epoch) + self.trainer.writer.add_scalar("val/mAP", all_ap, current_epoch) + self.trainer.writer.add_scalar("val/AP50", all_ap_50, current_epoch) + self.trainer.writer.add_scalar("val/AP25", all_ap_25, current_epoch) + if self.trainer.cfg.enable_wandb: + wandb.log( + { + "Epoch": current_epoch, + "val/loss": loss_avg, + "val/mAP": all_ap, + "val/AP50": all_ap_50, + "val/AP25": all_ap_25, + }, + step=wandb.run.step, + ) + self.trainer.logger.info( + "<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<" + ) + self.trainer.comm_info["current_metric_value"] = all_ap_50 # save for saver + self.trainer.comm_info["current_metric_name"] = "AP50" # save for saver + + +@HOOKS.register_module() +class ShapeNetPartSegEvaluator(HookBase): + def __init__(self, write_cls_iou=False): + self.write_cls_iou = write_cls_iou + + def before_train(self): + if self.trainer.writer is not None and self.trainer.cfg.enable_wandb: + wandb.define_metric("val/*", step_metric="Epoch") + + def after_epoch(self): + if self.trainer.cfg.evaluate: + self.eval() + + def eval(self): + self.trainer.logger.info( + ">>>>>>>>>>>>>>>> Start Part Segmentation Evaluation >>>>>>>>>>>>>>>>" + ) + self.trainer.model.eval() + + num_categories = len(self.trainer.val_loader.dataset.categories) + total_iou_category = torch.zeros(num_categories, device="cuda") + total_iou_count = torch.zeros(num_categories, device="cuda") + + # Iterate over all batches in the validation loader + for i, input_dict in enumerate(self.trainer.val_loader): + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + + with torch.no_grad(): + output_dict = self.trainer.model(input_dict) + + pred_scores = output_dict["seg_logits"] + pred_labels = torch.argmax(pred_scores, dim=-1) + + segment = input_dict["segment"] + cls_token = input_dict["cls_token"][0].cpu().numpy() + + if "inverse" in input_dict.keys(): + assert ( + "origin_segment" in input_dict.keys() + ), "origin_segment must be provided with inverse" + pred_labels = pred_labels[input_dict["inverse"]] + segment = input_dict["origin_segment"] + + category_name = self.trainer.val_loader.dataset.categories[cls_token] + parts_idx = self.trainer.val_loader.dataset.category2part[category_name] + parts_iou = torch.zeros(len(parts_idx), device="cuda") + for k, part_id in enumerate(parts_idx): + if (torch.sum(segment == part_id) == 0) and ( + torch.sum(pred_labels == part_id) == 0 + ): + parts_iou[k] = ( + 1.0 # This part is correctly not predicted and not present + ) + else: + intersection = torch.sum( + (segment == part_id) & (pred_labels == part_id) + ) + union = torch.sum((segment == part_id) | (pred_labels == part_id)) + parts_iou[k] = intersection / (union + 1e-10) + + sample_miou = parts_iou.mean() + + total_iou_category[cls_token] += sample_miou + total_iou_count[cls_token] += 1 + + if comm.get_world_size() > 1: + dist.all_reduce(total_iou_category), dist.all_reduce(total_iou_count) + total_iou_count = total_iou_count.cpu().numpy() + total_iou_category = total_iou_category.cpu().numpy() + # Instance-wise mIoU: average of all sample mIoUs + ins_mIoU = total_iou_category.sum() / (total_iou_count.sum() + 1e-10) + # Category-wise mIoU: average of per-category mIoUs + iou_per_cat = total_iou_category / (total_iou_count + 1e-10) + # Only average over categories that were actually present in the validation set + cat_mIoU = np.mean(iou_per_cat[total_iou_count > 0]) + + self.trainer.logger.info( + "Val result: ins.mIoU/cat.mIoU {:.4f}/{:.4f}.".format(ins_mIoU, cat_mIoU) + ) + + # Log detailed results for each category + for i in range(num_categories): + if total_iou_count[i] > 0: + self.trainer.logger.info( + "Class_{idx}-{name} Result: iou_cat/num_sample {iou_cat:.4f}/{iou_count:.0f}".format( + idx=i, + name=self.trainer.val_loader.dataset.categories[i], + iou_cat=iou_per_cat[i], + iou_count=total_iou_count[i], + ) + ) + + current_epoch = self.trainer.epoch + 1 + if self.trainer.writer is not None: + self.trainer.writer.add_scalar("val/ins_mIoU", ins_mIoU, current_epoch) + self.trainer.writer.add_scalar("val/cat_mIoU", cat_mIoU, current_epoch) + if self.trainer.cfg.enable_wandb: + wandb.log( + { + "Epoch": current_epoch, + "val/ins_mIoU": ins_mIoU, + "val/cat_mIoU": cat_mIoU, + }, + step=wandb.run.step, + ) + + if self.write_cls_iou: + for i in range(num_categories): + if total_iou_count[i] > 0: + category_name = self.trainer.val_loader.dataset.categories[i] + self.trainer.writer.add_scalar( + f"val/cls_{i}-{category_name}_IoU", + iou_per_cat[i], + current_epoch, + ) + + self.trainer.logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + + # Save the primary metric for checkpointing logic (e.g., saving the best model) + # Category mIoU is often a more robust metric for this. + self.trainer.comm_info["current_metric_value"] = cat_mIoU + self.trainer.comm_info["current_metric_name"] = "cat_mIoU" + + def after_train(self): + # Log the best performing metric at the very end of training. + self.trainer.logger.info( + "Best {}: {:.4f}".format( + self.trainer.comm_info.get("current_metric_name", "metric"), + self.trainer.best_metric_value, + ) + ) + + +@HOOKS.register_module() +class PartNetEPartSegEvaluator(HookBase): + def __init__(self, num_parts=None, write_part_iou=False): + self.num_parts = sum(num_parts) + self.write_part_iou = write_part_iou + + def before_train(self): + if self.trainer.writer is not None and self.trainer.cfg.enable_wandb: + wandb.define_metric("val/*", step_metric="Epoch") + + def after_epoch(self): + if self.trainer.cfg.evaluate: + self.eval() + + def eval(self): + self.trainer.logger.info( + ">>>>>>>>>>>>>>>> Start Part Segmentation Evaluation >>>>>>>>>>>>>>>>" + ) + self.trainer.model.eval() + + # Initialize numpy arrays to aggregate results over the entire validation set. + num_categories = len(self.trainer.val_loader.dataset.categories) + total_iou_parts = torch.zeros(self.num_parts, device="cuda") + total_iou_count = torch.zeros(self.num_parts, device="cuda") + + # Iterate over all batches in the validation loader + for i, input_dict in enumerate(self.trainer.val_loader): + assert len(input_dict["offset"]) == 1 + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + + with torch.no_grad(): + output_dict = self.trainer.model(input_dict) + + pred_scores = output_dict["seg_logits"] + segment = input_dict["segment"] + cls_token = input_dict["cls_token"][0].cpu().numpy() + category_name = self.trainer.val_loader.dataset.categories[cls_token] + parts_idx = self.trainer.val_loader.dataset.category2part[category_name] + pred_labels = torch.argmax(pred_scores, dim=-1) + + if "inverse" in input_dict.keys(): + assert ( + "origin_segment" in input_dict.keys() + ), "origin_segment must be provided with inverse" + pred_labels = pred_labels[input_dict["inverse"]] + segment = input_dict["origin_segment"] + + for k, part_id in enumerate(parts_idx): + if k == 0: + continue + if (segment == part_id).sum() == 0: + continue + if (torch.sum(segment == part_id) == 0) and ( + torch.sum(pred_labels == part_id) == 0 + ): + continue + else: + intersection = torch.sum( + (segment == part_id) & (pred_labels == part_id) + ) + union = torch.sum((segment == part_id) | (pred_labels == part_id)) + total_iou_parts[ + k + self.trainer.val_loader.dataset.num_part_offset[cls_token] + ] += intersection / (union + 1e-10) + total_iou_count[ + k + self.trainer.val_loader.dataset.num_part_offset[cls_token] + ] += 1 + if comm.get_world_size() > 1: + dist.all_reduce(total_iou_parts), dist.all_reduce(total_iou_count) + total_iou_count = total_iou_count.cpu().numpy() + total_iou_parts = total_iou_parts.cpu().numpy() + current_iou_count = total_iou_count[total_iou_count > 0] + current_iou_parts = total_iou_parts[total_iou_count > 0] + # part-wise mIoU: average of all sample mIoUs + part_mIoU = (current_iou_parts / current_iou_count).mean() + + self.trainer.logger.info("Val result: part mIoU {:.4f}.".format(part_mIoU)) + + # Log detailed results for each category + for i in range(self.num_parts): + if total_iou_count[i] > 0: + self.trainer.logger.info( + "Class_{idx}-{name} Result: iou_part/num_sample {iou_part:.4f}/{iou_count:.0f}".format( + idx=i, + name=self.trainer.val_loader.dataset.parts[i], + iou_part=total_iou_parts[i] / total_iou_count[i], + iou_count=total_iou_count[i], + ) + ) + + # Log metrics to TensorBoard / WandB + current_epoch = self.trainer.epoch + 1 + if self.trainer.writer is not None: + self.trainer.writer.add_scalar("val/part_mIoU", part_mIoU, current_epoch) + if self.trainer.cfg.enable_wandb: + wandb.log( + { + "Epoch": current_epoch, + "val/part_mIoU": part_mIoU, + }, + step=wandb.run.step, + ) + + if self.write_part_iou: + for i in range(self.num_parts): + if total_iou_count[i] > 0: + part_name = self.trainer.val_loader.dataset.parts[i] + self.trainer.writer.add_scalar( + f"val/part_{i}-{part_name}_IoU", + total_iou_parts[i] / total_iou_count[i], + current_epoch, + ) + + self.trainer.logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + + # Save the primary metric for checkpointing logic (e.g., saving the best model) + self.trainer.comm_info["current_metric_value"] = part_mIoU + self.trainer.comm_info["current_metric_name"] = "part_mIoU" + + def after_train(self): + """ + Log the best performing metric at the very end of training. + """ + self.trainer.logger.info( + "Best {}: {:.4f}".format( + self.trainer.comm_info.get("current_metric_name", "metric"), + self.trainer.best_metric_value, + ) + ) diff --git a/pointcept/engines/hooks/misc.py b/pointcept/engines/hooks/misc.py new file mode 100644 index 0000000000000000000000000000000000000000..28d9682ff8b288c71106f02e37b6aa435aa784c3 --- /dev/null +++ b/pointcept/engines/hooks/misc.py @@ -0,0 +1,553 @@ +""" +Misc Hook + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import sys +import glob +import os +import shutil +import time +import gc +import wandb +import torch +import torch.utils.data +from collections import OrderedDict + +if sys.version_info >= (3, 10): + from collections.abc import Sequence +else: + from collections import Sequence +from pointcept.utils.timer import Timer +from pointcept.utils.comm import is_main_process, synchronize +from pointcept.utils.cache import shared_dict +from pointcept.utils.scheduler import CosineScheduler +import pointcept.utils.comm as comm + +from .default import HookBase +from .builder import HOOKS + + +@HOOKS.register_module() +class IterationTimer(HookBase): + def __init__(self, warmup_iter=1): + self._warmup_iter = warmup_iter + self._start_time = time.perf_counter() + self._iter_timer = Timer() + self._remain_iter = 0 + + def before_train(self): + self._start_time = time.perf_counter() + _remain_epoch = self.trainer.max_epoch - self.trainer.start_epoch + self._remain_iter = _remain_epoch * len(self.trainer.train_loader) + + def before_epoch(self): + self._iter_timer.reset() + + def before_step(self): + data_time = self._iter_timer.seconds() + self.trainer.storage.put_scalar("data_time", data_time) + + def after_step(self): + batch_time = self._iter_timer.seconds() + self._iter_timer.reset() + self.trainer.storage.put_scalar("batch_time", batch_time) + self._remain_iter -= 1 + remain_time = self._remain_iter * self.trainer.storage.history("batch_time").avg + t_m, t_s = divmod(remain_time, 60) + t_h, t_m = divmod(t_m, 60) + remain_time = "{:02d}:{:02d}:{:02d}".format(int(t_h), int(t_m), int(t_s)) + if "iter_info" in self.trainer.comm_info.keys(): + info = ( + "Data {data_time_val:.3f} ({data_time_avg:.3f}) " + "Batch {batch_time_val:.3f} ({batch_time_avg:.3f}) " + "Remain {remain_time} ".format( + data_time_val=self.trainer.storage.history("data_time").val, + data_time_avg=self.trainer.storage.history("data_time").avg, + batch_time_val=self.trainer.storage.history("batch_time").val, + batch_time_avg=self.trainer.storage.history("batch_time").avg, + remain_time=remain_time, + ) + ) + self.trainer.comm_info["iter_info"] += info + if self.trainer.comm_info["iter"] <= self._warmup_iter: + self.trainer.storage.history("data_time").reset() + self.trainer.storage.history("batch_time").reset() + + +@HOOKS.register_module() +class InformationWriter(HookBase): + def __init__(self): + self.curr_iter = 0 + self.model_output_keys = [] + + def before_train(self): + self.trainer.comm_info["iter_info"] = "" + self.curr_iter = self.trainer.start_epoch * len(self.trainer.train_loader) + if self.trainer.writer is not None and self.trainer.cfg.enable_wandb: + wandb.define_metric("params/*", step_metric="Iter") + wandb.define_metric("train_batch/*", step_metric="Iter") + wandb.define_metric("train/*", step_metric="Epoch") + + def before_step(self): + self.curr_iter += 1 + info = "Train: [{epoch}/{max_epoch}][{iter}/{max_iter}] ".format( + epoch=self.trainer.epoch + 1, + max_epoch=self.trainer.max_epoch, + iter=self.trainer.comm_info["iter"] + 1, + max_iter=len(self.trainer.train_loader), + ) + self.trainer.comm_info["iter_info"] += info + + def after_step(self): + if "model_output_dict" in self.trainer.comm_info.keys(): + model_output_dict = self.trainer.comm_info["model_output_dict"] + self.model_output_keys = model_output_dict.keys() + for key in self.model_output_keys: + self.trainer.storage.put_scalar(key, model_output_dict[key].item()) + + for key in self.model_output_keys: + self.trainer.comm_info["iter_info"] += "{key}: {value:.4f} ".format( + key=key, value=self.trainer.storage.history(key).val + ) + lr = self.trainer.optimizer.state_dict()["param_groups"][0]["lr"] + self.trainer.comm_info["iter_info"] += "Lr: {lr:.5f}".format(lr=lr) + self.trainer.logger.info(self.trainer.comm_info["iter_info"]) + self.trainer.comm_info["iter_info"] = "" # reset iter info + if self.trainer.writer is not None: + self.trainer.writer.add_scalar("params/lr", lr, self.curr_iter) + for key in self.model_output_keys: + self.trainer.writer.add_scalar( + "train_batch/" + key, + self.trainer.storage.history(key).val, + self.curr_iter, + ) + if self.trainer.cfg.enable_wandb: + + wandb.log( + {"Iter": self.curr_iter, "params/lr": lr}, step=self.curr_iter + ) + for key in self.model_output_keys: + wandb.log( + { + "Iter": self.curr_iter, + f"train_batch/{key}": self.trainer.storage.history(key).val, + }, + step=wandb.run.step, + ) + + def after_epoch(self): + epoch_info = "Train result: " + for key in self.model_output_keys: + epoch_info += "{key}: {value:.4f} ".format( + key=key, value=self.trainer.storage.history(key).avg + ) + self.trainer.logger.info(epoch_info) + if self.trainer.writer is not None: + for key in self.model_output_keys: + self.trainer.writer.add_scalar( + "train/" + key, + self.trainer.storage.history(key).avg, + self.trainer.epoch + 1, + ) + + if self.trainer.cfg.enable_wandb: + + for key in self.model_output_keys: + wandb.log( + { + "Epoch": self.trainer.epoch + 1, + f"train/{key}": self.trainer.storage.history(key).avg, + }, + step=wandb.run.step, + ) + + +@HOOKS.register_module() +class CheckpointSaver(HookBase): + def __init__(self, save_freq=None): + self.save_freq = save_freq # None or int, None indicate only save model last + + def after_epoch(self): + if is_main_process(): + is_best = False + if self.trainer.cfg.evaluate: + current_metric_value = self.trainer.comm_info["current_metric_value"] + current_metric_name = self.trainer.comm_info["current_metric_name"] + if current_metric_value > self.trainer.best_metric_value: + self.trainer.best_metric_value = current_metric_value + is_best = True + self.trainer.logger.info( + "Best validation {} updated to: {:.4f}".format( + current_metric_name, current_metric_value + ) + ) + self.trainer.logger.info( + "Currently Best {}: {:.4f}".format( + current_metric_name, self.trainer.best_metric_value + ) + ) + + filename = os.path.join( + self.trainer.cfg.save_path, "model", "model_last.pth" + ) + self.trainer.logger.info("Saving checkpoint to: " + filename) + torch.save( + { + "epoch": self.trainer.epoch + 1, + "state_dict": self.trainer.model.state_dict(), + "optimizer": self.trainer.optimizer.state_dict(), + "scheduler": self.trainer.scheduler.state_dict(), + "scaler": ( + self.trainer.scaler.state_dict() + if self.trainer.cfg.enable_amp + else None + ), + "best_metric_value": self.trainer.best_metric_value, + }, + filename + ".tmp", + ) + os.replace(filename + ".tmp", filename) + if is_best: + shutil.copyfile( + filename, + os.path.join(self.trainer.cfg.save_path, "model", "model_best.pth"), + ) + if self.save_freq and (self.trainer.epoch + 1) % self.save_freq == 0: + shutil.copyfile( + filename, + os.path.join( + self.trainer.cfg.save_path, + "model", + f"epoch_{self.trainer.epoch + 1}.pth", + ), + ) + + +@HOOKS.register_module() +class CheckpointLoader(HookBase): + def __init__(self, keywords="", replacement=None, strict=False): + self.keywords = keywords + self.replacement = replacement if replacement is not None else keywords + self.strict = strict + + def before_train(self): + self.trainer.logger.info("=> Loading checkpoint & weight ...") + if self.trainer.cfg.weight and os.path.isfile(self.trainer.cfg.weight): + self.trainer.logger.info(f"Loading weight at: {self.trainer.cfg.weight}") + checkpoint = torch.load( + self.trainer.cfg.weight, + map_location=lambda storage, loc: storage.cuda(), + weights_only=False, + ) + self.trainer.logger.info( + f"Loading layer weights with keyword: {self.keywords}, " + f"replace keyword with: {self.replacement}" + ) + weight = OrderedDict() + for key, value in checkpoint["state_dict"].items(): + if not key.startswith("module."): + key = "module." + key # xxx.xxx -> module.xxx.xxx + # Now all keys contain "module." no matter DDP or not. + if self.keywords in key: + key = key.replace(self.keywords, self.replacement, 1) + if comm.get_world_size() == 1: + key = key[7:] # module.xxx.xxx -> xxx.xxx + weight[key] = value + load_state_info = self.trainer.model.load_state_dict( + weight, strict=self.strict + ) + self.trainer.logger.info(f"Missing keys: {load_state_info[0]}") + if self.trainer.cfg.resume: + self.trainer.logger.info( + f"Resuming train at eval epoch: {checkpoint['epoch']}" + ) + self.trainer.start_epoch = checkpoint["epoch"] + self.trainer.best_metric_value = checkpoint["best_metric_value"] + self.trainer.optimizer.load_state_dict(checkpoint["optimizer"]) + self.trainer.scheduler.load_state_dict(checkpoint["scheduler"]) + if self.trainer.cfg.enable_amp: + self.trainer.scaler.load_state_dict(checkpoint["scaler"]) + else: + self.trainer.logger.info(f"No weight found at: {self.trainer.cfg.weight}") + + +@HOOKS.register_module() +class PreciseEvaluator(HookBase): + def __init__(self, test_last=False): + self.test_last = test_last + + def after_train(self): + from pointcept.engines.test import TESTERS + + self.trainer.logger.info( + ">>>>>>>>>>>>>>>> Start Precise Evaluation >>>>>>>>>>>>>>>>" + ) + torch.cuda.empty_cache() + cfg = self.trainer.cfg + test_cfg = dict(cfg=cfg, model=self.trainer.model, **cfg.test) + tester = TESTERS.build(test_cfg) + if self.test_last: + self.trainer.logger.info("=> Testing on model_last ...") + else: + self.trainer.logger.info("=> Testing on model_best ...") + best_path = os.path.join( + self.trainer.cfg.save_path, "model", "model_best.pth" + ) + checkpoint = torch.load(best_path, weights_only=False) + weight = OrderedDict() + for key, value in checkpoint["state_dict"].items(): + if not key.startswith("module."): + key = "module." + key # xxx.xxx -> module.xxx.xxx + # Now all keys contain "module." no matter DDP or not. + if comm.get_world_size() == 1: + key = key[7:] # module.xxx.xxx -> xxx.xxx + weight[key] = value + tester.model.load_state_dict(weight, strict=True) + tester.test() + + +@HOOKS.register_module() +class DataCacheOperator(HookBase): + def __init__(self, data_root, split): + self.data_root = data_root + self.split = split + self.data_list = self.get_data_list() + + def get_data_list(self): + if isinstance(self.split, str): + data_list = glob.glob(os.path.join(self.data_root, self.split)) + elif isinstance(self.split, Sequence): + data_list = [] + for split in self.split: + data_list += glob.glob(os.path.join(self.data_root, split)) + else: + raise NotImplementedError + return data_list + + def get_cache_name(self, data_path): + data_name = data_path.replace(os.path.dirname(self.data_root), "") + return "pointcept" + data_name.replace(os.path.sep, "-") + + def before_train(self): + self.trainer.logger.info( + f"=> Caching dataset: {self.data_root}, split: {self.split} ..." + ) + if is_main_process(): + dataset = self.trainer.train_loader.dataset + for i in range(len(dataset)): + data_dict = dataset[i] + name = data_dict["name"] + shared_dict(f"Pointcept-{name}", data_dict) + synchronize() + + +@HOOKS.register_module() +class RuntimeProfiler(HookBase): + def __init__( + self, + forward=True, + backward=True, + interrupt=False, + warm_up=2, + sort_by="cuda_time_total", + row_limit=30, + ): + self.forward = forward + self.backward = backward + self.interrupt = interrupt + self.warm_up = warm_up + self.sort_by = sort_by + self.row_limit = row_limit + + def before_train(self): + self.trainer.logger.info("Profiling runtime ...") + from torch.profiler import profile, record_function, ProfilerActivity + + for i, input_dict in enumerate(self.trainer.train_loader): + if i == self.warm_up + 1: + break + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + if self.forward: + with profile( + activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], + record_shapes=True, + profile_memory=True, + with_stack=True, + ) as forward_prof: + with record_function("model_inference"): + output_dict = self.trainer.model(input_dict) + else: + output_dict = self.trainer.model(input_dict) + loss = output_dict["loss"] + if self.backward: + with profile( + activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], + record_shapes=True, + profile_memory=True, + with_stack=True, + ) as backward_prof: + with record_function("model_inference"): + loss.backward() + self.trainer.logger.info(f"Profile: [{i + 1}/{self.warm_up + 1}]") + if self.forward: + self.trainer.logger.info( + "Forward profile: \n" + + str( + forward_prof.key_averages().table( + sort_by=self.sort_by, row_limit=self.row_limit + ) + ) + ) + forward_prof.export_chrome_trace( + os.path.join(self.trainer.cfg.save_path, "forward_trace.json") + ) + + if self.backward: + self.trainer.logger.info( + "Backward profile: \n" + + str( + backward_prof.key_averages().table( + sort_by=self.sort_by, row_limit=self.row_limit + ) + ) + ) + backward_prof.export_chrome_trace( + os.path.join(self.trainer.cfg.save_path, "backward_trace.json") + ) + if self.interrupt: + sys.exit(0) + + +@HOOKS.register_module() +class RuntimeProfilerV2(HookBase): + def __init__( + self, + interrupt=False, + wait=1, + warmup=1, + active=10, + repeat=1, + sort_by="cuda_time_total", + row_limit=30, + ): + self.interrupt = interrupt + self.wait = wait + self.warmup = warmup + self.active = active + self.repeat = repeat + self.sort_by = sort_by + self.row_limit = row_limit + + def before_train(self): + self.trainer.logger.info("Profiling runtime ...") + from torch.profiler import ( + profile, + record_function, + ProfilerActivity, + schedule, + tensorboard_trace_handler, + ) + + prof = profile( + activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], + schedule=schedule( + wait=self.wait, + warmup=self.warmup, + active=self.active, + repeat=self.repeat, + ), + on_trace_ready=tensorboard_trace_handler(self.trainer.cfg.save_path), + record_shapes=True, + profile_memory=True, + with_stack=True, + ) + prof.start() + for i, input_dict in enumerate(self.trainer.train_loader): + if i >= (self.wait + self.warmup + self.active) * self.repeat: + break + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + with record_function("model_forward"): + output_dict = self.trainer.model(input_dict) + loss = output_dict["loss"] + with record_function("model_backward"): + loss.backward() + prof.step() + self.trainer.logger.info( + f"Profile: [{i + 1}/{(self.wait + self.warmup + self.active) * self.repeat}]" + ) + self.trainer.logger.info( + "Profile: \n" + + str( + prof.key_averages().table( + sort_by=self.sort_by, row_limit=self.row_limit + ) + ) + ) + prof.stop() + + if self.interrupt: + sys.exit(0) + + +@HOOKS.register_module() +class WeightDecaySchedular(HookBase): + def __init__( + self, + base_value=0.04, + final_value=0.2, + ): + self.base_value = base_value + self.final_value = final_value + self.scheduler = None + + def before_train(self): + curr_step = self.trainer.start_epoch * len(self.trainer.train_loader) + self.scheduler = CosineScheduler( + base_value=self.base_value, + final_value=self.final_value, + total_iters=self.trainer.cfg.scheduler.total_steps, + ) + self.scheduler.iter = curr_step + + def before_step(self): + wd = self.scheduler.step() + for param_group in self.trainer.optimizer.param_groups: + param_group["weight_decay"] = wd + if self.trainer.writer is not None: + self.trainer.writer.add_scalar("params/wd", wd, self.scheduler.iter) + + +@HOOKS.register_module() +class GarbageHandler(HookBase): + def __init__(self, interval=150, disable_auto=True, empty_cache=False): + self.interval = interval + self.disable_auto = disable_auto + self.empty_cache = empty_cache + self.iter = 1 + + def before_train(self): + if self.disable_auto: + gc.disable() + self.trainer.logger.info("Disable automatic garbage collection") + + def before_epoch(self): + self.iter = 1 + + def after_step(self): + if self.iter % self.interval == 0: + gc.collect() + if self.empty_cache: + torch.cuda.empty_cache() + self.trainer.logger.info("Garbage collected") + self.iter += 1 + + def after_train(self): + gc.collect() + torch.cuda.empty_cache() diff --git a/pointcept/engines/launch.py b/pointcept/engines/launch.py new file mode 100644 index 0000000000000000000000000000000000000000..f37976ea7412f8a7b68ab11b81b7ac5b1e566927 --- /dev/null +++ b/pointcept/engines/launch.py @@ -0,0 +1,136 @@ +""" +Launcher + +modified from detectron2(https://github.com/facebookresearch/detectron2) + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import logging +from datetime import timedelta +import torch +import torch.distributed as dist +import torch.multiprocessing as mp + +from pointcept.utils import comm + +__all__ = ["DEFAULT_TIMEOUT", "launch"] + +DEFAULT_TIMEOUT = timedelta(minutes=60) + + +def _find_free_port(): + import socket + + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + # Binding to port 0 will cause the OS to find an available port for us + sock.bind(("", 0)) + port = sock.getsockname()[1] + sock.close() + # NOTE: there is still a chance the port could be taken by other processes. + return port + + +def launch( + main_func, + num_gpus_per_machine, + num_machines=1, + machine_rank=0, + dist_url=None, + cfg=(), + timeout=DEFAULT_TIMEOUT, +): + """ + Launch multi-gpu or distributed training. + This function must be called on all machines involved in the training. + It will spawn child processes (defined by ``num_gpus_per_machine``) on each machine. + Args: + main_func: a function that will be called by `main_func(*args)` + num_gpus_per_machine (int): number of GPUs per machine + num_machines (int): the total number of machines + machine_rank (int): the rank of this machine + dist_url (str): url to connect to for distributed jobs, including protocol + e.g. "tcp://127.0.0.1:8686". + Can be set to "auto" to automatically select a free port on localhost + timeout (timedelta): timeout of the distributed workers + args (tuple): arguments passed to main_func + """ + world_size = num_machines * num_gpus_per_machine + if world_size > 1: + if dist_url == "auto": + assert ( + num_machines == 1 + ), "dist_url=auto not supported in multi-machine jobs." + port = _find_free_port() + dist_url = f"tcp://127.0.0.1:{port}" + if num_machines > 1 and dist_url.startswith("file://"): + logger = logging.getLogger(__name__) + logger.warning( + "file:// is not a reliable init_method in multi-machine jobs. Prefer tcp://" + ) + mp.spawn( + _distributed_worker, + nprocs=num_gpus_per_machine, + args=( + main_func, + world_size, + num_gpus_per_machine, + machine_rank, + dist_url, + cfg, + timeout, + ), + daemon=False, + ) + else: + main_func(*cfg) + + +def _distributed_worker( + local_rank, + main_func, + world_size, + num_gpus_per_machine, + machine_rank, + dist_url, + cfg, + timeout=DEFAULT_TIMEOUT, +): + assert ( + torch.cuda.is_available() + ), "cuda is not available. Please check your installation." + global_rank = machine_rank * num_gpus_per_machine + local_rank + try: + dist.init_process_group( + backend="NCCL", + init_method=dist_url, + world_size=world_size, + rank=global_rank, + timeout=timeout, + ) + except Exception as e: + logger = logging.getLogger(__name__) + logger.error("Process group URL: {}".format(dist_url)) + raise e + + # Setup the local process group (which contains ranks within the same machine) + assert comm._LOCAL_PROCESS_GROUP is None + num_machines = world_size // num_gpus_per_machine + for i in range(num_machines): + ranks_on_i = list( + range(i * num_gpus_per_machine, (i + 1) * num_gpus_per_machine) + ) + pg = dist.new_group(ranks_on_i) + if i == machine_rank: + comm._LOCAL_PROCESS_GROUP = pg + + assert num_gpus_per_machine <= torch.cuda.device_count() + torch.cuda.set_device(local_rank) + + # synchronize is needed here to prevent a possible timeout after calling init_process_group + # See: https://github.com/facebookresearch/maskrcnn-benchmark/issues/172 + comm.synchronize() + + main_func(*cfg) diff --git a/pointcept/engines/test.py b/pointcept/engines/test.py new file mode 100644 index 0000000000000000000000000000000000000000..02db14e3f0cacf67c344ac72f43b274464104762 --- /dev/null +++ b/pointcept/engines/test.py @@ -0,0 +1,1586 @@ +""" +Tester + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import json +from uuid import uuid4 +import os +import time +import numpy as np +from collections import OrderedDict +import torch +import torch.distributed as dist +import torch.nn.functional as F +import torch.utils.data + +from .defaults import create_ddp_model +import pointcept.utils.comm as comm +from pointcept.datasets import build_dataset, collate_fn +from pointcept.models import build_model +from pointcept.utils.logger import get_root_logger +from pointcept.utils.registry import Registry +from pointcept.utils.misc import ( + AverageMeter, + intersection_and_union, + intersection_and_union_gpu, + make_dirs, +) + +try: + import pointops +except: + pointops = None + +TESTERS = Registry("testers") + + +class TesterBase: + def __init__(self, cfg, model=None, test_loader=None, verbose=False) -> None: + torch.multiprocessing.set_sharing_strategy("file_system") + self.logger = get_root_logger( + log_file=os.path.join(cfg.save_path, "test.log"), + file_mode="a" if cfg.resume else "w", + ) + self.logger.info("=> Loading config ...") + self.cfg = cfg + self.verbose = verbose + if self.verbose and model is None: + # if model is not none, trigger tester with trainer, no need to print config + self.logger.info(f"Save path: {cfg.save_path}") + self.logger.info(f"Config:\n{cfg.pretty_text}") + if model is None: + self.logger.info("=> Building model ...") + self.model = self.build_model() + else: + self.model = model + if test_loader is None: + self.logger.info("=> Building test dataset & dataloader ...") + self.test_loader = self.build_test_loader() + else: + self.test_loader = test_loader + + def build_model(self): + model = build_model(self.cfg.model) + n_parameters = sum(p.numel() for p in model.parameters() if p.requires_grad) + self.logger.info(f"Num params: {n_parameters}") + model = create_ddp_model( + model.cuda(), + broadcast_buffers=False, + find_unused_parameters=self.cfg.find_unused_parameters, + ) + if os.path.isfile(self.cfg.weight): + self.logger.info(f"Loading weight at: {self.cfg.weight}") + checkpoint = torch.load(self.cfg.weight, weights_only=False) + weight = OrderedDict() + for key, value in checkpoint["state_dict"].items(): + if key.startswith("module."): + if comm.get_world_size() == 1: + key = key[7:] # module.xxx.xxx -> xxx.xxx + else: + if comm.get_world_size() > 1: + key = "module." + key # xxx.xxx -> module.xxx.xxx + weight[key] = value + model.load_state_dict(weight, strict=True) + self.logger.info( + "=> Loaded weight '{}' (epoch {})".format( + self.cfg.weight, checkpoint["epoch"] + ) + ) + else: + raise RuntimeError("=> No checkpoint found at '{}'".format(self.cfg.weight)) + return model + + def build_test_loader(self): + test_dataset = build_dataset(self.cfg.data.test) + if comm.get_world_size() > 1: + test_sampler = torch.utils.data.distributed.DistributedSampler(test_dataset) + else: + test_sampler = None + test_loader = torch.utils.data.DataLoader( + test_dataset, + batch_size=self.cfg.batch_size_test_per_gpu, + shuffle=False, + num_workers=self.cfg.batch_size_test_per_gpu, + pin_memory=True, + sampler=test_sampler, + collate_fn=self.__class__.collate_fn, + ) + return test_loader + + def test(self): + raise NotImplementedError + + @staticmethod + def collate_fn(batch): + raise collate_fn(batch) + + +@TESTERS.register_module() +class SemSegTester(TesterBase): + def test(self): + assert self.test_loader.batch_size == 1 + logger = get_root_logger() + logger.info(">>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>") + + batch_time = AverageMeter() + intersection_meter = AverageMeter() + union_meter = AverageMeter() + target_meter = AverageMeter() + self.model.eval() + + save_path = os.path.join(self.cfg.save_path, "result") + make_dirs(save_path) + # create submit folder only on main process + if ( + self.cfg.data.test.type == "ScanNetDataset" + or self.cfg.data.test.type == "ScanNet200Dataset" + or self.cfg.data.test.type == "ScanNetPPDataset" + ) and comm.is_main_process(): + make_dirs(os.path.join(save_path, "submit")) + elif ( + self.cfg.data.test.type == "SemanticKITTIDataset" and comm.is_main_process() + ): + make_dirs(os.path.join(save_path, "submit")) + elif self.cfg.data.test.type == "NuScenesDataset" and comm.is_main_process(): + import json + + make_dirs(os.path.join(save_path, "submit", "lidarseg", "test")) + make_dirs(os.path.join(save_path, "submit", "test")) + submission = dict( + meta=dict( + use_camera=False, + use_lidar=True, + use_radar=False, + use_map=False, + use_external=False, + ) + ) + with open( + os.path.join(save_path, "submit", "test", "submission.json"), "w" + ) as f: + json.dump(submission, f, indent=4) + comm.synchronize() + record = {} + # fragment inference + for idx, data_dict in enumerate(self.test_loader): + start = time.time() + data_dict = data_dict[0] # current assume batch size is 1 + fragment_list = data_dict.pop("fragment_list") + segment = data_dict.pop("segment") + data_name = data_dict.pop("name") + pred_save_path = os.path.join(save_path, "{}_pred.npy".format(data_name)) + if os.path.isfile(pred_save_path): + logger.info( + "{}/{}: {}, loaded pred and label.".format( + idx + 1, len(self.test_loader), data_name + ) + ) + pred = np.load(pred_save_path) + if "origin_segment" in data_dict.keys(): + segment = data_dict["origin_segment"] + else: + pred = torch.zeros((segment.size, self.cfg.data.num_classes)).cuda() + for i in range(len(fragment_list)): + fragment_batch_size = 1 + s_i, e_i = i * fragment_batch_size, min( + (i + 1) * fragment_batch_size, len(fragment_list) + ) + input_dict = collate_fn(fragment_list[s_i:e_i]) + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + idx_part = input_dict["index"] + with torch.no_grad(): + pred_part = self.model(input_dict)["seg_logits"] # (n, k) + pred_part = F.softmax(pred_part, -1) + if self.cfg.empty_cache: + torch.cuda.empty_cache() + bs = 0 + for be in input_dict["offset"]: + pred[idx_part[bs:be], :] += pred_part[bs:be] + bs = be + + logger.info( + "Test: {}/{}-{data_name}, Batch: {batch_idx}/{batch_num}".format( + idx + 1, + len(self.test_loader), + data_name=data_name, + batch_idx=i, + batch_num=len(fragment_list), + ) + ) + if self.cfg.data.test.type == "ScanNetPPDataset": + pred = pred.topk(3, dim=1)[1].data.cpu().numpy() + else: + pred = pred.max(1)[1].data.cpu().numpy() + if "origin_segment" in data_dict.keys(): + assert "inverse" in data_dict.keys() + pred = pred[data_dict["inverse"]] + segment = data_dict["origin_segment"] + np.save(pred_save_path, pred) + if ( + self.cfg.data.test.type == "ScanNetDataset" + or self.cfg.data.test.type == "ScanNet200Dataset" + ): + np.savetxt( + os.path.join(save_path, "submit", "{}.txt".format(data_name)), + self.test_loader.dataset.class2id[pred].reshape([-1, 1]), + fmt="%d", + ) + elif self.cfg.data.test.type == "ScanNetPPDataset": + np.savetxt( + os.path.join(save_path, "submit", "{}.txt".format(data_name)), + pred.astype(np.int32), + delimiter=",", + fmt="%d", + ) + pred = pred[:, 0] # for mIoU, TODO: support top3 mIoU + elif self.cfg.data.test.type == "SemanticKITTIDataset": + # 00_000000 -> 00, 000000 + sequence_name, frame_name = data_name.split("_") + os.makedirs( + os.path.join( + save_path, "submit", "sequences", sequence_name, "predictions" + ), + exist_ok=True, + ) + submit = pred.astype(np.uint32) + submit = np.vectorize( + self.test_loader.dataset.learning_map_inv.__getitem__ + )(submit).astype(np.uint32) + submit.tofile( + os.path.join( + save_path, + "submit", + "sequences", + sequence_name, + "predictions", + f"{frame_name}.label", + ) + ) + elif self.cfg.data.test.type == "NuScenesDataset": + np.array(pred + 1).astype(np.uint8).tofile( + os.path.join( + save_path, + "submit", + "lidarseg", + "test", + "{}_lidarseg.bin".format(data_name), + ) + ) + + intersection, union, target = intersection_and_union( + pred, segment, self.cfg.data.num_classes, self.cfg.data.ignore_index + ) + intersection_meter.update(intersection) + union_meter.update(union) + target_meter.update(target) + record[data_name] = dict( + intersection=intersection, union=union, target=target + ) + + mask = union != 0 + iou_class = intersection / (union + 1e-10) + iou = np.mean(iou_class[mask]) + acc = sum(intersection) / (sum(target) + 1e-10) + + m_iou = np.mean(intersection_meter.sum / (union_meter.sum + 1e-10)) + m_acc = np.mean(intersection_meter.sum / (target_meter.sum + 1e-10)) + + batch_time.update(time.time() - start) + logger.info( + "Test: {} [{}/{}]-{} " + "Batch {batch_time.val:.3f} ({batch_time.avg:.3f}) " + "Accuracy {acc:.4f} ({m_acc:.4f}) " + "mIoU {iou:.4f} ({m_iou:.4f})".format( + data_name, + idx + 1, + len(self.test_loader), + segment.size, + batch_time=batch_time, + acc=acc, + m_acc=m_acc, + iou=iou, + m_iou=m_iou, + ) + ) + + logger.info("Syncing ...") + comm.synchronize() + record_sync = comm.gather(record, dst=0) + + if comm.is_main_process(): + record = {} + for _ in range(len(record_sync)): + r = record_sync.pop() + record.update(r) + del r + intersection = np.sum( + [meters["intersection"] for _, meters in record.items()], axis=0 + ) + union = np.sum([meters["union"] for _, meters in record.items()], axis=0) + target = np.sum([meters["target"] for _, meters in record.items()], axis=0) + + if self.cfg.data.test.type == "S3DISDataset": + torch.save( + dict(intersection=intersection, union=union, target=target), + os.path.join(save_path, f"{self.test_loader.dataset.split}.pth"), + ) + + iou_class = intersection / (union + 1e-10) + accuracy_class = intersection / (target + 1e-10) + mIoU = np.mean(iou_class) + mAcc = np.mean(accuracy_class) + allAcc = sum(intersection) / (sum(target) + 1e-10) + + logger.info( + "Val result: mIoU/mAcc/allAcc {:.4f}/{:.4f}/{:.4f}".format( + mIoU, mAcc, allAcc + ) + ) + for i in range(self.cfg.data.num_classes): + logger.info( + "Class_{idx} - {name} Result: iou/accuracy {iou:.4f}/{accuracy:.4f}".format( + idx=i, + name=self.cfg.data.names[i], + iou=iou_class[i], + accuracy=accuracy_class[i], + ) + ) + logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + + @staticmethod + def collate_fn(batch): + return batch + + +@TESTERS.register_module() +class DINOSemSegTester(TesterBase): + def test(self): + assert self.test_loader.batch_size == 1 + logger = get_root_logger() + logger.info(">>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>") + + batch_time = AverageMeter() + intersection_meter = AverageMeter() + union_meter = AverageMeter() + target_meter = AverageMeter() + self.model.eval() + + save_path = os.path.join(self.cfg.save_path, "result") + make_dirs(save_path) + # create submit folder only on main process + if ( + self.cfg.data.test.type == "ScanNetDataset" + or self.cfg.data.test.type == "ScanNet200Dataset" + or self.cfg.data.test.type == "ScanNetPPDataset" + ) and comm.is_main_process(): + make_dirs(os.path.join(save_path, "submit")) + elif ( + self.cfg.data.test.type == "SemanticKITTIDataset" and comm.is_main_process() + ): + make_dirs(os.path.join(save_path, "submit")) + elif self.cfg.data.test.type == "NuScenesDataset" and comm.is_main_process(): + import json + + make_dirs(os.path.join(save_path, "submit", "lidarseg", "test")) + make_dirs(os.path.join(save_path, "submit", "test")) + submission = dict( + meta=dict( + use_camera=False, + use_lidar=True, + use_radar=False, + use_map=False, + use_external=False, + ) + ) + with open( + os.path.join(save_path, "submit", "test", "submission.json"), "w" + ) as f: + json.dump(submission, f, indent=4) + comm.synchronize() + record = {} + # fragment inference + for idx, data_dict in enumerate(self.test_loader): + end = time.time() + data_dict = data_dict[0] # current assume batch size is 1 + fragment_list = data_dict.pop("fragment_list") + segment = data_dict.pop("segment") + data_name = data_dict.pop("name") + dino_coord = data_dict.pop("dino_coord").cuda(non_blocking=True) + dino_feat = data_dict.pop("dino_feat").cuda(non_blocking=True) + dino_offset = data_dict.pop("dino_offset").cuda(non_blocking=True) + pred_save_path = os.path.join(save_path, "{}_pred.npy".format(data_name)) + if os.path.isfile(pred_save_path): + logger.info( + "{}/{}: {}, loaded pred and label.".format( + idx + 1, len(self.test_loader), data_name + ) + ) + pred = np.load(pred_save_path) + if "origin_segment" in data_dict.keys(): + segment = data_dict["origin_segment"] + else: + pred = torch.zeros((segment.size, self.cfg.data.num_classes)).cuda() + for i in range(len(fragment_list)): + fragment_batch_size = 1 + s_i, e_i = i * fragment_batch_size, min( + (i + 1) * fragment_batch_size, len(fragment_list) + ) + input_dict = collate_fn(fragment_list[s_i:e_i]) + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + input_dict["dino_coord"] = dino_coord + input_dict["dino_feat"] = dino_feat + input_dict["dino_offset"] = dino_offset + idx_part = input_dict["index"] + with torch.no_grad(): + pred_part = self.model(input_dict)["seg_logits"] # (n, k) + pred_part = F.softmax(pred_part, -1) + if self.cfg.empty_cache: + torch.cuda.empty_cache() + bs = 0 + for be in input_dict["offset"]: + pred[idx_part[bs:be], :] += pred_part[bs:be] + bs = be + + logger.info( + "Test: {}/{}-{data_name}, Batch: {batch_idx}/{batch_num}".format( + idx + 1, + len(self.test_loader), + data_name=data_name, + batch_idx=i, + batch_num=len(fragment_list), + ) + ) + if self.cfg.data.test.type == "ScanNetPPDataset": + pred = pred.topk(3, dim=1)[1].data.cpu().numpy() + else: + pred = pred.max(1)[1].data.cpu().numpy() + if "origin_segment" in data_dict.keys(): + assert "inverse" in data_dict.keys() + pred = pred[data_dict["inverse"]] + segment = data_dict["origin_segment"] + np.save(pred_save_path, pred) + if ( + self.cfg.data.test.type == "ScanNetDataset" + or self.cfg.data.test.type == "ScanNet200Dataset" + ): + np.savetxt( + os.path.join(save_path, "submit", "{}.txt".format(data_name)), + self.test_loader.dataset.class2id[pred].reshape([-1, 1]), + fmt="%d", + ) + elif self.cfg.data.test.type == "ScanNetPPDataset": + np.savetxt( + os.path.join(save_path, "submit", "{}.txt".format(data_name)), + pred.astype(np.int32), + delimiter=",", + fmt="%d", + ) + pred = pred[:, 0] # for mIoU, TODO: support top3 mIoU + elif self.cfg.data.test.type == "SemanticKITTIDataset": + # 00_000000 -> 00, 000000 + sequence_name, frame_name = data_name.split("_") + os.makedirs( + os.path.join( + save_path, "submit", "sequences", sequence_name, "predictions" + ), + exist_ok=True, + ) + submit = pred.astype(np.uint32) + submit = np.vectorize( + self.test_loader.dataset.learning_map_inv.__getitem__ + )(submit).astype(np.uint32) + submit.tofile( + os.path.join( + save_path, + "submit", + "sequences", + sequence_name, + "predictions", + f"{frame_name}.label", + ) + ) + elif self.cfg.data.test.type == "NuScenesDataset": + np.array(pred + 1).astype(np.uint8).tofile( + os.path.join( + save_path, + "submit", + "lidarseg", + "test", + "{}_lidarseg.bin".format(data_name), + ) + ) + + intersection, union, target = intersection_and_union( + pred, segment, self.cfg.data.num_classes, self.cfg.data.ignore_index + ) + intersection_meter.update(intersection) + union_meter.update(union) + target_meter.update(target) + record[data_name] = dict( + intersection=intersection, union=union, target=target + ) + + mask = union != 0 + iou_class = intersection / (union + 1e-10) + iou = np.mean(iou_class[mask]) + acc = sum(intersection) / (sum(target) + 1e-10) + + m_iou = np.mean(intersection_meter.sum / (union_meter.sum + 1e-10)) + m_acc = np.mean(intersection_meter.sum / (target_meter.sum + 1e-10)) + + batch_time.update(time.time() - end) + logger.info( + "Test: {} [{}/{}]-{} " + "Batch {batch_time.val:.3f} ({batch_time.avg:.3f}) " + "Accuracy {acc:.4f} ({m_acc:.4f}) " + "mIoU {iou:.4f} ({m_iou:.4f})".format( + data_name, + idx + 1, + len(self.test_loader), + segment.size, + batch_time=batch_time, + acc=acc, + m_acc=m_acc, + iou=iou, + m_iou=m_iou, + ) + ) + + logger.info("Syncing ...") + comm.synchronize() + record_sync = comm.gather(record, dst=0) + + if comm.is_main_process(): + record = {} + for _ in range(len(record_sync)): + r = record_sync.pop() + record.update(r) + del r + intersection = np.sum( + [meters["intersection"] for _, meters in record.items()], axis=0 + ) + union = np.sum([meters["union"] for _, meters in record.items()], axis=0) + target = np.sum([meters["target"] for _, meters in record.items()], axis=0) + + if self.cfg.data.test.type == "S3DISDataset": + torch.save( + dict(intersection=intersection, union=union, target=target), + os.path.join(save_path, f"{self.test_loader.dataset.split}.pth"), + ) + + iou_class = intersection / (union + 1e-10) + accuracy_class = intersection / (target + 1e-10) + mIoU = np.mean(iou_class) + mAcc = np.mean(accuracy_class) + allAcc = sum(intersection) / (sum(target) + 1e-10) + + logger.info( + "Val result: mIoU/mAcc/allAcc {:.4f}/{:.4f}/{:.4f}".format( + mIoU, mAcc, allAcc + ) + ) + for i in range(self.cfg.data.num_classes): + logger.info( + "Class_{idx} - {name} Result: iou/accuracy {iou:.4f}/{accuracy:.4f}".format( + idx=i, + name=self.cfg.data.names[i], + iou=iou_class[i], + accuracy=accuracy_class[i], + ) + ) + logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + + @staticmethod + def collate_fn(batch): + return batch + + +@TESTERS.register_module() +class ClsTester(TesterBase): + def test(self): + logger = get_root_logger() + logger.info(">>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>") + batch_time = AverageMeter() + intersection_meter = AverageMeter() + union_meter = AverageMeter() + target_meter = AverageMeter() + record = {} + self.model.eval() + + for i, input_dict in enumerate(self.test_loader): + data_name = input_dict.get("name", None) + if data_name is None: + raise RuntimeError( + "ClsTester requires sample `name` for deduplicated evaluation." + ) + if isinstance(data_name, str): + data_name = [data_name] + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + end = time.time() + with torch.no_grad(): + output_dict = self.model(input_dict) + output = output_dict["cls_logits"] + pred = output.max(1)[1] + label = input_dict["category"] + if len(data_name) != pred.shape[0]: + raise RuntimeError( + "Number of sample names does not match batch size in ClsTester." + ) + for b, name in enumerate(data_name): + sample_pred = pred[b : b + 1].reshape(-1) + sample_label = label[b : b + 1].reshape(-1) + sample_intersection, sample_union, sample_target = ( + intersection_and_union_gpu( + sample_pred, + sample_label, + self.cfg.data.num_classes, + self.cfg.data.ignore_index, + ) + ) + record[name] = dict( + intersection=sample_intersection.cpu().numpy(), + union=sample_union.cpu().numpy(), + target=sample_target.cpu().numpy(), + ) + intersection, union, target = intersection_and_union_gpu( + pred, label, self.cfg.data.num_classes, self.cfg.data.ignore_index + ) + intersection, union, target = ( + intersection.cpu().numpy(), + union.cpu().numpy(), + target.cpu().numpy(), + ) + intersection_meter.update(intersection), union_meter.update( + union + ), target_meter.update(target) + + accuracy = sum(intersection_meter.val) / (sum(target_meter.val) + 1e-10) + batch_time.update(time.time() - end) + + logger.info( + "Test: [{}/{}] " + "Batch {batch_time.val:.3f} ({batch_time.avg:.3f}) " + "Accuracy {accuracy:.4f} ".format( + i + 1, + len(self.test_loader), + batch_time=batch_time, + accuracy=accuracy, + ) + ) + + logger.info("Syncing ...") + comm.synchronize() + record_sync = comm.gather(record, dst=0) + + if comm.is_main_process(): + record = {} + for _ in range(len(record_sync)): + r = record_sync.pop() + record.update(r) + del r + intersection = np.sum( + [meters["intersection"] for _, meters in record.items()], axis=0 + ) + union = np.sum([meters["union"] for _, meters in record.items()], axis=0) + target = np.sum([meters["target"] for _, meters in record.items()], axis=0) + + iou_class = intersection / (union + 1e-10) + accuracy_class = intersection / (target + 1e-10) + mIoU = np.mean(iou_class) + mAcc = np.mean(accuracy_class) + allAcc = sum(intersection) / (sum(target) + 1e-10) + logger.info( + "Val result: mIoU/mAcc/allAcc {:.4f}/{:.4f}/{:.4f}.".format( + mIoU, mAcc, allAcc + ) + ) + + for i in range(self.cfg.data.num_classes): + logger.info( + "Class_{idx} - {name} Result: iou/accuracy {iou:.4f}/{accuracy:.4f}".format( + idx=i, + name=self.cfg.data.names[i], + iou=iou_class[i], + accuracy=accuracy_class[i], + ) + ) + logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + + @staticmethod + def collate_fn(batch): + return collate_fn(batch) + + +@TESTERS.register_module() +class ClsVotingTester(TesterBase): + def __init__( + self, + num_repeat=100, + metric="allAcc", + **kwargs, + ): + super().__init__(**kwargs) + self.num_repeat = num_repeat + self.metric = metric + self.best_idx = 0 + self.best_record = None + self.best_metric = 0 + + def test(self): + for i in range(self.num_repeat): + logger = get_root_logger() + logger.info(f">>>>>>>>>>>>>>>> Start Evaluation {i + 1} >>>>>>>>>>>>>>>>") + record = self.test_once() + if comm.is_main_process(): + if record[self.metric] > self.best_metric: + self.best_record = record + self.best_idx = i + self.best_metric = record[self.metric] + info = f"Current best record is Evaluation {i + 1}: " + for m in self.best_record.keys(): + info += f"{m}: {self.best_record[m]:.4f} " + logger.info(info) + + def test_once(self): + logger = get_root_logger() + batch_time = AverageMeter() + intersection_meter = AverageMeter() + target_meter = AverageMeter() + record = {} + self.model.eval() + + for idx, data_dict in enumerate(self.test_loader): + end = time.time() + data_dict = data_dict[0] # current assume batch size is 1 + voting_list = data_dict.pop("voting_list") + category = data_dict.pop("category") + data_name = data_dict.pop("name") + # pred = torch.zeros([1, self.cfg.data.num_classes]).cuda() + # for i in range(len(voting_list)): + # input_dict = voting_list[i] + # for key in input_dict.keys(): + # if isinstance(input_dict[key], torch.Tensor): + # input_dict[key] = input_dict[key].cuda(non_blocking=True) + # with torch.no_grad(): + # pred += F.softmax(self.model(input_dict)["cls_logits"], -1) + input_dict = collate_fn(voting_list) + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + with torch.no_grad(): + pred = F.softmax(self.model(input_dict)["cls_logits"], -1).sum( + 0, keepdim=True + ) + pred = pred.max(1)[1].cpu().numpy() + intersection, union, target = intersection_and_union( + pred, category, self.cfg.data.num_classes, self.cfg.data.ignore_index + ) + intersection_meter.update(intersection) + target_meter.update(target) + record[data_name] = dict(intersection=intersection, target=target) + acc = sum(intersection) / (sum(target) + 1e-10) + m_acc = np.mean(intersection_meter.sum / (target_meter.sum + 1e-10)) + batch_time.update(time.time() - end) + logger.info( + "Test: {} [{}/{}] " + "Batch {batch_time.val:.3f} ({batch_time.avg:.3f}) " + "Accuracy {acc:.4f} ({m_acc:.4f}) ".format( + data_name, + idx + 1, + len(self.test_loader), + batch_time=batch_time, + acc=acc, + m_acc=m_acc, + ) + ) + + logger.info("Syncing ...") + comm.synchronize() + record_sync = comm.gather(record, dst=0) + + if comm.is_main_process(): + record = {} + for _ in range(len(record_sync)): + r = record_sync.pop() + record.update(r) + del r + intersection = np.sum( + [meters["intersection"] for _, meters in record.items()], axis=0 + ) + target = np.sum([meters["target"] for _, meters in record.items()], axis=0) + accuracy_class = intersection / (target + 1e-10) + mAcc = np.mean(accuracy_class) + allAcc = sum(intersection) / (sum(target) + 1e-10) + + logger.info("Val result: mAcc/allAcc {:.4f}/{:.4f}".format(mAcc, allAcc)) + for i in range(self.cfg.data.num_classes): + logger.info( + "Class_{idx} - {name} Result: iou/accuracy {accuracy:.4f}".format( + idx=i, + name=self.cfg.data.names[i], + accuracy=accuracy_class[i], + ) + ) + return dict(mAcc=mAcc, allAcc=allAcc) + + @staticmethod + def collate_fn(batch): + return batch + + +@TESTERS.register_module() +class ShapeNetPartSegTester(TesterBase): + def test(self): + logger = get_root_logger() + logger.info(">>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>") + + batch_time = AverageMeter() + + num_categories = len(self.test_loader.dataset.categories) + record = {} + self.model.eval() + + save_path = os.path.join(self.cfg.save_path, "result") + make_dirs(save_path) + + comm.synchronize() + for idx, data_dict in enumerate(self.test_loader): + start = time.time() + data_dict = data_dict[0] # current assume batch size is 1 + fragment_list = data_dict.pop("fragment_list") + segment = data_dict.pop("segment") + data_name = data_dict.pop("name") + pred_save_path = os.path.join(save_path, "{}_pred.npy".format(data_name)) + if os.path.isfile(pred_save_path): + logger.info( + "{}/{}: {}, loaded pred and label.".format( + idx + 1, len(self.test_loader), data_name + ) + ) + pred = np.load(pred_save_path) + pred = torch.from_numpy(pred).cuda() + if "origin_segment" in data_dict.keys(): + segment = data_dict["origin_segment"] + else: + pred = torch.zeros((segment.size, self.cfg.data.num_classes)).cuda() + for i in range(len(fragment_list)): + fragment_batch_size = 1 + s_i, e_i = i * fragment_batch_size, min( + (i + 1) * fragment_batch_size, len(fragment_list) + ) + input_dict = collate_fn(fragment_list[s_i:e_i]) + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + idx_part = input_dict["index"] + with torch.no_grad(): + pred_part = self.model(input_dict)["seg_logits"] # (n, k) + pred_part = F.softmax(pred_part, -1) + if self.cfg.empty_cache: + torch.cuda.empty_cache() + bs = 0 + for be in input_dict["offset"]: + pred[idx_part[bs:be], :] += pred_part[bs:be] + bs = be + logger.info( + "Test: {}/{}-{data_name}, Batch: {batch_idx}/{batch_num}".format( + idx + 1, + len(self.test_loader), + data_name=data_name, + batch_idx=i + 1, + batch_num=len(fragment_list), + ) + ) + pred = pred.max(1)[1].data + pred_np = pred.cpu().numpy() + if "origin_segment" in data_dict.keys(): + assert "inverse" in data_dict.keys() + pred = pred[data_dict["inverse"]] + segment = data_dict["origin_segment"] + np.save(pred_save_path, pred_np) + + category_index = fragment_list[0]["cls_token"] + category = self.test_loader.dataset.categories[category_index] + parts_idx = self.test_loader.dataset.category2part[category] + parts_iou = torch.zeros(len(parts_idx), device="cuda") + + segment = torch.from_numpy(segment).cuda() + for j, part in enumerate(parts_idx): + if (torch.sum(segment == part) == 0) and (torch.sum(pred == part) == 0): + parts_iou[j] = 1.0 + else: + i = (segment == part) & (pred == part) + u = (segment == part) | (pred == part) + parts_iou[j] = torch.sum(i) / (torch.sum(u) + 1e-10) + parts_iou_mean = parts_iou.mean().item() + record[data_name] = dict( + category_index=int(category_index), + parts_iou_mean=parts_iou_mean, + ) + + batch_time.update(time.time() - start) + logger.info( + "Test: {} [{}/{}] " + "Batch {batch_time.val:.3f} " + "({batch_time.avg:.3f}) " + "Mean IoU {iou:.3f}".format( + data_name, + idx + 1, + len(self.test_loader), + batch_time=batch_time, + iou=parts_iou_mean, + ) + ) + logger.info("Syncing ...") + comm.synchronize() + record_sync = comm.gather(record, dst=0) + + if comm.is_main_process(): + record = {} + for _ in range(len(record_sync)): + r = record_sync.pop() + record.update(r) + del r + + iou_category = np.zeros(num_categories, dtype=np.float64) + iou_count = np.zeros(num_categories, dtype=np.float64) + for _, meters in record.items(): + category_index = meters["category_index"] + iou_category[category_index] += meters["parts_iou_mean"] + iou_count[category_index] += 1 + + ins_mIoU = iou_category.sum() / (iou_count.sum() + 1e-10) + iou_per_cat = iou_category / (iou_count + 1e-10) + cat_mIoU = ( + np.mean(iou_per_cat[iou_count > 0]) + if np.any(iou_count > 0) + else float("nan") + ) + logger.info( + "Val result: ins.mIoU/cat.mIoU {:.4f}/{:.4f}.".format( + ins_mIoU, cat_mIoU + ) + ) + for i in range(num_categories): + if iou_count[i] == 0: + continue + logger.info( + "Class_{idx}-{name} Result: iou_cat/num_sample {iou_cat:.4f}/{iou_count:.4f}".format( + idx=i, + name=self.test_loader.dataset.categories[i], + iou_cat=iou_category[i] / (iou_count[i] + 1e-10), + iou_count=int(iou_count[i]), + ) + ) + logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + + @staticmethod + def collate_fn(batch): + return batch + + +@TESTERS.register_module() +class PartNetEPartSegTester(TesterBase): + def test(self): + logger = get_root_logger() + logger.info(">>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>") + + batch_time = AverageMeter() + + self.model.eval() + + save_path = os.path.join(self.cfg.save_path, "result") + make_dirs(save_path) + + record = {} + num_parts_total = sum(self.test_loader.dataset.num_parts) + local_total_iou_parts = np.zeros(num_parts_total, dtype=np.float64) + local_total_iou_count = np.zeros(num_parts_total, dtype=np.float64) + + comm.synchronize() + for idx, data_dict in enumerate(self.test_loader): + start = time.time() + data_dict = data_dict[0] # current assume batch size is 1 + fragment_list = data_dict.pop("fragment_list") + segment = data_dict.pop("segment") + data_name = data_dict.pop("name") + pred_save_path = os.path.join(save_path, "{}_pred.npy".format(data_name)) + cls_token = fragment_list[0]["cls_token"] + category = self.test_loader.dataset.categories[cls_token] + parts_idx = self.test_loader.dataset.category2part[category] + if os.path.isfile(pred_save_path): + logger.info( + "{}/{}: {}, loaded pred and label.".format( + idx + 1, len(self.test_loader), data_name + ) + ) + pred = np.load(pred_save_path) + pred = torch.from_numpy(pred).cuda() + if "origin_segment" in data_dict.keys(): + segment = data_dict["origin_segment"] + else: + pred = torch.zeros((segment.size, self.cfg.data.num_classes)).cuda() + for i in range(len(fragment_list)): + fragment_batch_size = 1 + s_i, e_i = i * fragment_batch_size, min( + (i + 1) * fragment_batch_size, len(fragment_list) + ) + input_dict = collate_fn(fragment_list[s_i:e_i]) + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + idx_part = input_dict["index"] + with torch.no_grad(): + pred_part = self.model(input_dict)["seg_logits"] # (n, k) + pred_part = F.softmax(pred_part, -1) + if self.cfg.empty_cache: + torch.cuda.empty_cache() + bs = 0 + for be in input_dict["offset"]: + pred[idx_part[bs:be], :] += pred_part[bs:be] + bs = be + logger.info( + "Test: {}/{}-{data_name}, Batch: {batch_idx}/{batch_num}".format( + idx + 1, + len(self.test_loader), + data_name=data_name, + batch_idx=i + 1, + batch_num=len(fragment_list), + ) + ) + pred = pred.max(1)[1].data + pred_np = pred.cpu().numpy() + if "origin_segment" in data_dict.keys(): + assert "inverse" in data_dict.keys() + pred = pred[data_dict["inverse"]] + segment = data_dict["origin_segment"] + np.save(pred_save_path, pred_np) + + segment = torch.from_numpy(segment).cuda() + sample_part_record = {} + for k, part_id in enumerate(parts_idx): + if k == 0: + continue + if (segment == part_id).sum() == 0: + continue + if (torch.sum(segment == part_id) == 0) and ( + torch.sum(pred == part_id) == 0 + ): + continue + else: + intersection = torch.sum((segment == part_id) & (pred == part_id)) + union = torch.sum((segment == part_id) | (pred == part_id)) + part_idx = int( + k + self.test_loader.dataset.num_part_offset[cls_token] + ) + sample_part_record[part_idx] = float( + (intersection / (union + 1e-10)).item() + ) + record[data_name] = dict(part_iou=sample_part_record) + for part_idx, part_iou in sample_part_record.items(): + local_total_iou_parts[int(part_idx)] += part_iou + local_total_iou_count[int(part_idx)] += 1 + + current_iou_count = local_total_iou_count[local_total_iou_count > 0] + current_iou_parts = local_total_iou_parts[local_total_iou_count > 0] + current_iou = current_iou_parts / current_iou_count + current_iou_mean = ( + current_iou.mean() if current_iou.shape[0] > 0 else float("nan") + ) + batch_time.update(time.time() - start) + logger.info( + "Test: {} [{}/{}] " + "Batch {batch_time.val:.3f} " + "({batch_time.avg:.3f}) " + "Mean IoU {iou:.3f}".format( + data_name, + idx + 1, + len(self.test_loader), + batch_time=batch_time, + iou=current_iou_mean, + ) + ) + logger.info("Syncing ...") + comm.synchronize() + record_sync = comm.gather(record, dst=0) + + if comm.is_main_process(): + record = {} + for _ in range(len(record_sync)): + r = record_sync.pop() + record.update(r) + del r + + total_iou_parts = np.zeros(num_parts_total, dtype=np.float64) + total_iou_count = np.zeros(num_parts_total, dtype=np.float64) + for _, meters in record.items(): + for part_idx, part_iou in meters["part_iou"].items(): + total_iou_parts[int(part_idx)] += part_iou + total_iou_count[int(part_idx)] += 1 + + current_iou_count = total_iou_count[total_iou_count > 0] + current_iou_parts = total_iou_parts[total_iou_count > 0] + # part-wise mIoU: average of all sample mIoUs + part_mIoU = ( + (current_iou_parts / (current_iou_count + 1e-10)).mean() + if current_iou_count.shape[0] > 0 + else float("nan") + ) + logger.info("Val result: part mIoU {:.4f}.".format(part_mIoU)) + for i in range(sum(self.test_loader.dataset.num_parts)): + part_name = self.test_loader.dataset.parts[i] + if total_iou_count[i] == 0: + continue + logger.info( + "Class_{idx}-{name} Result: iou_part/num_sample {iou_part:.4f}/{iou_count:.4f}".format( + idx=i, + name=part_name, + iou_part=total_iou_parts[i] / (total_iou_count[i] + 1e-10), + iou_count=int(total_iou_count[i]), + ) + ) + logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + + @staticmethod + def collate_fn(batch): + return batch + + +@TESTERS.register_module() +class InsSegTester(TesterBase): + def __init__( + self, + segment_ignore_index, + instance_ignore_index, + **kwargs, + ): + super().__init__(**kwargs) + self.segment_ignore_index = segment_ignore_index + self.instance_ignore_index = instance_ignore_index + self.valid_class_names = [ + self.cfg.data.names[i] + for i in range(self.cfg.data.num_classes) + if i not in self.segment_ignore_index + ] + self.overlaps = np.append(np.arange(0.5, 0.95, 0.05), 0.25) + self.min_region_sizes = 100 + self.distance_threshes = float("inf") + self.distance_confs = -float("inf") + + def test(self): + assert self.test_loader.batch_size == 1 + logger = get_root_logger() + logger.info(">>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>") + + batch_time = AverageMeter() + + self.model.eval() + scenes = {} + + for idx, data_dict in enumerate(self.test_loader): + start = time.time() + data_name = data_dict.pop("name") + for key in data_dict.keys(): + if isinstance(data_dict[key], torch.Tensor): + data_dict[key] = data_dict[key].cuda(non_blocking=True) + with torch.no_grad(): + output_dict = self.model(data_dict) + segment = data_dict["segment"] + instance = data_dict["instance"] + + if "origin_coord" in data_dict.keys(): + reverse, _ = pointops.knn_query( + 1, + data_dict["coord"].float(), + data_dict["offset"].int(), + data_dict["origin_coord"].float(), + data_dict["origin_offset"].int(), + ) + reverse = reverse.cpu().flatten().long() + output_dict["pred_masks"] = output_dict["pred_masks"][:, reverse] + segment = data_dict["origin_segment"] + instance = data_dict["origin_instance"] + + gt_instances, pred_instance = self.associate_instances( + output_dict, segment, instance + ) + + scenes[data_name] = dict(gt=gt_instances, pred=pred_instance) + batch_time.update(time.time() - start) + logger.info( + "Test: {} [{}/{}] " + "Batch {batch_time.val:.3f} ({batch_time.avg:.3f}) ".format( + data_name, + idx + 1, + len(self.test_loader), + batch_time=batch_time, + ) + ) + if self.cfg.data.test.type == "ScanNetPPDataset": + self.write_scannetpp_results( + output_dict["pred_scores"], + output_dict["pred_masks"], + output_dict["pred_classes"], + data_name, + ) + + comm.synchronize() + scenes_sync = comm.gather(scenes, dst=0) + + if comm.is_main_process(): + scenes = {} + for _ in range(len(scenes_sync)): + r = scenes_sync.pop() + scenes.update(r) + del r + scenes = list(scenes.values()) + ap_scores = self.evaluate_matches(scenes) + all_ap = ap_scores["all_ap"] + all_ap_50 = ap_scores["all_ap_50%"] + all_ap_25 = ap_scores["all_ap_25%"] + logger.info( + "Val result: mAP/AP50/AP25 {:.4f}/{:.4f}/{:.4f}.".format( + all_ap, all_ap_50, all_ap_25 + ) + ) + for i, label_name in enumerate(self.valid_class_names): + ap = ap_scores["classes"][label_name]["ap"] + ap_50 = ap_scores["classes"][label_name]["ap50%"] + ap_25 = ap_scores["classes"][label_name]["ap25%"] + logger.info( + "Class_{idx}-{name} Result: AP/AP50/AP25 {AP:.4f}/{AP50:.4f}/{AP25:.4f}".format( + idx=i, name=label_name, AP=ap, AP50=ap_50, AP25=ap_25 + ) + ) + logger.info("<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<") + + def write_scannetpp_results( + self, + pred_scores, + pred_masks, + pred_classes, + data_name, + ): + pred_scores[pred_scores < 0] = 0 + pred_scores[pred_scores >= 0] = 1 + + save_dir = os.path.join(self.cfg.save_path, "result", "submit") + mask_dir = os.path.join(save_dir, "predicted_masks") + make_dirs(mask_dir) + + result_path = os.path.join(save_dir, f"{data_name}.txt") + result_file = open(result_path, "w") + for i, (score, mask, cls) in enumerate( + zip( + pred_scores.cpu().numpy(), + pred_masks.cpu().numpy(), + pred_classes.cpu().numpy(), + ) + ): + mask = mask.astype(np.uint8) + length = mask.shape[0] + mask = np.concatenate([[0], mask, [0]]) + runs = np.where(mask[1:] != mask[:-1])[0] + 1 + runs[1::2] -= runs[::2] + counts = " ".join(str(x) for x in runs) + rle = dict(length=length, counts=counts) + + mask_path = os.path.join(mask_dir, f"{data_name}_{i:03d}.json") + relative_path = os.path.join("predicted_masks", f"{data_name}_{i:03d}.json") + with open(mask_path, "w") as mask_file: + json.dump(rle, mask_file, indent=2) + result_file.write(f"{relative_path} {cls} {score:.3f}\n") + result_file.close() + + def associate_instances(self, pred, segment, instance): + segment = segment.cpu().numpy() + instance = instance.cpu().numpy() + void_mask = np.in1d(segment, self.segment_ignore_index) + + assert ( + pred["pred_classes"].shape[0] + == pred["pred_scores"].shape[0] + == pred["pred_masks"].shape[0] + ) + assert pred["pred_masks"].shape[1] == segment.shape[0] == instance.shape[0] + # get gt instances + gt_instances = dict() + for i in range(self.cfg.data.num_classes): + if i not in self.segment_ignore_index: + gt_instances[self.cfg.data.names[i]] = [] + instance_ids, idx, counts = np.unique( + instance, return_index=True, return_counts=True + ) + segment_ids = segment[idx] + for i in range(len(instance_ids)): + if instance_ids[i] == self.instance_ignore_index: + continue + if segment_ids[i] in self.segment_ignore_index: + continue + gt_inst = dict() + gt_inst["instance_id"] = instance_ids[i] + gt_inst["segment_id"] = segment_ids[i] + gt_inst["dist_conf"] = 0.0 + gt_inst["med_dist"] = -1.0 + gt_inst["vert_count"] = counts[i] + gt_inst["matched_pred"] = [] + gt_instances[self.cfg.data.names[segment_ids[i]]].append(gt_inst) + + # get pred instances and associate with gt + pred_instances = dict() + for i in range(self.cfg.data.num_classes): + if i not in self.segment_ignore_index: + pred_instances[self.cfg.data.names[i]] = [] + instance_id = 0 + for i in range(len(pred["pred_classes"])): + if pred["pred_classes"][i] in self.segment_ignore_index: + continue + pred_inst = dict() + pred_inst["uuid"] = uuid4() + pred_inst["instance_id"] = instance_id + pred_inst["segment_id"] = pred["pred_classes"][i] + pred_inst["confidence"] = pred["pred_scores"][i] + pred_inst["mask"] = np.not_equal(pred["pred_masks"][i], 0) + pred_inst["vert_count"] = np.count_nonzero(pred_inst["mask"]) + pred_inst["void_intersection"] = np.count_nonzero( + np.logical_and(void_mask, pred_inst["mask"]) + ) + if pred_inst["vert_count"] < self.min_region_sizes: + continue # skip if empty + segment_name = self.cfg.data.names[pred_inst["segment_id"]] + matched_gt = [] + for gt_idx, gt_inst in enumerate(gt_instances[segment_name]): + intersection = np.count_nonzero( + np.logical_and( + instance == gt_inst["instance_id"], pred_inst["mask"] + ) + ) + if intersection > 0: + gt_inst_ = gt_inst.copy() + pred_inst_ = pred_inst.copy() + gt_inst_["intersection"] = intersection + pred_inst_["intersection"] = intersection + matched_gt.append(gt_inst_) + gt_inst["matched_pred"].append(pred_inst_) + pred_inst["matched_gt"] = matched_gt + pred_instances[segment_name].append(pred_inst) + instance_id += 1 + return gt_instances, pred_instances + + def evaluate_matches(self, scenes): + overlaps = self.overlaps + min_region_sizes = [self.min_region_sizes] + dist_threshes = [self.distance_threshes] + dist_confs = [self.distance_confs] + + # results: class x overlap + ap_table = np.zeros( + (len(dist_threshes), len(self.valid_class_names), len(overlaps)), float + ) + for di, (min_region_size, distance_thresh, distance_conf) in enumerate( + zip(min_region_sizes, dist_threshes, dist_confs) + ): + for oi, overlap_th in enumerate(overlaps): + pred_visited = {} + for scene in scenes: + for _ in scene["pred"]: + for label_name in self.valid_class_names: + for p in scene["pred"][label_name]: + if "uuid" in p: + pred_visited[p["uuid"]] = False + for li, label_name in enumerate(self.valid_class_names): + y_true = np.empty(0) + y_score = np.empty(0) + hard_false_negatives = 0 + has_gt = False + has_pred = False + for scene in scenes: + pred_instances = scene["pred"][label_name] + gt_instances = scene["gt"][label_name] + # filter groups in ground truth + gt_instances = [ + gt + for gt in gt_instances + if gt["vert_count"] >= min_region_size + and gt["med_dist"] <= distance_thresh + and gt["dist_conf"] >= distance_conf + ] + if gt_instances: + has_gt = True + if pred_instances: + has_pred = True + + cur_true = np.ones(len(gt_instances)) + cur_score = np.ones(len(gt_instances)) * (-float("inf")) + cur_match = np.zeros(len(gt_instances), dtype=bool) + # collect matches + for gti, gt in enumerate(gt_instances): + found_match = False + for pred in gt["matched_pred"]: + # greedy assignments + if pred_visited[pred["uuid"]]: + continue + overlap = float(pred["intersection"]) / ( + gt["vert_count"] + + pred["vert_count"] + - pred["intersection"] + ) + if overlap > overlap_th: + confidence = pred["confidence"] + # if already have a prediction for this gt, + # the prediction with the lower score is automatically a false positive + if cur_match[gti]: + max_score = max(cur_score[gti], confidence) + min_score = min(cur_score[gti], confidence) + cur_score[gti] = max_score + # append false positive + cur_true = np.append(cur_true, 0) + cur_score = np.append(cur_score, min_score) + cur_match = np.append(cur_match, True) + # otherwise set score + else: + found_match = True + cur_match[gti] = True + cur_score[gti] = confidence + pred_visited[pred["uuid"]] = True + if not found_match: + hard_false_negatives += 1 + # remove non-matched ground truth instances + cur_true = cur_true[cur_match] + cur_score = cur_score[cur_match] + + # collect non-matched predictions as false positive + for pred in pred_instances: + found_gt = False + for gt in pred["matched_gt"]: + overlap = float(gt["intersection"]) / ( + gt["vert_count"] + + pred["vert_count"] + - gt["intersection"] + ) + if overlap > overlap_th: + found_gt = True + break + if not found_gt: + num_ignore = pred["void_intersection"] + for gt in pred["matched_gt"]: + if gt["segment_id"] in self.segment_ignore_index: + num_ignore += gt["intersection"] + # small ground truth instances + if ( + gt["vert_count"] < min_region_size + or gt["med_dist"] > distance_thresh + or gt["dist_conf"] < distance_conf + ): + num_ignore += gt["intersection"] + proportion_ignore = ( + float(num_ignore) / pred["vert_count"] + ) + # if not ignored append false positive + if proportion_ignore <= overlap_th: + cur_true = np.append(cur_true, 0) + confidence = pred["confidence"] + cur_score = np.append(cur_score, confidence) + + # append to overall results + y_true = np.append(y_true, cur_true) + y_score = np.append(y_score, cur_score) + + # compute average precision + if has_gt and has_pred: + # compute precision recall curve first + + # sorting and cumsum + score_arg_sort = np.argsort(y_score) + y_score_sorted = y_score[score_arg_sort] + y_true_sorted = y_true[score_arg_sort] + y_true_sorted_cumsum = np.cumsum(y_true_sorted) + + # unique thresholds + thresholds, unique_indices = np.unique( + y_score_sorted, return_index=True + ) + num_prec_recall = len(unique_indices) + 1 + + # prepare precision recall + num_examples = len(y_score_sorted) + # https://github.com/ScanNet/ScanNet/pull/26 + # all predictions are non-matched but also all of them are ignored and not counted as FP + # y_true_sorted_cumsum is empty + # num_true_examples = y_true_sorted_cumsum[-1] + num_true_examples = ( + y_true_sorted_cumsum[-1] + if len(y_true_sorted_cumsum) > 0 + else 0 + ) + precision = np.zeros(num_prec_recall) + recall = np.zeros(num_prec_recall) + + # deal with the first point + y_true_sorted_cumsum = np.append(y_true_sorted_cumsum, 0) + # deal with remaining + for idx_res, idx_scores in enumerate(unique_indices): + cumsum = y_true_sorted_cumsum[idx_scores - 1] + tp = num_true_examples - cumsum + fp = num_examples - idx_scores - tp + fn = cumsum + hard_false_negatives + p = float(tp) / (tp + fp) + r = float(tp) / (tp + fn) + precision[idx_res] = p + recall[idx_res] = r + + # first point in curve is artificial + precision[-1] = 1.0 + recall[-1] = 0.0 + + # compute average of precision-recall curve + recall_for_conv = np.copy(recall) + recall_for_conv = np.append(recall_for_conv[0], recall_for_conv) + recall_for_conv = np.append(recall_for_conv, 0.0) + + stepWidths = np.convolve( + recall_for_conv, [-0.5, 0, 0.5], "valid" + ) + # integrate is now simply a dot product + ap_current = np.dot(precision, stepWidths) + + elif has_gt: + ap_current = 0.0 + else: + ap_current = float("nan") + ap_table[di, li, oi] = ap_current + d_inf = 0 + o50 = np.where(np.isclose(self.overlaps, 0.5)) + o25 = np.where(np.isclose(self.overlaps, 0.25)) + oAllBut25 = np.where(np.logical_not(np.isclose(self.overlaps, 0.25))) + ap_scores = dict() + ap_scores["all_ap"] = np.nanmean(ap_table[d_inf, :, oAllBut25]) + ap_scores["all_ap_50%"] = np.nanmean(ap_table[d_inf, :, o50]) + ap_scores["all_ap_25%"] = np.nanmean(ap_table[d_inf, :, o25]) + ap_scores["classes"] = {} + for li, label_name in enumerate(self.valid_class_names): + ap_scores["classes"][label_name] = {} + ap_scores["classes"][label_name]["ap"] = np.average( + ap_table[d_inf, li, oAllBut25] + ) + ap_scores["classes"][label_name]["ap50%"] = np.average( + ap_table[d_inf, li, o50] + ) + ap_scores["classes"][label_name]["ap25%"] = np.average( + ap_table[d_inf, li, o25] + ) + return ap_scores + + @staticmethod + def collate_fn(batch): + # Restrict to bs 1 + return batch[0] diff --git a/pointcept/engines/train.py b/pointcept/engines/train.py new file mode 100644 index 0000000000000000000000000000000000000000..10c1e099527e61f521021e1f917280d54407b1a1 --- /dev/null +++ b/pointcept/engines/train.py @@ -0,0 +1,416 @@ +""" +Trainer + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import sys +import weakref +import wandb +import torch +import torch.nn as nn +import torch.utils.data +from packaging import version +from functools import partial +from pathlib import Path +import itertools + +if sys.version_info >= (3, 10): + from collections.abc import Iterator +else: + from collections import Iterator +from tensorboardX import SummaryWriter + +from .defaults import create_ddp_model, worker_init_fn +from .hooks import HookBase, build_hooks +import pointcept.utils.comm as comm +from pointcept.datasets import build_dataset, point_collate_fn, collate_fn +from pointcept.models import build_model +from pointcept.utils.logger import get_root_logger +from pointcept.utils.optimizer import build_optimizer +from pointcept.utils.scheduler import build_scheduler +from pointcept.utils.events import EventStorage, ExceptionWriter +from pointcept.utils.registry import Registry +from pointcept.datasets.dataloader import DistributedImbalancedSampler + +TRAINERS = Registry("trainers") +AMP_DTYPE = dict( + float16=torch.float16, + bfloat16=torch.bfloat16, +) + + +class TrainerBase: + def __init__(self) -> None: + self.hooks = [] + self.model = None + self.epoch = 0 + self.start_epoch = 0 + self.max_epoch = 0 + self.max_iter = 0 + self.comm_info = dict() + self.data_iterator: Iterator = enumerate([]) + self.storage: EventStorage + self.writer: SummaryWriter + + def register_hooks(self, hooks) -> None: + hooks = build_hooks(hooks) + for h in hooks: + assert isinstance(h, HookBase) + # To avoid circular reference, hooks and trainer cannot own each other. + # This normally does not matter, but will cause memory leak if the + # involved objects contain __del__: + # See http://engineering.hearsaysocial.com/2013/06/16/circular-references-in-python/ + h.trainer = weakref.proxy(self) + self.hooks.extend(hooks) + + def train(self): + with EventStorage() as self.storage: + # => before train + self.before_train() + for self.epoch in range(self.start_epoch, self.max_epoch): + # => before epoch + self.before_epoch() + # => run_epoch + for ( + self.comm_info["iter"], + self.comm_info["input_dict"], + ) in self.data_iterator: + # => before_step + self.before_step() + # => run_step + self.run_step() + # => after_step + self.after_step() + # => after epoch + self.after_epoch() + # => after train + self.after_train() + + def before_train(self): + for h in self.hooks: + h.before_train() + + def before_epoch(self): + for h in self.hooks: + h.before_epoch() + + def before_step(self): + for h in self.hooks: + h.before_step() + + def run_step(self): + raise NotImplementedError + + def after_step(self): + for h in self.hooks: + h.after_step() + + def after_epoch(self): + for h in self.hooks: + h.after_epoch() + self.storage.reset_histories() + + def after_train(self): + # Sync GPU before running train hooks + comm.synchronize() + for h in self.hooks: + h.after_train() + if comm.is_main_process(): + self.writer.close() + + +@TRAINERS.register_module("DefaultTrainer") +class Trainer(TrainerBase): + def __init__(self, cfg): + super(Trainer, self).__init__() + self.epoch = 0 + self.start_epoch = 0 + self.max_epoch = cfg.eval_epoch + self.best_metric_value = -torch.inf + self.logger = get_root_logger( + log_file=os.path.join(cfg.save_path, "train.log"), + file_mode="a" if cfg.resume else "w", + ) + self.logger.info("=> Loading config ...") + self.cfg = cfg + self.logger.info(f"Save path: {cfg.save_path}") + self.logger.info(f"Config:\n{cfg.pretty_text}") + self.logger.info("=> Building model ...") + self.model = self.build_model() + self.logger.info("=> Building writer ...") + self.writer = self.build_writer() + self.logger.info("=> Building train dataset & dataloader ...") + self.train_loader = self.build_train_loader() + self.logger.info("=> Building val dataset & dataloader ...") + self.val_loader = self.build_val_loader() + self.logger.info("=> Building optimize, scheduler, scaler(amp) ...") + self.optimizer = self.build_optimizer() + self.scheduler = self.build_scheduler() + self.scaler = self.build_scaler() + self.logger.info("=> Building hooks ...") + self.register_hooks(self.cfg.hooks) + self._gradient_accumulation_counter = 0 + + def train(self): + with EventStorage() as self.storage, ExceptionWriter(): + # => before train + self.before_train() + self.logger.info(">>>>>>>>>>>>>>>> Start Training >>>>>>>>>>>>>>>>") + for self.epoch in range(self.start_epoch, self.max_epoch): + # => before epoch + if comm.get_world_size() > 1: + self.train_loader.sampler.set_epoch(self.epoch) + self.model.train() + self.data_iterator = enumerate(self.train_loader) + self.before_epoch() + # => run_epoch + for ( + self.comm_info["iter"], + self.comm_info["input_dict"], + ) in self.data_iterator: + # => before_step + self.before_step() + # => run_step + self.run_step() + # => after_step + self.after_step() + # => after epoch + self.after_epoch() + # => after train + self.after_train() + + def run_step(self): + if version.parse(torch.__version__) >= version.parse("2.4"): + auto_cast = partial(torch.amp.autocast, device_type="cuda") + else: + # deprecated warning + auto_cast = torch.cuda.amp.autocast + + input_dict = self.comm_info["input_dict"] + for key in input_dict.keys(): + if isinstance(input_dict[key], torch.Tensor): + input_dict[key] = input_dict[key].cuda(non_blocking=True) + + # Only clear gradients on first accumulation step + if self._gradient_accumulation_counter == 0: + self.optimizer.zero_grad() + + # Forward pass + with auto_cast( + enabled=self.cfg.enable_amp, dtype=AMP_DTYPE[self.cfg.amp_dtype] + ): + output_dict = self.model(input_dict) + loss = ( + output_dict["loss"] / self.cfg.gradient_accumulation_steps + ) # scale loss + + # Backward pass + if self.cfg.enable_amp: + self.scaler.scale(loss).backward() + else: + loss.backward() + self._gradient_accumulation_counter += 1 + + # Perform optimizer step only when enough gradients have accumulated + if self._gradient_accumulation_counter >= self.cfg.gradient_accumulation_steps: + if self.cfg.enable_amp: + self.scaler.unscale_(self.optimizer) + if self.cfg.clip_grad is not None: + torch.nn.utils.clip_grad_norm_( + self.model.parameters(), self.cfg.clip_grad + ) + self.scaler.step(self.optimizer) + + # When enable amp, optimizer.step call are skipped if the loss scaling factor is too large. + # Fix torch warning scheduler step before optimizer step. + scale = self.scaler.get_scale() + self.scaler.update() + if scale <= self.scaler.get_scale(): + self.scheduler.step() + else: + if self.cfg.clip_grad is not None: + torch.nn.utils.clip_grad_norm_( + self.model.parameters(), self.cfg.clip_grad + ) + self.optimizer.step() + self.scheduler.step() + + # Reset grad accumulation counter + self._gradient_accumulation_counter = 0 + + if self.cfg.empty_cache: + torch.cuda.empty_cache() + self.comm_info["model_output_dict"] = output_dict + + def after_epoch(self): + for h in self.hooks: + h.after_epoch() + self.storage.reset_histories() + if self.cfg.empty_cache_per_epoch: + torch.cuda.empty_cache() + + def build_model(self): + model = build_model(self.cfg.model) + if self.cfg.sync_bn: + model = nn.SyncBatchNorm.convert_sync_batchnorm(model) + n_parameters = sum(p.numel() for p in model.parameters() if p.requires_grad) + # logger.info(f"Model: \n{self.model}") + self.logger.info(f"Num params: {n_parameters}") + model = create_ddp_model( + model.cuda(), + broadcast_buffers=False, + find_unused_parameters=self.cfg.find_unused_parameters, + ) + return model + + def build_writer(self): + writer = SummaryWriter(self.cfg.save_path) if comm.is_main_process() else None + self.logger.info(f"Tensorboard writer logging dir: {self.cfg.save_path}") + if self.cfg.enable_wandb and comm.is_main_process(): + tag, name = Path(self.cfg.save_path).parts[-2:] + wandb.init( + project=self.cfg.wandb_project, + name=f"{tag}/{name}", + tags=[tag], + dir=self.cfg.save_path, + settings=wandb.Settings(api_key=self.cfg.wandb_key), + config=self.cfg, + ) + return writer + + def build_train_loader(self): + train_data = build_dataset(self.cfg.data.train) + + if comm.get_world_size() > 1: + train_sampler = torch.utils.data.distributed.DistributedSampler(train_data) + else: + train_sampler = None + + init_fn = ( + partial( + worker_init_fn, + num_workers=self.cfg.num_worker_per_gpu, + rank=comm.get_rank(), + seed=self.cfg.seed, + ) + if self.cfg.seed is not None + else None + ) + + train_loader = torch.utils.data.DataLoader( + train_data, + batch_size=self.cfg.batch_size_per_gpu, + shuffle=(train_sampler is None), + num_workers=self.cfg.num_worker_per_gpu, + sampler=train_sampler, + collate_fn=partial(point_collate_fn, mix_prob=self.cfg.mix_prob), + pin_memory=True, + worker_init_fn=init_fn, + drop_last=len(train_data) > self.cfg.batch_size, + persistent_workers=True, + ) + return train_loader + + def build_val_loader(self): + val_loader = None + if self.cfg.evaluate: + val_data = build_dataset(self.cfg.data.val) + if comm.get_world_size() > 1: + val_sampler = torch.utils.data.distributed.DistributedSampler(val_data) + else: + val_sampler = None + val_loader = torch.utils.data.DataLoader( + val_data, + batch_size=self.cfg.batch_size_val_per_gpu, + shuffle=False, + num_workers=self.cfg.num_worker_per_gpu, + pin_memory=True, + sampler=val_sampler, + collate_fn=collate_fn, + ) + return val_loader + + def build_optimizer(self): + return build_optimizer(self.cfg.optimizer, self.model, self.cfg.param_dicts) + + def build_scheduler(self): + assert hasattr(self, "optimizer") + assert hasattr(self, "train_loader") + self.cfg.scheduler.total_steps = ( + len(self.train_loader) + * self.cfg.eval_epoch + // self.cfg.gradient_accumulation_steps + ) + return build_scheduler(self.cfg.scheduler, self.optimizer) + + def build_scaler(self): + if version.parse(torch.__version__) >= version.parse("2.4"): + grad_scaler = partial(torch.amp.GradScaler, device="cuda") + else: + # deprecated warning + grad_scaler = torch.cuda.amp.GradScaler + scaler = grad_scaler() if self.cfg.enable_amp else None + return scaler + + +@TRAINERS.register_module("PartialSampledTrainer") +class PartialSampledTrainer(Trainer): + def build_train_loader(self): + train_data = build_dataset(self.cfg.data.train) + sampled_dataset_index = self.cfg.data.sampled_dataset_index + sampled_dataset_limit = self.cfg.data.sampled_dataset_limit + + train_sampler = DistributedImbalancedSampler( + dataset=train_data, + sampled_dataset_index=sampled_dataset_index, + sampled_dataset_limit=sampled_dataset_limit, + num_replicas=comm.get_world_size(), + rank=comm.get_rank(), + shuffle=True, + seed=self.cfg.seed if self.cfg.seed is not None else 0, + ) + + init_fn = ( + partial( + worker_init_fn, + num_workers=self.cfg.num_worker_per_gpu, + rank=comm.get_rank(), + seed=self.cfg.seed, + ) + if self.cfg.seed is not None + else None + ) + + train_loader = torch.utils.data.DataLoader( + train_data, + batch_size=self.cfg.batch_size_per_gpu, + shuffle=False, + num_workers=self.cfg.num_worker_per_gpu, + sampler=train_sampler, + collate_fn=partial(point_collate_fn, mix_prob=self.cfg.mix_prob), + pin_memory=True, + worker_init_fn=init_fn, + drop_last=len(train_data) > self.cfg.batch_size, + persistent_workers=True, + ) + return train_loader + + +@TRAINERS.register_module("MultiDatasetTrainer") +class MultiDatasetTrainer(Trainer): + def build_train_loader(self): + from pointcept.datasets import MultiDatasetDataloader + + train_data = build_dataset(self.cfg.data.train) + train_loader = MultiDatasetDataloader( + train_data, + self.cfg.batch_size_per_gpu, + self.cfg.num_worker_per_gpu, + self.cfg.mix_prob, + self.cfg.seed, + ) + self.comm_info["iter_per_epoch"] = len(train_loader) + return train_loader diff --git a/pointcept/models/__init__.py b/pointcept/models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a4e401b6bf3a7b3af028010c1306f9f1bac39d70 --- /dev/null +++ b/pointcept/models/__init__.py @@ -0,0 +1,10 @@ +# Trimmed for inference-only use: this delivery only ever builds a PT-v3m1 +# backbone wrapped in DefaultSegmentorV2, so the other backbones/heads +# (which pull in extra heavy/compiled deps like torchsparse, pointgroup_ops, +# pointops, transformers) aren't imported. Add back whichever of these you +# need if you point `model.backbone.type`/`model.type` at something else. +from .builder import build_model +from .default import DefaultSegmentor, DefaultSegmentorV2, DefaultClassifier + +# Backbone actually used by the bundled checkpoint (PT-v3m1) +from .point_transformer_v3 import * diff --git a/pointcept/models/builder.py b/pointcept/models/builder.py new file mode 100644 index 0000000000000000000000000000000000000000..8c723d7fc3967485ff21a212fd8f7148c23377c2 --- /dev/null +++ b/pointcept/models/builder.py @@ -0,0 +1,17 @@ +""" +Model Builder + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import copy +from pointcept.utils.registry import Registry + +MODELS = Registry("models") +MODULES = Registry("modules") + + +def build_model(cfg): + """Build models.""" + return MODELS.build(copy.deepcopy(cfg)) diff --git a/pointcept/models/concerto/README.md b/pointcept/models/concerto/README.md new file mode 100644 index 0000000000000000000000000000000000000000..206b11385b6d31f380174eade85a02ea5f0f2e20 --- /dev/null +++ b/pointcept/models/concerto/README.md @@ -0,0 +1,420 @@ +# Concerto +This repo is the official training codebase of the paper **_Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations_**. Concerto steps farther from [Sonata](https://github.com/facebookresearch/sonata), leveraging the synergy of 2D images and 3D point clouds and emerging superior representations. + +*We recommend beginning with our inference [demo](https://github.com/Pointcept/Concerto), and it is also a tiny library for people who want to integrate pre-trained Concerto (as well as Sonata) into their project. This section is designed for users interested in reproducing our pre-training.* + + +
+teaser +
+ +## Quick start + +#### 1. Installation +Follow the instructions [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#installation). + +Additional Package: +```bash +pip install transformers==4.50.3 +pip install peft +``` + +#### 2. Dataset +Besides the same point cloud datasets as Sonata, Concerto includes the RE10K video dataset in training data. Each dataset has image assets and their correspondence to point clouds. + +The whole data is organized as follows: +``` +└── Default Dataset + ├── images + │ ├── train + │ │ ├── color + │ │ ├── correspondence + │ │ ├── depth (optional) + │ │ ├── intrinsic + │ │ └── pose + │ ├── val + │ └── test + ├── splits + │ ├── train.json + │ ├── val.json + │ └── test.json + ├── train + ├── val + └── test +``` +The `color`, `correspondence`, `depth`, `intrinsic`, and `pose` folders contain RGB images, point cloud correspondences, depth maps, camera intrinsics, and camera poses, respectively. The `train.json`, `val.json`, and `test.json` files in the _splits_ folder provide the indices for the training, validation, and test splits. + +For the _intrinsic_ folder, if only a single file is presented, it indicates that all images share the same intrinsic parameters. If multiple files exist, each corresponds to a different image. In the case of the HM3D dataset, where camera parameters are simulated, all images share identical intrinsics across the entire dataset. + +##### 2.1 [Option A] Preprocess Dataset from Scratch +For those who wish to do data preprocessing locally, please follow the instructions below: +- If you do not have the point cloud datasets for Sonata, you can process the next step without any modification. If you have already downloaded the processed Sonata datasets below, you can remove "-c" in the next step to only output image assets. + - ScanNet v2 - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#scannet-v2) ] + - ScanNet++ v2 - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#scannet) ] + - S3DIS - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#s3dis) ] + - ArkitScenes - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#arkitscenes) ] + - HM3D - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#habitat---matterport-3d-hm3d) ] + - Structured3D - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#structured3d) ] + +- Now you can follow the instructions below to processe the image assets. + - ScanNet v2 + - Download the [ScanNet](http://www.scan-net.org/) v2 dataset. + - Run preprocessing code for raw ScanNet as follows: + ```bash + # RAW_SCANNET_DIR: the directory of downloaded ScanNet v2 raw dataset. PROCESSED_SCANNET_DIR: the directory of the processed ScanNet dataset (output dir). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.sh -d ${RAW_SCANNET_DIR} -o ${PROCESSED_SCANNET_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/scannet/splits.py --dataset_root ${PROCESSED_SCANNET_DIR} + ``` + - ScanNet++ + - Download the [ScanNet++](https://kaldir.vc.in.tum.de/scannetpp/) dataset. + - Run preprocessing code for raw ScanNet++ as follows: + ```bash + # For Concerto, we use the unsplitted point cloud data. There is no need to split the large scene because we crop the scenes in code according to the selected camera views. + # RAW_SCANNETPP_DIR: the directory of downloaded ScanNet++ raw dataset. PROCESSED_SCANNETPP_DIR: the directory of the processed ScanNet++ dataset (output dir). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.sh -d ${RAW_SCANNETPP_DIR} -o ${PROCESSED_SCANNETPP_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/scannetpp/splits.py --dataset_root ${PROCESSED_SCANNETPP_DIR} + ``` + - S3DIS + - Download S3DIS data by filling this [Google form](https://docs.google.com/forms/d/e/1FAIpQLScDimvNMCGhy_rmBA2gHfDu3naktRm6A8BPwAWWDv-Uhm6Shw/viewform?c=0&w=1). Download the `Stanford3dDataset_v1.2.zip` file and unzip it. + - Fix error in `Area_5/office_19/Annotations/ceiling` Line 323474 (103.0�0000 => 103.000000). + - (Optional) Download Full 2D-3D S3DIS dataset (no XYZ) from [here](https://github.com/alexsax/2D-3D-Semantics) for parsing normal. + - Run preprocessing code for S3DIS as follows: + ```bash + # Default not use --align_angle + # S3DIS_DIR: the directory of downloaded Stanford3dDataset_v1.2 dataset. RAW_S3DIS_DIR: the directory of the Stanford2d3dDataset_noXYZ dataset. PROCESSED_S3DIS_DIR: the directory of the processed S3DIS dataset (output dir). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.sh -d ${S3DIS_DIR} -r ${RAW_S3DIS_DIR} -o ${PROCESSED_S3DIS_DIR} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/s3dis/splits.py --dataset_root ${PROCESSED_S3DIS_DIR} + ``` + - ARKitScenes + - Download ArkitScenes 3DOD split with the following commands: + ```bash + # RAW_AS_DIR: the directory of downloaded Raw ArkitScenes dataset. + git clone https://github.com/apple/ARKitScenes.git + cd ARKitScenes + python download_data.py 3dod --download_dir $RAW_AS_DIR --video_id_csv threedod/3dod_train_val_splits.csv + ``` + - Run preprocessing code for ArkitScenes as follows: + ```bash + # RAW_AS_DIR: the directory of the downloaded ARKitScenes dataset. PROCESSED_AS_DIR: the directory of processed ArkitScenes dataset (output dir). + # NUM_WORKERS: Number of workers for preprocessing, default same as CPU count (might OOM). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.sh -d ${RAW_AS_DIR} -o ${PROCESSED_AS_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/arkitscenes/splits.py --dataset_root ${PROCESSED_ARKITSCENES_DIR} + ``` + - Habitat - Matterport 3D (HM3D) + - Download HM3D `hm3d-train-glb-v0.2.tar` and `hm3d-val-glb-v0.2.tar` with instuction [here](https://github.com/facebookresearch/habitat-sim/blob/main/DATASETS.md#habitat-matterport-3d-research-dataset-hm3d) and unzip them. + - Run preprocessing code for HM3D as follows: + ```bash + # We leverage the Habitat-Sim to simulate the camera views. The detailed installation for Habitat-Sim can be found at Habitat-Sim(https://github.com/facebookresearch/habitat-sim) + # RAW_HM_DIR: the directory of downloaded HM3D dataset. PROCESSED_HM_DIR: the directory of processed HM3D dataset (output dir). + # NUM_WORKERS: Number of workers for preprocessing, default same as CPU count (might OOM). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/hm3d/preprocess_hm3d.sh -d ${RAW_HM3D_DIR} -o ${PROCESSED_HM3D_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/hm3d/splits.py --dataset_root ${PROCESSED_HM3D_DIR} + ``` + - Structured3D + - Download Structured3D panorama related and perspective (full) related zip files by filling this [Google form](https://docs.google.com/forms/d/e/1FAIpQLSc0qtvh4vHSoZaW6UvlXYy79MbcGdZfICjh4_t4bYofQIVIdw/viewform?pli=1) (no need to unzip them). + - Organize all downloaded zip file in one folder (`${STRUCT3D_DIR}`). + - Run preprocessing code for Structured3D as follows: + ```bash + # RAW_STRUCT3D_DIR: the directory of downloaded Structured3D dataset. PROCESSED_STRUCT3D_DIR: the directory of processed Structured3D dataset (output dir). + # NUM_WORKERS: Number of workers for preprocessing, default same as CPU count (might OOM). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.sh -d ${RAW_STRUCT3D_DIR} -o ${PROCESSED_STRUCT3D_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/structured3d/splits.py --dataset_root ${PROCESSED_STRUCT3D_DIR} + ``` + - RealEstate10K (RE10K) + Additionally, we leverage video lifted RE10K by [VGGT](https://github.com/facebookresearch/vggt). Currently, the python file below only supports single data inference through VGGT. However, it is OK for you to start several python processes by setting num_workers and thread_id. + - Install VGGT as a package + ```bash + git clone https://github.com/facebookresearch/vggt.git + pip install -e . + ``` + - Download RealEstate10K inPixelSplat format[ [Here](http://schadenfreude.csail.mit.edu:8000/)] + - Preprocess the RE10K dataset + ```bash + # RAW_RE10K_DIR: the directory of downloaded RE10K dataset. + # PROCESSED_RE10K_DIR: the directory of processed RE10K dataset (output dir). + # using --parse_depths in the shell script to parse depths of the selected images. + python pointcept/datasets/preprocessing/concerto/re10k/preprocess_re10k.py --dataset_root ${RAW_RE10K_DIR} --output_root ${PROCESSED_RE10K_DIR} --num_workers ${NUM_WORKERS} --thread_id ${THREAD_ID} + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/re10k/splits.py --dataset_root ${PROCESSED_RE10K_DIR} + ``` + - SemanticKITTI + - Download [SemanticKITTI](http://www.semantic-kitti.org/dataset.html#download) dataset. + - SemanticKitti does not need any preprocessing. + ```bash + # SEMANTIC_KITTI_DIR: the directory of SemanticKITTI dataset. + # |- SEMANTIC_KITTI_DIR + # |- dataset + # |- sequences + # |- 00 + # |- 01 + # |- ... + ``` + + - nuScenes + - Download the official [NuScene](https://www.nuscenes.org/nuscenes#download) dataset (with Lidar Segmentation) and organize the downloaded files as follows: + ```bash + NUSCENES_DIR + │── samples + │── sweeps + │── lidarseg + ... + │── v1.0-trainval + │── v1.0-test + ``` + - Run information preprocessing code (modified from OpenPCDet) for nuScenes as follows: + ```bash + # NUSCENES_DIR: the directory of downloaded nuScenes dataset. + # PROCESSED_NUSCENES_DIR: the directory of processed nuScenes dataset (output dir). + # MAX_SWEEPS: Max number of sweeps. Default: 10. + pip install nuscenes-devkit pyquaternion + python pointcept/datasets/preprocessing/concerto/nuscenes/preprocess_nuscenes_info.py --dataset_root ${NUSCENES_DIR} --output_root ${PROCESSED_NUSCENES_DIR} --max_sweeps ${MAX_SWEEPS} --with_camera + ``` + + - Link raw dataset to processed NuScene dataset folder: + ```bash + # NUSCENES_DIR: the directory of downloaded nuScenes dataset. + # PROCESSED_NUSCENES_DIR: the directory of processed nuScenes dataset (output dir). + ln -s ${NUSCENES_DIR} {PROCESSED_NUSCENES_DIR}/raw + ``` + then the processed nuscenes folder is organized as follows: + ```bash + nuscene + |── raw + │── samples + │── sweeps + │── lidarseg + ... + │── v1.0-trainval + │── v1.0-test + |── info + ``` + - Waymo + - Download the official [Waymo](https://waymo.com/open/download/) dataset (v1.4.3) and organize the downloaded files as follows: + ```bash + WAYMO_RAW_DIR + │── training + │── validation + │── testing + ``` + - Install the following dependence: + ```bash + # If shows "No matching distribution found", download whl directly from Pypi and install the package. + conda create -n waymo python=3.10 -y + conda activate waymo + pip install waymo-open-dataset-tf-2-12-0 + ``` + - Run the preprocessing code as follows: + ```bash + # WAYMO_DIR: the directory of the downloaded Waymo dataset. + # PROCESSED_WAYMO_DIR: the directory of the processed Waymo dataset (output dir). + # NUM_WORKERS: num workers for preprocessing + bash pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.sh -d ${WAYMO_DIR} -o ${PROCESSED_WAYMO_DIR} -n ${NUM_WORKERS} -s "training validation" + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/waymo/splits.py --dataset_root ${PROCESSED_WAYMO_DIR} + ``` + + - After all the preprocessing, link processed dataset above to the codebase. + ```bash + # PROCESSED_DIR: the directory of the processed dataset (output dir). + # DATASET_NAME: the dataset name, which should be consistent with 'data_root' in the corresponding config, such as scannet, scannetpp, s3dis, arkitscenes, hm3d, structured3d, re10k, semantic_kitti, nuscenes, waymo + mkdir data + ln -s ${PROCESSED_DIR} ${CODEBASE_DIR}/data/${DATASET_NAME} + ``` + +##### 2.2 [Option B] Download Preprocessed Dataset +You can download the preprocessed datasets from Huggingface. The downloaded datasets should be put in the folders named after the ${DATASET_NAME}, which is the string in front of .tar.gz. + +- ScanNet v2 - [ [here](https://huggingface.co/datasets/Pointcept/concerto_scannet_compressed) ] +- ScanNet++ v2 - [ [here](https://huggingface.co/datasets/Pointcept/concerto_scannetpp_compressed) ] +- S3DIS - [ [here](https://huggingface.co/datasets/Pointcept/concerto_s3dis_compressed) ] +- ArkitScenes - [ [here](https://huggingface.co/datasets/Pointcept/concerto_arkitscenes_compressed) ] +- HM3D - [ [here](https://huggingface.co/datasets/Pointcept/concerto_hm3d_compressed) ] +- Structured3D - [ [here](https://huggingface.co/datasets/Pointcept/concerto_structured3d_compressed) ] +- RE10K - [ [here](https://huggingface.co/datasets/Pointcept/concerto_re10k_compressed) ] + +Then you need to decompress the tar.gz using: +```bash +cat ${DATASET_NAME}/${DATASET_NAME}.tar.gz.* | tar -xzvf - +``` +After decompressing, link processed dataset above to the codebase. +```bash +# PROCESSED_DIR: the directory of the processed dataset (output dir). +# DATASET_NAME: the dataset name, which should be consistent with 'data_root' in the corresponding config, such as scannet, scannetpp, s3dis, arkitscenes, hm3d, structured3d, re10k, semantic_kitti, nuscenes, waymo +mkdir data +ln -s ${PROCESSED_DIR} ${CODEBASE_DIR}/data/${DATASET_NAME} +``` + +#### 3. Pre-training +Enable pre-training with 32 GPUs (can be adjusted according to your hardware resources) by running the following script: +```bash +# default config +sh scripts/train.sh -m 4 -g 8 -d concerto -c pretrain-concerto-v1m1-0-base -n pretrain-concerto-v1m1-0-base + +# (or) a large version PTv3 backbone +sh scripts/train.sh -m 4 -g 8 -d concerto -c pretrain-concerto-v1m1-1-large-base -n pretrain-concerto-v1m1-1-large-base + +# (or) a large version PTv3 backbone with video-lifted RE10K data +sh scripts/train.sh -m 4 -g 8 -d concerto -c pretrain-concerto-v1m1-2-large-video -n pretrain-concerto-v1m1-2-large-video + +# (or) if OOM +# half the batch size and half the base learning rate + +# (or) if Nan appears +# try making grad clip lower as 2.0 or 1.0 +# try making enable_amp=False +``` + +#### 4. Probing and Tuning +Our pre-trained model weight can be downloaded [here](https://huggingface.co/Pointcept/Concerto/blob/main/pretrain-concerto-v1m1-2-large-video.pth). Read [model card](https://huggingface.co/Pointcept/Concerto/blob/main/README.md) on huggingface for more information about different model weights. For `concerto_large/base/small/large_outdoor.pth` on [huggingface](https://huggingface.co/Pointcept/Concerto/tree/main), replace the lines below in the config to correctly load the model weight: +```bash +dict( + type="CheckpointLoader", + keywords="module.student.backbone", + replacement="module.backbone", + ), +``` +to +```bash +dict( + type="CheckpointLoader", + keywords="module", + replacement="module.backbone", + ), +``` +Here are the example commands for probing and tuning on large model: +```bash +# Assume the pre-trained experiment is recorded in: +# exp/concerto/pretrain-concerto-v1m1-0-large-base + +# ScanNet +# linear probing +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0a-scannet-lin -n semseg-ptv3-large-v1m1-0a-scannet-lin -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +# decoder probing +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0b-scannet-dec -n semseg-ptv3-large-v1m1-0b-scannet-dec -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +# full fine-tuning +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0c-scannet-ft -n semseg-ptv3-large-v1m1-0c-scannet-ft -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +# PPT multi-dataset joint fine-tuning (heavy, not necessary) +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0d-scannet-ppt -n semseg-ptv3-large-v1m1-0d-scannet-ppt -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +# LoRA finetuning, need to change the backbone_path in config with your model that needs to be LoRA finetuned +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0f-scannet-ft-lora -n semseg-ptv3-large-v1m1-0f-scannet-ft-lora + +# ScanNet Data Efficiency full fine-tuning +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e0-scannet-ft-la20 -n semseg-ptv3-large-v1m1-0e0-scannet-ft-la20 -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e1-scannet-ft-la50 -n semseg-ptv3-large-v1m1-0e1-scannet-ft-la50 -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e2-scannet-ft-la100 -n semseg-ptv3-large-v1m1-0e2-scannet-ft-la100 -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e3-scannet-ft-la200 -n semseg-ptv3-large-v1m1-0e3-scannet-ft-la200 -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e4-scannet-ft-lr1 -n semseg-ptv3-large-v1m1-0e4-scannet-ft-lr1 -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e5-scannet-ft-lr5 -n semseg-ptv3-large-v1m1-0e5-scannet-ft-lr5 -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e6-scannet-ft-lr10 -n semseg-ptv3-large-v1m1-0e6-scannet-ft-lr10 -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e7-scannet-ft-lr20 -n semseg-ptv3-large-v1m1-0e7-scannet-ft-lr20 -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +# LoRA finetuning, need to change the backbone_path in config with your model that needs to be LoRA finetuned +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e8-scannet-ft-lora-la20 -n semseg-ptv3-large-v1m1-0e8-scannet-ft-lora-la20 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e9-scannet-ft-lora-la50 -n semseg-ptv3-large-v1m1-0e9-scannet-ft-lora-la50 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e10-scannet-ft-lora-la100 -n semseg-ptv3-large-v1m1-0e10-scannet-ft-lora-la100 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e11-scannet-ft-lora-la200 -n semseg-ptv3-large-v1m1-0e11-scannet-ft-lora-la200 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e12-scannet-ft-lora-lr1 -n semseg-ptv3-large-v1m1-0e12-scannet-ft-lora-lr1 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e13-scannet-ft-lora-lr5 -n semseg-ptv3-large-v1m1-0e13-scannet-ft-lora-lr5 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e14-scannet-ft-lora-lr10 -n semseg-ptv3-large-v1m1-0e14-scannet-ft-lora-lr10 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-0e15-scannet-ft-lora-lr20 -n semseg-ptv3-large-v1m1-0e15-scannet-ft-lora-lr20 + +# ScanNet200 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-1a-scannet200-lin -n semseg-ptv3-large-v1m1-1a-scannet200-lin -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-1b-scannet200-dec -n semseg-ptv3-large-v1m1-1b-scannet200-dec -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-1c-scannet200-ft -n semseg-ptv3-large-v1m1-1c-scannet200-ft -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +# LoRA finetuning, need to change the backbone_path in config with your model that needs to be LoRA finetuned +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-1e-scannet200-ft-lora -n semseg-ptv3-large-v1m1-1e-scannet200-ft-lora + +# ScanNetpp +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-2a-scannetpp-lin -n semseg-ptv3-large-v1m1-2a-scannetpp-lin -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-2b-scannetpp-dec -n semseg-ptv3-large-v1m1-2b-scannetpp-dec -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-2c-scannetpp-ft -n semseg-ptv3-large-v1m1-2c-scannetpp-ft -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +# LoRA finetuning, need to change the backbone_path in config with your model that needs to be LoRA finetuned +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-2g-scannetpp-ft-lora -n semseg-ptv3-large-v1m1-2g-scannetpp-ft-lora + +# S3DIS Area 5 +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-3a-s3dis-lin -n semseg-ptv3-large-v1m1-3a-s3dis-lin -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-3b-s3dis-dec -n semseg-ptv3-large-v1m1-3b-s3dis-dec -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-3c-s3dis-ft -n semseg-ptv3-large-v1m1-3c-s3dis-ft -w exp/concerto/pretrain-concerto-v1m1-0-large-base/model/model_last.pth +# LoRA finetuning, need to change the backbone_path in config with your model which need to be LoRA finetuned +sh scripts/train.sh -m 1 -g 8 -d concerto -c semseg-ptv3-large-v1m1-3e-s3dis-ft-lora -n semseg-ptv3-large-v1m1-3e-s3dis-ft-lora +``` +The above configs have not been verified with one additional run. If you encounter any problems, please feel free to let me know. Also, comparing changes between each configuration would be helpful in handling the configuration system of Pointcept. + +## Citation +If you find _Concerto_ useful to your research, please consider citing our line of works as an acknowledgment. (੭ˊ꒳​ˋ)੭✧ +```bib +@inproceedings{zhang2025concerto, + title={Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations}, + author={Zhang, Yujia and Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Tian, Zhuotao and Wang, Naiyan and Zhao, Hengshuang}, + booktitle={NeurIPS}, + year={2025} +} +``` + +```bib +@inproceedings{wu2025sonata, + title={Sonata: Self-Supervised Learning of Reliable Point Representations}, + author={Wu, Xiaoyang and DeTone, Daniel and Frost, Duncan and Shen, Tianwei and Xie, Chris and Yang, Nan and Engel, Jakob and Newcombe, Richard and Zhao, Hengshuang and Straub, Julian}, + booktitle={CVPR}, + year={2025} +} +``` + +```bib +@inproceedings{wu2024ptv3, + title={Point Transformer V3: Simpler, Faster, Stronger}, + author={Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang}, + booktitle={CVPR}, + year={2024} +} +``` +```bib +@inproceedings{wu2024ppt, + title={Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training}, + author={Wu, Xiaoyang and Tian, Zhuotao and Wen, Xin and Peng, Bohao and Liu, Xihui and Yu, Kaicheng and Zhao, Hengshuang}, + booktitle={CVPR}, + year={2024} +} +``` +```bib +@inproceedings{wu2023masked, + title={Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning}, + author={Wu, Xiaoyang and Wen, Xin and Liu, Xihui and Zhao, Hengshuang}, + journal={CVPR}, + year={2023} +} +``` +```bib +@inproceedings{wu2022ptv2, + title={Point transformer V2: Grouped Vector Attention and Partition-based Pooling}, + author={Wu, Xiaoyang and Lao, Yixing and Jiang, Li and Liu, Xihui and Zhao, Hengshuang}, + booktitle={NeurIPS}, + year={2022} +} +``` +```bib +@misc{pointcept2023, + title={Pointcept: A Codebase for Point Cloud Perception Research}, + author={Pointcept Contributors}, + howpublished={\url{https://github.com/Pointcept/Pointcept}}, + year={2023} +} +``` + +## License +- Concerto code is based on Sonata, which is released by Meta under the [Apache 2.0 license](https://github.com/facebookresearch/sonata/blob/main/LICENSE), +- Concerto weight is released under the [CC-BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en) +(restricted by NC of datasets like HM3D, ArkitScenes). +- For commercial usage, please removing datasets restrict by non-commercial license. diff --git a/pointcept/models/concerto/__init__.py b/pointcept/models/concerto/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a4bd3b9c528cdbb078ec59bc3f0799fa8670ec2b --- /dev/null +++ b/pointcept/models/concerto/__init__.py @@ -0,0 +1,2 @@ +from .concerto_v1m1_base import * +from .concerto_v1m2_distill import * diff --git a/pointcept/models/concerto/concerto_v1m1_base.py b/pointcept/models/concerto/concerto_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..56e417fcdb5bbbf3cf82bb3b9013507360ded70d --- /dev/null +++ b/pointcept/models/concerto/concerto_v1m1_base.py @@ -0,0 +1,872 @@ +""" +Concerto V1M1 + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from itertools import chain +from packaging import version +from functools import partial +import torch +import torch.nn as nn +import torch.nn.functional as F +import torch.distributed as dist +import torch_scatter +from timm.layers import trunc_normal_ +from transformers import AutoModel +from copy import deepcopy + +import pointops +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS, build_model +from pointcept.models.modules import PointModel +from pointcept.models.utils import ( + offset2batch, + offset2bincount, + batch2offset, + bincount2offset, +) +from pointcept.utils.comm import get_world_size, all_gather +from pointcept.utils.scheduler import CosineScheduler + + +class OnlineCluster(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=4096, + embed_channels=512, + num_prototypes=4096, + enable_mlp=True, + ): + super().__init__() + if enable_mlp: + self.mlp = nn.Sequential( + nn.Linear(in_channels, hidden_channels), + nn.GELU(), + nn.Linear(hidden_channels, embed_channels), + ) + self.apply(self._init_weights) + if version.parse(torch.__version__) >= version.parse("2.1.0"): + self.prototype = torch.nn.utils.parametrizations.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.parametrizations.weight.original0.data.fill_(1) + self.prototype.parametrizations.weight.original0.requires_grad = False + + else: + self.prototype = torch.nn.utils.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.weight_g.data.fill_(1) + self.prototype.weight_g.requires_grad = False + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + + def forward(self, feat): + if hasattr(self, "mlp"): + feat = self.mlp(feat) + eps = 1e-6 if feat.dtype == torch.float16 else 1e-12 + feat = nn.functional.normalize(feat, dim=-1, p=2, eps=eps) + similarity = self.prototype(feat) + return similarity + + +@MODELS.register_module("Concerto-v1m1") +class Concerto(PointModel): + def __init__( + self, + image_weight_name, + image_weight_path, + backbone, + head_in_channels, + backbone_out_channels, + embedding_channels, + patch_w, + patch_h, + student_pretrained_path=None, + teacher_pretrained_path=None, + student_pretrained=False, + head_hidden_channels=4096, + head_embed_channels=512, + head_num_prototypes=4096, + enc2d_head_in_channels=384, + enc2d_head_hidden_channels=4096, + enc2d_head_embed_channels=256, + enc2d_head_num_prototypes=384, + teacher_custom=None, + num_global_view=2, + num_local_view=4, + mask_size_start=0.1, + mask_size_base=0.4, + mask_size_warmup_ratio=0.05, + mask_ratio_start=0.3, + mask_ratio_base=0.7, + mask_ratio_warmup_ratio=0.05, + mask_jitter=None, + teacher_temp_start=0.04, + teacher_temp_base=0.07, + teacher_temp_warmup_ratio=0.05, + student_temp=0.1, + mask_loss_weight=2 / 10, + roll_mask_loss_weight=2 / 10, + unmask_loss_weight=4 / 10, + enc2d_loss_weight=2 / 10, + momentum_base=0.996, + momentum_final=1, + match_max_k=8, + match_max_r=0.08, + up_cast_level=2, + enc2d_upcast_level=4, + enc2d_cos_shift=True, + sonata_model_type="offline", + ): + super(Concerto, self).__init__() + assert sonata_model_type in ["online", "offline"] + self.mask_loss_weight = mask_loss_weight + self.roll_mask_loss_weight = roll_mask_loss_weight + self.unmask_loss_weight = unmask_loss_weight + self.enc2d_loss_weight = enc2d_loss_weight + + self.num_global_view = num_global_view + self.num_local_view = num_local_view + + # masking and scheduler + self.mask_size = mask_size_start + self.mask_size_start = mask_size_start + self.mask_size_base = mask_size_base + self.mask_size_warmup_ratio = mask_size_warmup_ratio + self.mask_size_scheduler = None + + self.mask_ratio = mask_ratio_start + self.mask_ratio_start = mask_ratio_start + self.mask_ratio_base = mask_ratio_base + self.mask_ratio_warmup_ratio = mask_ratio_warmup_ratio + self.mask_ratio_scheduler = None + + self.mask_jitter = mask_jitter + + # temperature and scheduler + self.teacher_temp = teacher_temp_start + self.teacher_temp_start = teacher_temp_start + self.teacher_temp_base = teacher_temp_base + self.teacher_temp_warmup_ratio = teacher_temp_warmup_ratio + self.teacher_temp_scheduler = None + self.student_temp = student_temp + + # momentum and scheduler + self.momentum = momentum_base + self.momentum_base = momentum_base + self.momentum_final = momentum_final + self.momentum_scheduler = None + + # dynamic matching + self.match_max_k = match_max_k + self.match_max_r = match_max_r + + # up cast level + self.up_cast_level = up_cast_level + self.enc2d_upcast_level = enc2d_upcast_level + + # one of unmask, mask, roll mask loss enable + assert ( + unmask_loss_weight + + mask_loss_weight + + roll_mask_loss_weight + + enc2d_loss_weight + > 0 + ) + # roll mask loss need more than one global view + assert num_global_view > 1 or roll_mask_loss_weight == 0 + # current roll mask only support two global views + assert num_global_view == 1 or num_global_view == 2 + + student_model_dict = dict() + teacher_model_dict = dict() + if teacher_custom is None: + teacher_custom = {} + student_backbone = build_model(backbone) + if student_pretrained_path != None: + print("Load pretrained student model") + student_backbone = self.load_sonata( + student_backbone, path=student_pretrained_path + ) + + # turn off parameters like drop path for teacher model + backbone.update(teacher_custom) + + teacher_backbone = build_model(backbone) + if sonata_model_type == "offline": + teacher_backbone = self.load_sonata( + teacher_backbone, path=teacher_pretrained_path + ) + student_model_dict["backbone"] = student_backbone + teacher_model_dict["backbone"] = teacher_backbone + + if self.enc2d_loss_weight > 0: + self.patch_h = patch_h + self.patch_w = patch_w + self.image_weight_name = image_weight_name + + # Load Model + self.enc2d_model = self.load_enc2d(image_weight_name, image_weight_path) + self.enc2d_model.requires_grad_(False) + self._num_channels = enc2d_head_in_channels + self.patch_proj = torch.nn.Linear(backbone_out_channels, self._num_channels) + + enc2d_head = partial( + OnlineCluster, + in_channels=enc2d_head_in_channels, + hidden_channels=enc2d_head_hidden_channels, + embed_channels=enc2d_head_in_channels, + num_prototypes=enc2d_head_num_prototypes, + ) + enc2d_head_ = partial( + OnlineCluster, + in_channels=enc2d_head_in_channels, + hidden_channels=enc2d_head_hidden_channels, + embed_channels=enc2d_head_in_channels, + num_prototypes=enc2d_head_num_prototypes, + enable_mlp=False, + ) + + self.enc2d_head_student = enc2d_head() + self.enc2d_head_teacher = enc2d_head_() + self.enc2d_head_student.prototype.load_state_dict( + self.enc2d_head_teacher.prototype.state_dict() + ) + for p in self.enc2d_head_teacher.parameters(): + p.requires_grad = False + + head = partial( + OnlineCluster, + in_channels=head_in_channels, + hidden_channels=head_hidden_channels, + embed_channels=head_embed_channels, + num_prototypes=head_num_prototypes, + ) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + student_model_dict["mask_head"] = head() + teacher_model_dict["mask_head"] = head() + if self.unmask_loss_weight > 0: + student_model_dict["unmask_head"] = head() + teacher_model_dict["unmask_head"] = head() + if ( + self.enc2d_loss_weight > 0 + and self.unmask_loss_weight + + self.mask_loss_weight + + self.roll_mask_loss_weight + == 0 + ): + student_model_dict["unmask_head"] = head() + teacher_model_dict["unmask_head"] = head() + + self.student = nn.ModuleDict(student_model_dict) + self.teacher = nn.ModuleDict(teacher_model_dict) + + for k, v in self.student.items(): + if "head" in k: + self.teacher[k].load_state_dict(self.student[k].state_dict()) + if sonata_model_type == "online": + self.teacher.backbone.load_state_dict(self.student.backbone.state_dict()) + for n, p in self.teacher.named_parameters(): + p.requires_grad = False + + self.enc2d_cos_shift = enc2d_cos_shift + self.sonata_model_type = sonata_model_type + + def load_enc2d(self, model_name, model_weight): + model = AutoModel.from_pretrained(model_weight, trust_remote_code=True) + return model.eval() + + def load_sonata(self, model, path): + checkpoint = torch.load(path, map_location=lambda storage, loc: storage.cuda()) + weight = {} + whether_weight = False + if "state_dict" in checkpoint.keys(): + checkpoint = checkpoint["state_dict"] + for key, value in checkpoint.items(): + if "module.student.backbone." in key: + whether_weight = True + key = key.replace("module.student.backbone.", "module.") + key = key[7:] # module.xxx.xxx -> xxx.xxx + weight[key] = value + if whether_weight: + load_state_info = model.load_state_dict(weight) + else: + load_state_info = model.load_state_dict(checkpoint) + print(f"Missing keys: {load_state_info[0]}") + print(f"Unexpected keys: {load_state_info[1]}") + return model + + @torch.no_grad() + def ENC2D_forward(self, x): + # RADIO + if "radio" in self.image_weight_name: + summary, features = self.enc2d_model(x) + features = features.reshape( + -1, self.patch_h * self.patch_w, self._num_channels + ) + return features + # SigLIPv2 + if hasattr(self.enc2d_model, "vision_model"): + outputs = self.enc2d_model.vision_model(x) + features = outputs.last_hidden_state + # DINOv2.5 + else: + outputs = self.enc2d_model(x) + features = outputs.last_hidden_state[:, -self.patch_h * self.patch_w :, :] + return features + + def before_train(self): + # make ModelHook after CheckPointLoader + total_steps = self.trainer.cfg.scheduler.total_steps + curr_step = self.trainer.start_epoch * len(self.trainer.train_loader) + # mask size scheduler + self.mask_size_scheduler = CosineScheduler( + start_value=self.mask_size_start, + base_value=self.mask_size_base, + final_value=self.mask_size_base, + warmup_iters=int(total_steps * self.mask_size_warmup_ratio), + total_iters=total_steps, + ) + self.mask_size_scheduler.iter = curr_step + + # mask ratio scheduler + self.mask_ratio_scheduler = CosineScheduler( + start_value=self.mask_ratio_start, + base_value=self.mask_ratio_base, + final_value=self.mask_ratio_base, + warmup_iters=int(total_steps * self.mask_ratio_warmup_ratio), + total_iters=total_steps, + ) + self.mask_ratio_scheduler.iter = curr_step + + # teacher temperature scheduler + self.teacher_temp_scheduler = CosineScheduler( + start_value=self.teacher_temp_start, + base_value=self.teacher_temp_base, + final_value=self.teacher_temp_base, + warmup_iters=int(total_steps * self.teacher_temp_warmup_ratio), + total_iters=total_steps, + ) + self.teacher_temp_scheduler.iter = curr_step + + # momentum scheduler + self.momentum_scheduler = CosineScheduler( + base_value=self.momentum_base, + final_value=self.momentum_final, + total_iters=total_steps, + ) + self.momentum_scheduler.iter = curr_step + + def before_step(self): + # update parameters from schedulers + self.mask_size = self.mask_size_scheduler.step() + self.mask_ratio = self.mask_ratio_scheduler.step() + self.teacher_temp = self.teacher_temp_scheduler.step() + self.momentum = self.momentum_scheduler.step() + + if self.trainer.writer is not None: + self.trainer.writer.add_scalar( + "params/mask_size", + self.mask_size, + self.mask_size_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/mask_ratio", + self.mask_ratio, + self.mask_ratio_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/teacher_temp", + self.teacher_temp, + self.teacher_temp_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/momentum", + self.momentum, + self.momentum_scheduler.iter, + ) + + def after_step(self): + # pass + # EMA update teacher + with torch.no_grad(): + m = self.momentum + if self.sonata_model_type == "online": + student_param_list = list(self.student.backbone.parameters()) + teacher_param_list = list(self.teacher.backbone.parameters()) + torch._foreach_mul_(teacher_param_list, m) + torch._foreach_add_(teacher_param_list, student_param_list, alpha=1 - m) + + student_param_list = [ + p for n, p in self.student.named_parameters() if "head" in n + ] + teacher_param_list = [ + p for n, p in self.teacher.named_parameters() if "head" in n + ] + torch._foreach_mul_(teacher_param_list, m) + torch._foreach_add_(teacher_param_list, student_param_list, alpha=1 - m) + + if self.enc2d_loss_weight > 0: + enc2d_student_param_list = [ + p + for n, p in self.enc2d_head_student.named_parameters() + if "prototype" in n + ] + enc2d_teacher_param_list = [ + p + for n, p in self.enc2d_head_teacher.named_parameters() + if "prototype" in n + ] + torch._foreach_copy_(enc2d_teacher_param_list, enc2d_student_param_list) + + @staticmethod + def sinkhorn_knopp(feat, temp, num_iter=3): + feat = feat.float() + q = torch.exp(feat / temp).t() + n = sum(all_gather(q.shape[1])) # number of samples to assign + k = q.shape[0] # number of prototypes + + # make the matrix sums to 1 + sum_q = q.sum() + if get_world_size() > 1: + dist.all_reduce(sum_q) + q = q / sum_q + + for i in range(num_iter): + # normalize each row: total weight per prototype must be 1/k + q_row_sum = q.sum(dim=1, keepdim=True) + if get_world_size() > 1: + dist.all_reduce(q_row_sum) + q = q / q_row_sum / k + + # normalize each column: total weight per sample must be 1/n + q = q / q.sum(dim=0, keepdim=True) / n + + q *= n # the columns must sum to 1 so that Q is an assignment + return q.t() + + def generate_mask(self, coord, offset): + batch = offset2batch(offset) + mask_size = self.mask_size + mask_ratio = self.mask_ratio + + # Grouping points with grid patch + min_coord = torch_scatter.segment_coo(coord, batch, reduce="min") + grid_coord = ((coord - min_coord[batch]) // mask_size).int() + grid_coord = torch.cat([batch.unsqueeze(-1), grid_coord], dim=-1) + unique, point_cluster, counts = torch.unique( + grid_coord, dim=0, sorted=True, return_inverse=True, return_counts=True + ) + patch_num = unique.shape[0] + mask_patch_num = int(patch_num * mask_ratio) + patch_index = torch.randperm(patch_num, device=coord.device) + mask_patch_index = patch_index[:mask_patch_num] + point_mask = torch.isin(point_cluster, mask_patch_index) + return point_mask, point_cluster + + @torch.no_grad() + def match_neighbour( + self, + view1_coord, + view1_offset, + view2_coord, + view2_offset, + ): + index2, distance = pointops.knn_query( + 1, + view2_coord.float(), + view2_offset.int(), + view1_coord.float(), + view1_offset.int(), + ) + index1 = torch.arange( + index2.shape[0], device=index2.device, dtype=torch.long + ).unsqueeze(-1) + index = torch.cat([index1, index2], dim=-1)[ + distance.squeeze(-1) < self.match_max_r + ] + return index + + @torch.no_grad() + def roll_point(self, point): + n = self.num_global_view + # [pc1, pc1', pc2, pc2'] -> [pc1', pc1, pc2', pc2], only support num_global_view == 2 + bs = len(point.offset) // self.num_global_view + data_dict = {} + for key in point.keys(): + if key in ["feat", "coord", "origin_coord", "batch"]: + value = point[key].split(offset2bincount(point.offset).tolist()) + value = chain(*[value[n * b : n * (b + 1)][::-1] for b in range(bs)]) + if key == "batch": + value = [torch.ones_like(v) * i for i, v in enumerate(value)] + data_dict[key] = torch.cat(list(value), dim=0) + return Point(data_dict) + + def up_cast(self, point, upcast_level=None): + if upcast_level is None: + upcast_level = self.up_cast_level + else: + upcast_level = upcast_level + for _ in range(upcast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return point + + @staticmethod + def pool_corr(point, correspondence): + inverse_list = [] + idx_ptr_list = [] + point_feat = dict(offset=point.offset, feat=point.feat) + while "pooling_parent" in point.keys(): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + assert "idx_ptr" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + idx_ptr = point.pop("idx_ptr") + inverse_list.append(inverse) + idx_ptr_list.append(idx_ptr) + point = parent + inverse_list.reverse() + idx_ptr_list.reverse() + for inverse, idx_ptr in zip(inverse_list, idx_ptr_list): + _, indices = torch.sort(inverse) + img_num = correspondence.shape[1] + correspondence_all = [] + if img_num == 0: + correspondence_all = -torch.ones((idx_ptr.shape[0] - 1, 0, 2)).cuda() + else: + for img_id in range(img_num): + mask = torch.all( + correspondence[:, img_id] != torch.tensor([-1, -1]).cuda(), + dim=1, + ).float() + counts = torch_scatter.segment_csr( + mask[indices], idx_ptr, reduce="sum" + ) + counts[counts == 0] = 100000 + correspondence_img = deepcopy(correspondence[:, img_id]) + correspondence_img[correspondence_img == -1] = 0 + mask_sum = torch_scatter.segment_csr( + correspondence_img[indices], idx_ptr, reduce="sum" + ) + mask_sum = mask_sum / counts.unsqueeze(1) + mask_sum[counts == 100000] = -1 + correspondence_all.append(mask_sum) + correspondence_all = torch.stack(correspondence_all, dim=1) + correspondence = correspondence_all + point_feat["correspondence"] = correspondence + point_feat = Point(point_feat) + return point_feat + + def forward(self, data_dict, return_point=False): + if return_point: + point = self.teacher.backbone(data_dict) + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return dict(point=point) + + # prepare global_point, mask_global_point, local_point + with torch.no_grad(): + # global_point & masking + global_point = Point( + feat=data_dict["global_feat"], + coord=data_dict["global_coord"], + origin_coord=data_dict["global_origin_coord"], + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + + global_mask, global_cluster = self.generate_mask( + global_point.coord, global_point.offset + ) + mask_global_coord = global_point.coord.clone().detach() + if self.mask_jitter is not None: + mask_global_coord[global_mask] += torch.clip( + torch.randn_like(mask_global_coord[global_mask]).mul( + self.mask_jitter + ), + max=self.mask_jitter * 2, + ) + + mask_global_point = Point( + feat=data_dict["global_feat"], + coord=mask_global_coord, + origin_coord=data_dict["global_origin_coord"], + mask=global_mask, + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + major_view_correspondence = data_dict["global_correspondence"] + + # local point & matching + local_point = Point( + feat=data_dict["local_feat"], + coord=data_dict["local_coord"], + origin_coord=data_dict["local_origin_coord"], + offset=data_dict["local_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # create result dictionary for return + result_dict = dict(loss=[]) + # teacher forward + global_point_ = self.teacher.backbone(global_point) + global_point_ = self.up_cast(global_point_) + # teacher head forward + # only use one shared head for both mask and unmask + # priority: mask (global) > unmask (local) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + global_point_.feat = self.teacher.mask_head(global_point_.feat) + else: + global_point_.feat = self.teacher.unmask_head(global_point_.feat) + + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + # student forward + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_pred_sim = self.student.mask_head(mask_global_point_.feat) + + if self.mask_loss_weight > 0: + with torch.no_grad(): + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + global_point_.origin_coord, + global_point_.offset, + ) + # teacher forward + mask_target_sim = self.sinkhorn_knopp( + global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + # loss + mask_loss = -torch.sum( + mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + + mask_loss = torch_scatter.segment_coo( + mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["mask_loss"] = mask_loss + result_dict["loss"].append(mask_loss * self.mask_loss_weight) + + if self.roll_mask_loss_weight > 0: + roll_global_point_ = self.roll_point(global_point_) + with torch.no_grad(): + # match index for pred and roll target + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + roll_global_point_.origin_coord, + roll_global_point_.offset, + ) + # teacher forward + roll_mask_target_sim = self.sinkhorn_knopp( + roll_global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + roll_mask_loss = -torch.sum( + roll_mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + roll_mask_loss = torch_scatter.segment_coo( + roll_mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["roll_mask_loss"] = roll_mask_loss + result_dict["loss"].append(roll_mask_loss * self.roll_mask_loss_weight) + if self.unmask_loss_weight > 0: + # student forward + local_point_ = self.student.backbone(local_point) + local_point_ = self.up_cast(local_point_) + unmask_pred_sim = self.student.unmask_head(local_point_.feat) + with torch.no_grad(): + principal_view_mask = global_point_.batch % self.num_global_view == 0 + principal_view_batch = ( + global_point_.batch[principal_view_mask] // self.num_global_view + ) + match_index = self.match_neighbour( + local_point_.origin_coord, + local_point_.offset[self.num_local_view - 1 :: self.num_local_view], + global_point_.origin_coord[principal_view_mask], + batch2offset(principal_view_batch), + ) + # teacher forward + unmask_target_sim = self.sinkhorn_knopp( + global_point_.feat[principal_view_mask][match_index[:, 1]], + self.teacher_temp, + ) + # loss + unmask_loss = -torch.sum( + unmask_target_sim + * F.log_softmax( + unmask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + unmask_loss = torch_scatter.segment_coo( + unmask_loss, + index=local_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["unmask_loss"] = unmask_loss + result_dict["loss"].append(unmask_loss * self.unmask_loss_weight) + if self.enc2d_loss_weight > 0: + if self.mask_loss_weight == 0 or self.roll_mask_loss_weight == 0: + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_global_point_enc2d = self.up_cast( + mask_global_point_, + upcast_level=self.enc2d_upcast_level - self.up_cast_level, + ) + to_feature = self.pool_corr( + mask_global_point_enc2d, major_view_correspondence + ) + data_dict_global_offset = torch.cat( + [torch.tensor([0]).cuda(), to_feature["offset"]], dim=0 + ) + enc2d_count = ( + data_dict_global_offset[ + 1 : len(data_dict_global_offset) : self.num_global_view + ] + - data_dict_global_offset[ + 0 : len(data_dict_global_offset) - 1 : self.num_global_view + ] + ) + enc2d_offset = torch.cat( + [torch.tensor([0]).cuda(), torch.cumsum(enc2d_count, dim=0)] + ) + enc2d_mask = torch.cat( + [ + torch.arange(0, c, device=enc2d_count.device) + + data_dict_global_offset[i * self.num_global_view] + for i, c in enumerate(enc2d_count) + ], + dim=0, + ) + + offset_points_3d = enc2d_offset[1:] + batch_points_3d = offset2batch(offset_points_3d) + imgs = data_dict["images"] + feature3d = to_feature["feat"][enc2d_mask] + enc2d_global_mask = global_mask[enc2d_mask] + correspondence = to_feature["correspondence"][enc2d_mask] + v0 = correspondence.shape[1] + mask = torch.any(correspondence != torch.tensor([-1, -1]).cuda(), dim=2) + enc2d_global_mask = enc2d_global_mask.unsqueeze(1).expand(-1, v0) + valid_index = torch.where(mask) # 0: 3d points index, 1: view index + + bincount_img_num = data_dict["img_num"] + offset_img_num = bincount2offset(bincount_img_num) + total_img_num = offset_img_num[-1] + + if total_img_num > 0: + # expand + with torch.no_grad(): + feature2d = self.ENC2D_forward(imgs) + feature2d = feature2d.contiguous().view(-1, feature2d.shape[-1]) + feature2d_mask = feature2d + + offset_img_num = torch.cat([torch.tensor([0]).cuda(), offset_img_num])[ + :-1 + ] + batch_index = batch_points_3d[valid_index[0]] + batch_img_num = offset_img_num[batch_index] + + feature3d_mask = feature3d[valid_index[0]] + + feature_index = torch.cat( + [ + batch_img_num.unsqueeze(-1), + valid_index[1].unsqueeze(-1), + correspondence[valid_index], + ], + dim=-1, + ) + feature_index = feature_index.long() + feature_index = ( + feature_index[:, 0] * self.patch_h * self.patch_w + + feature_index[:, 1] * self.patch_h * self.patch_w + + feature_index[:, 2] * self.patch_w + + feature_index[:, 3] + ) + + feature_index = feature_index.long() + feature3d_mask = torch_scatter.scatter_mean( + feature3d_mask, feature_index, dim=0, dim_size=feature2d.shape[0] + ) + feature3d_mask = self.patch_proj(feature3d_mask) + feature_index = torch.unique(feature_index) + feature2d_mask = feature2d_mask[feature_index] + feature3d_mask = feature3d_mask[feature_index] + + if self.enc2d_cos_shift: + feature2d_mask = feature2d_mask - feature2d_mask.mean( + dim=-1, keepdim=True + ) + feature3d_mask = feature3d_mask - feature3d_mask.mean( + dim=-1, keepdim=True + ) + cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6) + loss = (1 - cos(feature2d_mask, feature3d_mask)).mean() * 10 + + result_dict["enc2d_loss"] = loss + result_dict["loss"].append(loss * self.enc2d_loss_weight) + del ( + feature2d, + feature3d, + feature2d_mask, + feature3d_mask, + correspondence, + feature_index, + ) + elif ( + self.mask_loss_weight + + self.unmask_loss_weight + + self.roll_mask_loss_weight + > 0 + ): + result_ssl_loss = sum(result_dict["loss"]) / ( + self.mask_loss_weight + + self.unmask_loss_weight + + self.roll_mask_loss_weight + ) + result_dict["enc2d_loss"] = result_ssl_loss + result_dict["loss"].append(result_ssl_loss * self.enc2d_loss_weight) + result_dict["loss"] = sum(result_dict["loss"]) + + if get_world_size() > 1: + for loss_id, loss in result_dict.items(): + dist.all_reduce(loss, op=dist.ReduceOp.AVG) + return result_dict diff --git a/pointcept/models/concerto/concerto_v1m2_distill.py b/pointcept/models/concerto/concerto_v1m2_distill.py new file mode 100644 index 0000000000000000000000000000000000000000..187f1375bec21ccd5ff34ec849f2c31d3bc411e3 --- /dev/null +++ b/pointcept/models/concerto/concerto_v1m2_distill.py @@ -0,0 +1,824 @@ +""" +Concerto V1m2 Distill + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from itertools import chain +from packaging import version +from functools import partial +import torch +import torch.nn as nn +import torch.nn.functional as F +import torch.distributed as dist +import torch_scatter +from timm.layers import trunc_normal_ +from transformers import AutoModel, AutoProcessor +from copy import deepcopy + +import pointops +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS, build_model +from pointcept.models.modules import PointModel +from pointcept.models.utils import ( + offset2batch, + offset2bincount, + batch2offset, + bincount2offset, +) +from pointcept.utils.comm import get_world_size, all_gather +from pointcept.utils.scheduler import CosineScheduler + + +class OnlineCluster(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=4096, + embed_channels=512, + num_prototypes=4096, + enable_mlp=True, + ): + super().__init__() + if enable_mlp: + self.mlp = nn.Sequential( + nn.Linear(in_channels, hidden_channels), + nn.GELU(), + nn.Linear(hidden_channels, embed_channels), + ) + self.apply(self._init_weights) + if version.parse(torch.__version__) >= version.parse("2.1.0"): + self.prototype = torch.nn.utils.parametrizations.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.parametrizations.weight.original0.data.fill_(1) + self.prototype.parametrizations.weight.original0.requires_grad = False + + else: + self.prototype = torch.nn.utils.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.weight_g.data.fill_(1) + self.prototype.weight_g.requires_grad = False + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + + def forward(self, feat): + if hasattr(self, "mlp"): + feat = self.mlp(feat) + eps = 1e-6 if feat.dtype == torch.float16 else 1e-12 + feat = nn.functional.normalize(feat, dim=-1, p=2, eps=eps) + similarity = self.prototype(feat) + return similarity + + +@MODELS.register_module("Concerto-v1m2_distill") +class Concerto(PointModel): + def __init__( + self, + image_weight_name, + image_weight_path, + backbone_s, + backbone_t, + head_in_channels_s, + head_in_channels_t, + backbone_out_channels, + patch_w, + patch_h, + head_hidden_channels=4096, + head_embed_channels=512, + head_num_prototypes=4096, + enc2d_head_in_channels=384, + enc2d_head_hidden_channels=4096, + enc2d_head_embed_channels=256, + enc2d_head_num_prototypes=384, + teacher_custom=None, + num_global_view=2, + num_local_view=4, + mask_size_start=5, + mask_size_base=20, + mask_size_warmup_ratio=0.05, + mask_ratio_start=0.3, + mask_ratio_base=0.7, + mask_ratio_warmup_ratio=0.05, + mask_jitter=None, + teacher_temp_start=0.04, + teacher_temp_base=0.07, + teacher_temp_warmup_ratio=0.05, + student_temp=0.1, + mask_loss_weight=2 / 10, + roll_mask_loss_weight=2 / 10, + unmask_loss_weight=4 / 10, + enc2d_loss_weight=2 / 10, + momentum_base=0.996, + momentum_final=1, + match_max_k=8, + match_max_r=0.08, + up_cast_level=2, + teacher_pretrained_path=None, + enc2d_upcast_level=4, + enc2d_cos_shift=True, + ): + super(Concerto, self).__init__() + self.mask_loss_weight = mask_loss_weight + self.roll_mask_loss_weight = roll_mask_loss_weight + self.unmask_loss_weight = unmask_loss_weight + self.enc2d_loss_weight = enc2d_loss_weight + + self.num_global_view = num_global_view + self.num_local_view = num_local_view + + # masking and scheduler + self.mask_size = mask_size_start + self.mask_size_start = mask_size_start + self.mask_size_base = mask_size_base + self.mask_size_warmup_ratio = mask_size_warmup_ratio + self.mask_size_scheduler = None + + self.mask_ratio = mask_ratio_start + self.mask_ratio_start = mask_ratio_start + self.mask_ratio_base = mask_ratio_base + self.mask_ratio_warmup_ratio = mask_ratio_warmup_ratio + self.mask_ratio_scheduler = None + + self.mask_jitter = mask_jitter + + # temperature and scheduler + self.teacher_temp = teacher_temp_start + self.teacher_temp_start = teacher_temp_start + self.teacher_temp_base = teacher_temp_base + self.teacher_temp_warmup_ratio = teacher_temp_warmup_ratio + self.teacher_temp_scheduler = None + self.student_temp = student_temp + + # momentum and scheduler + self.momentum = momentum_base + self.momentum_base = momentum_base + self.momentum_final = momentum_final + self.momentum_scheduler = None + + # dynamic matching + self.match_max_k = match_max_k + self.match_max_r = match_max_r + + # up cast level + self.up_cast_level = up_cast_level + self.enc2d_upcast_level = enc2d_upcast_level + + # one of unmask, mask, roll mask loss enable + assert ( + unmask_loss_weight + + mask_loss_weight + + roll_mask_loss_weight + + enc2d_loss_weight + > 0 + ) + # roll mask loss need more than one global view + assert num_global_view > 1 or roll_mask_loss_weight == 0 + # current roll mask only support two global views + assert num_global_view == 1 or num_global_view == 2 + + student_model_dict = dict() + teacher_model_dict = dict() + if teacher_custom is None: + teacher_custom = {} + student_backbone = build_model(backbone_s) + # turn off parameters like drop path for teacher model + backbone_t.update(teacher_custom) + + teacher_backbone = build_model(backbone_t) + student_model_dict["backbone"] = student_backbone + teacher_model_dict["backbone"] = teacher_backbone + + if self.enc2d_loss_weight > 0: + self.patch_h = patch_h + self.patch_w = patch_w + self.image_weight_name = image_weight_name + + # Load Model + self.enc2d_model = self.load_enc2d(image_weight_name, image_weight_path) + self.enc2d_model.requires_grad_(False) + self._num_channels = enc2d_head_in_channels + self.patch_proj = torch.nn.Linear(backbone_out_channels, self._num_channels) + + enc2d_head = partial( + OnlineCluster, + in_channels=enc2d_head_in_channels, + hidden_channels=enc2d_head_hidden_channels, + embed_channels=enc2d_head_in_channels, + num_prototypes=enc2d_head_num_prototypes, + ) + enc2d_head_ = partial( + OnlineCluster, + in_channels=enc2d_head_in_channels, + hidden_channels=enc2d_head_hidden_channels, + embed_channels=enc2d_head_in_channels, + num_prototypes=enc2d_head_num_prototypes, + enable_mlp=False, + ) + + self.enc2d_head_student = enc2d_head() + self.enc2d_head_teacher = enc2d_head_() + self.enc2d_head_student.prototype.load_state_dict( + self.enc2d_head_teacher.prototype.state_dict() + ) + for p in self.enc2d_head_teacher.parameters(): + p.requires_grad = False + + head_t = partial( + OnlineCluster, + in_channels=head_in_channels_t, + hidden_channels=head_hidden_channels, + embed_channels=head_embed_channels, + num_prototypes=head_num_prototypes, + ) + head_s = partial( + OnlineCluster, + in_channels=head_in_channels_s, + hidden_channels=head_hidden_channels, + embed_channels=head_embed_channels, + num_prototypes=head_num_prototypes, + ) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + student_model_dict["mask_head"] = head_s() + teacher_model_dict["mask_head"] = head_t() + if self.unmask_loss_weight > 0: + student_model_dict["unmask_head"] = head_s() + teacher_model_dict["unmask_head"] = head_t() + + self.student = nn.ModuleDict(student_model_dict) + self.teacher = nn.ModuleDict(teacher_model_dict) + self.teacher = self.load_sonata(self.teacher, path=teacher_pretrained_path) + for p in self.teacher.parameters(): + p.requires_grad = False + + self.enc2d_cos_shift = enc2d_cos_shift + + def load_enc2d(self, model_name, model_weight): + model = AutoModel.from_pretrained(model_weight, trust_remote_code=True) + return model.eval() + + def load_sonata(self, model, path, if_loadhead=True): + checkpoint = torch.load(path, map_location=lambda storage, loc: storage.cuda()) + weight = {} + whether_weight = False + if "state_dict" in checkpoint.keys(): + checkpoint = checkpoint["state_dict"] + for key, value in checkpoint.items(): + if if_loadhead: + if "module.student." in key: + whether_weight = True + key = key.replace("module.student.", "module.") + key = key[7:] # module.xxx.xxx -> xxx.xxx + weight[key] = value + else: + if "module.student.backbone." in key: + whether_weight = True + key = key.replace("module.student.backbone.", "module.") + key = key[7:] # module.xxx.xxx -> xxx.xxx + weight[key] = value + if whether_weight: + load_state_info = model.load_state_dict(weight, strict=False) + else: + load_state_info = model.load_state_dict(checkpoint) + print(f"Missing keys: {load_state_info[0]}") + print(f"Unexpected keys: {load_state_info[1]}") + return model + + @torch.no_grad() + def ENC2D_forward(self, x): + # RADIO + if "radio" in self.image_weight_name: + summary, features = self.enc2d_model(x) + features = features.reshape( + -1, self.patch_h * self.patch_w, self._num_channels + ) + return features + # SigLIPv2 + if hasattr(self.enc2d_model, "vision_model"): + outputs = self.enc2d_model.vision_model(x) + features = outputs.last_hidden_state + # DINOv2.5 + else: + outputs = self.enc2d_model(x) + features = outputs.last_hidden_state[:, -self.patch_h * self.patch_w :, :] + return features + + def before_train(self): + # make ModelHook after CheckPointLoader + total_steps = self.trainer.cfg.scheduler.total_steps + curr_step = self.trainer.start_epoch * len(self.trainer.train_loader) + # mask size scheduler + self.mask_size_scheduler = CosineScheduler( + start_value=self.mask_size_start, + base_value=self.mask_size_base, + final_value=self.mask_size_base, + warmup_iters=int(total_steps * self.mask_size_warmup_ratio), + total_iters=total_steps, + ) + self.mask_size_scheduler.iter = curr_step + + # mask ratio scheduler + self.mask_ratio_scheduler = CosineScheduler( + start_value=self.mask_ratio_start, + base_value=self.mask_ratio_base, + final_value=self.mask_ratio_base, + warmup_iters=int(total_steps * self.mask_ratio_warmup_ratio), + total_iters=total_steps, + ) + self.mask_ratio_scheduler.iter = curr_step + + # teacher temperature scheduler + self.teacher_temp_scheduler = CosineScheduler( + start_value=self.teacher_temp_start, + base_value=self.teacher_temp_base, + final_value=self.teacher_temp_base, + warmup_iters=int(total_steps * self.teacher_temp_warmup_ratio), + total_iters=total_steps, + ) + self.teacher_temp_scheduler.iter = curr_step + + # momentum scheduler + self.momentum_scheduler = CosineScheduler( + base_value=self.momentum_base, + final_value=self.momentum_final, + total_iters=total_steps, + ) + self.momentum_scheduler.iter = curr_step + + def before_step(self): + # update parameters from schedulers + self.mask_size = self.mask_size_scheduler.step() + self.mask_ratio = self.mask_ratio_scheduler.step() + self.teacher_temp = self.teacher_temp_scheduler.step() + self.momentum = self.momentum_scheduler.step() + + if self.trainer.writer is not None: + self.trainer.writer.add_scalar( + "params/mask_size", + self.mask_size, + self.mask_size_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/mask_ratio", + self.mask_ratio, + self.mask_ratio_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/teacher_temp", + self.teacher_temp, + self.teacher_temp_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/momentum", + self.momentum, + self.momentum_scheduler.iter, + ) + + def after_step(self): + pass + + @staticmethod + def sinkhorn_knopp(feat, temp, num_iter=3): + feat = feat.float() + q = torch.exp(feat / temp).t() + n = sum(all_gather(q.shape[1])) # number of samples to assign + k = q.shape[0] # number of prototypes + + # make the matrix sums to 1 + sum_q = q.sum() + if get_world_size() > 1: + dist.all_reduce(sum_q) + q = q / sum_q + + for i in range(num_iter): + # normalize each row: total weight per prototype must be 1/k + q_row_sum = q.sum(dim=1, keepdim=True) + if get_world_size() > 1: + dist.all_reduce(q_row_sum) + q = q / q_row_sum / k + + # normalize each column: total weight per sample must be 1/n + q = q / q.sum(dim=0, keepdim=True) / n + + q *= n # the columns must sum to 1 so that Q is an assignment + return q.t() + + def generate_mask(self, coord, offset, grid_size): + batch = offset2batch(offset) + mask_size = self.mask_size * grid_size + mask_ratio = self.mask_ratio + + # Grouping points with grid patch + min_coord = torch_scatter.segment_coo(coord, batch, reduce="min") + grid_coord = ((coord - min_coord[batch]) // mask_size).int() + grid_coord = torch.cat([batch.unsqueeze(-1), grid_coord], dim=-1) + unique, point_cluster, counts = torch.unique( + grid_coord, dim=0, sorted=True, return_inverse=True, return_counts=True + ) + patch_num = unique.shape[0] + mask_patch_num = int(patch_num * mask_ratio) + patch_index = torch.randperm(patch_num, device=coord.device) + mask_patch_index = patch_index[:mask_patch_num] + point_mask = torch.isin(point_cluster, mask_patch_index) + return point_mask, point_cluster + + @torch.no_grad() + def match_neighbour( + self, + view1_coord, + view1_offset, + view2_coord, + view2_offset, + ): + index2, distance = pointops.knn_query( + 1, + view2_coord.float(), + view2_offset.int(), + view1_coord.float(), + view1_offset.int(), + ) + index1 = torch.arange( + index2.shape[0], device=index2.device, dtype=torch.long + ).unsqueeze(-1) + index = torch.cat([index1, index2], dim=-1)[ + distance.squeeze(-1) < self.match_max_r + ] + return index + + @torch.no_grad() + def roll_point(self, point): + n = self.num_global_view + # [pc1, pc1', pc2, pc2'] -> [pc1', pc1, pc2', pc2], only support num_global_view == 2 + bs = len(point.offset) // self.num_global_view + data_dict = {} + for key in point.keys(): + if key in ["feat", "coord", "origin_coord", "batch"]: + value = point[key].split(offset2bincount(point.offset).tolist()) + value = chain(*[value[n * b : n * (b + 1)][::-1] for b in range(bs)]) + if key == "batch": + value = [torch.ones_like(v) * i for i, v in enumerate(value)] + data_dict[key] = torch.cat(list(value), dim=0) + return Point(data_dict) + + def up_cast(self, point, upcast_level=None): + if upcast_level is None: + upcast_level = self.up_cast_level + else: + upcast_level = upcast_level + for _ in range(upcast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return point + + @staticmethod + def pool_corr(point, correspondence): + inverse_list = [] + idx_ptr_list = [] + point_feat = dict(offset=point.offset, feat=point.feat) + while "pooling_parent" in point.keys(): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + assert "idx_ptr" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + idx_ptr = point.pop("idx_ptr") + inverse_list.append(inverse) + idx_ptr_list.append(idx_ptr) + point = parent + inverse_list.reverse() + idx_ptr_list.reverse() + for inverse, idx_ptr in zip(inverse_list, idx_ptr_list): + _, indices = torch.sort(inverse) + img_num = correspondence.shape[1] + correspondence_all = [] + if img_num == 0: + correspondence_all = -torch.ones((idx_ptr.shape[0] - 1, 0, 2)).cuda() + else: + for img_id in range(img_num): + mask = torch.all( + correspondence[:, img_id] != torch.tensor([-1, -1]).cuda(), + dim=1, + ).float() + counts = torch_scatter.segment_csr( + mask[indices], idx_ptr, reduce="sum" + ) + counts[counts == 0] = 100000 + correspondence_img = deepcopy(correspondence[:, img_id]) + correspondence_img[correspondence_img == -1] = 0 + mask_sum = torch_scatter.segment_csr( + correspondence_img[indices], idx_ptr, reduce="sum" + ) + mask_sum = mask_sum / counts.unsqueeze(1) + mask_sum[counts == 100000] = -1 + correspondence_all.append(mask_sum) + correspondence_all = torch.stack(correspondence_all, dim=1) + correspondence = correspondence_all + point_feat["correspondence"] = correspondence + point_feat = Point(point_feat) + return point_feat + + def forward(self, data_dict, return_point=False): + if return_point: + point = self.teacher.backbone(data_dict) + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return dict(point=point) + + # prepare global_point, mask_global_point, local_point + with torch.no_grad(): + # global_point & masking + global_point = Point( + feat=data_dict["global_feat_full"], + coord=data_dict["global_coord"], + origin_coord=data_dict["global_origin_coord"], + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + global_mask, global_cluster = self.generate_mask( + global_point.coord, global_point.offset, global_point.grid_size + ) + mask_global_coord = global_point.coord.clone().detach() + if self.mask_jitter is not None: + mask_global_coord[global_mask] += torch.clip( + torch.randn_like(mask_global_coord[global_mask]).mul( + self.mask_jitter * data_dict["grid_size"][0] + ), + max=(self.mask_jitter * data_dict["grid_size"][0]) * 2, + ) + + mask_global_point = Point( + feat=data_dict["global_feat"], + coord=mask_global_coord, + origin_coord=data_dict["global_origin_coord"], + mask=global_mask, + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + major_view_correspondence = data_dict["global_correspondence"] + + # local point & matching + local_point = Point( + feat=data_dict["local_feat"], + coord=data_dict["local_coord"], + origin_coord=data_dict["local_origin_coord"], + offset=data_dict["local_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # create result dictionary for return + result_dict = dict(loss=[]) + # teacher backbone forward (shared with mask and unmask) + global_point_ = self.teacher.backbone(global_point) + global_point_ = self.up_cast(global_point_) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + global_point_.feat = self.teacher.mask_head(global_point_.feat) + else: + global_point_.feat = self.teacher.unmask_head(global_point_.feat) + + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + # student forward + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_pred_sim = self.student.mask_head(mask_global_point_.feat) + + if self.mask_loss_weight > 0: + with torch.no_grad(): + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + global_point_.origin_coord, + global_point_.offset, + ) + # teacher forward + mask_target_sim = self.sinkhorn_knopp( + global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + # loss + mask_loss = -torch.sum( + mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + mask_loss = torch_scatter.segment_coo( + mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["mask_loss"] = mask_loss + result_dict["loss"].append(mask_loss * self.mask_loss_weight) + + if self.roll_mask_loss_weight > 0: + roll_global_point_ = self.roll_point(global_point_) + with torch.no_grad(): + # match index for pred and roll target + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + roll_global_point_.origin_coord, + roll_global_point_.offset, + ) + # teacher forward + roll_mask_target_sim = self.sinkhorn_knopp( + roll_global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + roll_mask_loss = -torch.sum( + roll_mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + roll_mask_loss = torch_scatter.segment_coo( + roll_mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["roll_mask_loss"] = roll_mask_loss + result_dict["loss"].append(roll_mask_loss * self.roll_mask_loss_weight) + if self.unmask_loss_weight > 0: + # student forward + local_point_ = self.student.backbone(local_point) + local_point_ = self.up_cast(local_point_) + unmask_pred_sim = self.student.unmask_head(local_point_.feat) + with torch.no_grad(): + principal_view_mask = global_point_.batch % self.num_global_view == 0 + principal_view_batch = ( + global_point_.batch[principal_view_mask] // self.num_global_view + ) + match_index = self.match_neighbour( + local_point_.origin_coord, + local_point_.offset[self.num_local_view - 1 :: self.num_local_view], + global_point_.origin_coord[principal_view_mask], + batch2offset(principal_view_batch), + ) + # teacher forward + unmask_target_sim = self.sinkhorn_knopp( + global_point_.feat[principal_view_mask][match_index[:, 1]], + self.teacher_temp, + ) + # loss + unmask_loss = -torch.sum( + unmask_target_sim + * F.log_softmax( + unmask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + unmask_loss = torch_scatter.segment_coo( + unmask_loss, + index=local_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["unmask_loss"] = unmask_loss + result_dict["loss"].append(unmask_loss * self.unmask_loss_weight) + if self.enc2d_loss_weight > 0: + if self.mask_loss_weight == 0 or self.roll_mask_loss_weight == 0: + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_global_point_enc2d = self.up_cast( + mask_global_point_, + upcast_level=self.enc2d_upcast_level - self.up_cast_level, + ) + to_feature = self.pool_corr( + mask_global_point_enc2d, major_view_correspondence + ) + data_dict_global_offset = torch.cat( + [torch.tensor([0]).cuda(), to_feature["offset"]], dim=0 + ) + enc2d_count = ( + data_dict_global_offset[ + 1 : len(data_dict_global_offset) : self.num_global_view + ] + - data_dict_global_offset[ + 0 : len(data_dict_global_offset) - 1 : self.num_global_view + ] + ) + enc2d_offset = torch.cat( + [torch.tensor([0]).cuda(), torch.cumsum(enc2d_count, dim=0)] + ) + enc2d_mask = torch.cat( + [ + torch.arange(0, c, device=enc2d_count.device) + + data_dict_global_offset[i * self.num_global_view] + for i, c in enumerate(enc2d_count) + ], + dim=0, + ) + + offset_points_3d = enc2d_offset[1:] + batch_points_3d = offset2batch(offset_points_3d) + imgs = data_dict["images"] + feature3d = to_feature["feat"][enc2d_mask] + enc2d_global_mask = global_mask[enc2d_mask] + correspondence = to_feature["correspondence"][enc2d_mask] + v0 = correspondence.shape[1] + mask = torch.any(correspondence != torch.tensor([-1, -1]).cuda(), dim=2) + enc2d_global_mask = enc2d_global_mask.unsqueeze(1).expand(-1, v0) + valid_index = torch.where(mask) # 0: 3d points index, 1: view index + + bincount_img_num = data_dict["img_num"] + offset_img_num = bincount2offset(bincount_img_num) + total_img_num = offset_img_num[-1] + + if total_img_num > 0: + # expand + with torch.no_grad(): + feature2d = self.ENC2D_forward(imgs) + feature2d = feature2d.contiguous().view(-1, feature2d.shape[-1]) + feature2d_mask = feature2d + + offset_img_num = torch.cat([torch.tensor([0]).cuda(), offset_img_num])[ + :-1 + ] + batch_index = batch_points_3d[valid_index[0]] + batch_img_num = offset_img_num[batch_index] + + feature3d_mask = feature3d[valid_index[0]] + + feature_index = torch.cat( + [ + batch_img_num.unsqueeze(-1), + valid_index[1].unsqueeze(-1), + correspondence[valid_index], + ], + dim=-1, + ) + feature_index = feature_index.long() + feature_index = ( + feature_index[:, 0] * self.patch_h * self.patch_w + + feature_index[:, 1] * self.patch_h * self.patch_w + + feature_index[:, 2] * self.patch_w + + feature_index[:, 3] + ) + + feature_index = feature_index.long() + feature3d_mask = torch_scatter.scatter_mean( + feature3d_mask, feature_index, dim=0, dim_size=feature2d.shape[0] + ) + feature3d_mask = self.patch_proj(feature3d_mask) + feature_index = torch.unique(feature_index) + feature2d_mask = feature2d_mask[feature_index] + feature3d_mask = feature3d_mask[feature_index] + + if self.enc2d_cos_shift: + feature2d_mask = feature2d_mask - feature2d_mask.mean( + dim=-1, keepdim=True + ) + feature3d_mask = feature3d_mask - feature3d_mask.mean( + dim=-1, keepdim=True + ) + cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6) + loss = (1 - cos(feature2d_mask, feature3d_mask)).mean() * 10 + + result_dict["enc2d_loss"] = loss + result_dict["loss"].append(loss * self.enc2d_loss_weight) + del ( + feature2d, + feature3d, + feature2d_mask, + feature3d_mask, + correspondence, + feature_index, + ) + elif ( + self.mask_loss_weight + + self.unmask_loss_weight + + self.roll_mask_loss_weight + > 0 + ): + result_ssl_loss = sum(result_dict["loss"]) / ( + self.mask_loss_weight + + self.unmask_loss_weight + + self.roll_mask_loss_weight + ) + result_dict["enc2d_loss"] = result_ssl_loss + result_dict["loss"].append(result_ssl_loss * self.enc2d_loss_weight) + # torch.cuda.empty_cache() + result_dict["loss"] = sum(result_dict["loss"]) + + if get_world_size() > 1: + for loss in result_dict.values(): + dist.all_reduce(loss, op=dist.ReduceOp.AVG) + return result_dict diff --git a/pointcept/models/context_aware_classifier/__init__.py b/pointcept/models/context_aware_classifier/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..8327900bbf92258fa242f4b99d5b235bc1ae44aa --- /dev/null +++ b/pointcept/models/context_aware_classifier/__init__.py @@ -0,0 +1 @@ +from .context_aware_classifier_v1m1_base import CACSegmentor diff --git a/pointcept/models/context_aware_classifier/context_aware_classifier_v1m1_base.py b/pointcept/models/context_aware_classifier/context_aware_classifier_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..893f4c8fc6d68a1fb7dc24e1ceb3b3ace11ebec6 --- /dev/null +++ b/pointcept/models/context_aware_classifier/context_aware_classifier_v1m1_base.py @@ -0,0 +1,275 @@ +""" +Context-aware Classifier for Semantic Segmentation + +Author: Zhuotao Tian, Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +import torch.nn as nn +import torch.nn.functional as F +from pointcept.models.losses import build_criteria +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS, build_model + + +@MODELS.register_module("CAC-v1m1") +class CACSegmentor(nn.Module): + def __init__( + self, + num_classes, + backbone_out_channels, + backbone=None, + criteria=None, + cos_temp=15, + main_weight=1, + pre_weight=1, + pre_self_weight=1, + kl_weight=1, + conf_thresh=0, + detach_pre_logits=False, + ): + super().__init__() + self.num_classes = num_classes + self.cos_temp = cos_temp + self.main_weight = main_weight + self.pre_weight = pre_weight + self.pre_self_weight = pre_self_weight + self.kl_weight = kl_weight + self.conf_thresh = conf_thresh + self.detach_pre_logits = detach_pre_logits + + # backbone + self.backbone = build_model(backbone) + # heads + self.seg_head = nn.Linear(backbone_out_channels, num_classes) + self.proj = nn.Sequential( + nn.Linear(backbone_out_channels * 2, backbone_out_channels * 2, bias=False), + nn.ReLU(inplace=True), + nn.Linear(backbone_out_channels * 2, backbone_out_channels), + ) + self.apd_proj = nn.Sequential( + nn.Linear(backbone_out_channels * 2, backbone_out_channels * 2, bias=False), + nn.ReLU(inplace=True), + nn.Linear(backbone_out_channels * 2, backbone_out_channels), + ) + self.feat_proj_layer = nn.Sequential( + nn.Linear(backbone_out_channels, backbone_out_channels, bias=False), + nn.BatchNorm1d(backbone_out_channels), + nn.ReLU(inplace=True), + nn.Linear(backbone_out_channels, backbone_out_channels), + ) + # Criteria + self.criteria = build_criteria(criteria) + + @staticmethod + def get_pred(x, proto): + # x: [n,c]; proto: [cls, c] + x = F.normalize(x, 2, 1) + proto = F.normalize(proto, 2, 1) + pred = x @ proto.permute(1, 0) # [n,c] x [c, cls] -> [n, cls] + return pred + + def get_adaptive_perspective(self, feat, target, new_proto, proto): + raw_feat = feat.clone() + # target: [n] + # feat: [n,c] + # proto: [cls, c] + unique_y = list(target.unique()) + if -1 in unique_y: + unique_y.remove(-1) + target = target.unsqueeze(-1) # [n, 1] + + for tmp_y in unique_y: + tmp_mask = (target == tmp_y).float() + tmp_proto = (feat * tmp_mask).sum(0) / (tmp_mask.sum(0) + 1e-4) # c + onehot_vec = torch.zeros(new_proto.shape[0], 1).cuda() # cls, 1 + onehot_vec[tmp_y.long()] = 1 + new_proto = ( + new_proto * (1 - onehot_vec) + tmp_proto.unsqueeze(0) * onehot_vec + ) + + new_proto = torch.cat([new_proto, proto], -1) + new_proto = self.apd_proj(new_proto) + raw_feat = self.feat_proj_layer(raw_feat) + pred = self.get_pred(raw_feat, new_proto) + return pred + + def post_refine_proto_batch(self, feat, pred, proto, offset=None): + # x: [n, c]; pred: [n, cls]; proto: [cls, c] + pred_list = [] + x = feat + raw_x = x.clone() + if self.detach_pre_logits: + pred = pred.detach() + raw_pred = pred.clone() + + if offset is None: + raw_x = x.clone() + n, n_cls = pred.shape[:] + pred = pred.view(n, n_cls) + pred = F.softmax(pred, 1).permute(1, 0) # [n, cls] -> [cls, n] + if self.conf_thresh > 0: + max_pred = ( + (pred.max(0)[0] >= self.conf_thresh).float().unsqueeze(0) + ) # 1, n + pred = pred * max_pred + pred_proto = (pred / (pred.sum(-1).unsqueeze(-1) + 1e-7)) @ raw_x # cls, c + + pred_proto = torch.cat([pred_proto, proto], -1) # cls, 2c + pred_proto = self.proj(pred_proto) + raw_x = self.feat_proj_layer(raw_x) + new_pred = self.get_pred(raw_x, pred_proto) + else: + for i in range(len(offset)): + if i == 0: + start = 0 + end = offset[i] + else: + start, end = offset[i - 1], offset[i] + tmp_x = raw_x[start:end] + pred = raw_pred[start:end] + n, n_cls = pred.shape[:] + pred = pred.view(n, n_cls) + pred = F.softmax(pred, 1).permute(1, 0) # [n, cls] -> [cls, n] + if self.conf_thresh > 0: + max_pred = ( + (pred.max(0)[0] >= self.conf_thresh).float().unsqueeze(0) + ) # 1, n + pred = pred * max_pred + pred_proto = ( + pred / (pred.sum(-1).unsqueeze(-1) + 1e-7) + ) @ tmp_x # cls, c + + pred_proto = torch.cat([pred_proto, proto], -1) # cls, 2c + pred_proto = self.proj(pred_proto) + tmp_x = self.feat_proj_layer(tmp_x) + new_pred = self.get_pred(tmp_x, pred_proto) + pred_list.append(new_pred) + new_pred = torch.cat(pred_list, 0) + return new_pred + + @staticmethod + def get_distill_loss(pred, soft, target, smoothness=0.5, eps=0): + """ + knowledge distillation loss + """ + n, c = soft.shape[:] + soft = soft.detach() + target = target.unsqueeze(-1) # n, 1 + onehot = target.view(-1, 1) # n, 1 + ignore_mask = (onehot == -1).float() + sm_soft = F.softmax(soft / 1, 1) # n, c + + onehot = onehot * (1 - ignore_mask) + onehot = torch.zeros(n, c).cuda().scatter_(1, onehot.long(), 1) # n, c + smoothed_label = smoothness * sm_soft + (1 - smoothness) * onehot + if eps > 0: + smoothed_label = smoothed_label * (1 - eps) + (1 - smoothed_label) * eps / ( + smoothed_label.shape[1] - 1 + ) + + loss = torch.mul(-1 * F.log_softmax(pred, dim=1), smoothed_label) # b, n, h, w + loss = loss.sum(1) + + sm_soft = F.softmax(soft / 1, 1) # n, c + entropy_mask = -1 * (sm_soft * torch.log(sm_soft + 1e-4)).sum(1) + + # for class-wise entropy estimation + target = target.squeeze(-1) + unique_classes = list(target.unique()) + if -1 in unique_classes: + unique_classes.remove(-1) + valid_mask = (target != -1).float() + entropy_mask = entropy_mask * valid_mask + loss_list = [] + weight_list = [] + for tmp_y in unique_classes: + tmp_mask = (target == tmp_y).float().squeeze() + tmp_entropy_mask = entropy_mask * tmp_mask + class_weight = 1 + tmp_loss = (loss * tmp_entropy_mask).sum() / (tmp_entropy_mask.sum() + 1e-4) + loss_list.append(class_weight * tmp_loss) + weight_list.append(class_weight) + + if len(weight_list) > 0: + loss = sum(loss_list) / (sum(weight_list) + 1e-4) + else: + loss = torch.zeros(1).cuda().mean() + return loss + + def forward(self, data_dict): + offset = data_dict["offset"] + point = self.backbone(data_dict) + if isinstance(point, Point): + feat = point.feat + else: + feat = point + seg_logits = self.seg_head(feat) + + if self.training: + target = data_dict["segment"] + pre_logits = seg_logits.clone() + refine_logits = ( + self.post_refine_proto_batch( + feat=feat, + pred=seg_logits, + proto=self.seg_head.weight.squeeze(), + offset=offset, + ) + * self.cos_temp + ) + + cac_pred = ( + self.get_adaptive_perspective( + feat=feat, + target=target, + new_proto=self.seg_head.weight.detach().data.squeeze(), + proto=self.seg_head.weight.squeeze(), + ) + * self.cos_temp + ) + + seg_loss = self.criteria(refine_logits, target) * self.main_weight + pre_loss = self.criteria(cac_pred, target) * self.pre_weight + pre_self_loss = self.criteria(pre_logits, target) * self.pre_self_weight + kl_loss = ( + self.get_distill_loss( + pred=refine_logits, soft=cac_pred.detach(), target=target + ) + * self.kl_weight + ) + loss = seg_loss + pre_loss + pre_self_loss + kl_loss + return dict( + loss=loss, + seg_loss=seg_loss, + pre_loss=pre_loss, + pre_self_loss=pre_self_loss, + kl_loss=kl_loss, + ) + + elif "segment" in data_dict.keys(): + refine_logits = ( + self.post_refine_proto_batch( + feat=feat, + pred=seg_logits, + proto=self.seg_head.weight.squeeze(), + offset=offset, + ) + * self.cos_temp + ) + + loss = self.criteria(seg_logits, data_dict["segment"]) + return dict(loss=loss, seg_logits=refine_logits) + + else: + refine_logits = ( + self.post_refine_proto_batch( + feat=feat, + pred=seg_logits, + proto=self.seg_head.weight.squeeze(), + offset=offset, + ) + * self.cos_temp + ) + return dict(seg_logits=refine_logits) diff --git a/pointcept/models/default.py b/pointcept/models/default.py new file mode 100644 index 0000000000000000000000000000000000000000..a04d1c427ff3446c91b609ee8d57fa73d66345ad --- /dev/null +++ b/pointcept/models/default.py @@ -0,0 +1,348 @@ +import torch +import torch.nn as nn +import torch_scatter +import torch_cluster +from peft import LoraConfig, get_peft_model +from collections import OrderedDict + +from pointcept.models.losses import build_criteria +from pointcept.models.utils.structure import Point +from pointcept.models.utils import offset2batch +from .builder import MODELS, build_model + + +@MODELS.register_module() +class DefaultSegmentor(nn.Module): + def __init__(self, backbone=None, criteria=None): + super().__init__() + self.backbone = build_model(backbone) + self.criteria = build_criteria(criteria) + + def forward(self, input_dict): + if "condition" in input_dict.keys(): + # PPT (https://arxiv.org/abs/2308.09718) + # currently, only support one batch one condition + input_dict["condition"] = input_dict["condition"][0] + seg_logits = self.backbone(input_dict) + # train + if self.training: + loss = self.criteria(seg_logits, input_dict["segment"]) + return dict(loss=loss) + # eval + elif "segment" in input_dict.keys(): + loss = self.criteria(seg_logits, input_dict["segment"]) + return dict(loss=loss, seg_logits=seg_logits) + # test + else: + return dict(seg_logits=seg_logits) + + +@MODELS.register_module() +class DefaultSegmentorV2(nn.Module): + def __init__( + self, + num_classes, + backbone_out_channels, + backbone=None, + criteria=None, + freeze_backbone=False, + ): + super().__init__() + self.seg_head = ( + nn.Linear(backbone_out_channels, num_classes) + if num_classes > 0 + else nn.Identity() + ) + self.backbone = build_model(backbone) + self.criteria = build_criteria(criteria) + self.freeze_backbone = freeze_backbone + if self.freeze_backbone: + for p in self.backbone.parameters(): + p.requires_grad = False + + def forward(self, input_dict, return_point=False): + point = Point(input_dict) + point = self.backbone(point) + # Backbone added after v1.5.0 return Point instead of feat and use DefaultSegmentorV2 + # TODO: remove this part after make all backbone return Point only. + if isinstance(point, Point): + while "pooling_parent" in point.keys(): + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + feat = point.feat + else: + feat = point + seg_logits = self.seg_head(feat) + return_dict = dict() + if return_point: + # PCA evaluator parse feat and coord in point + return_dict["point"] = point + # train + if self.training: + loss = self.criteria(seg_logits, input_dict["segment"]) + return_dict["loss"] = loss + # eval + elif "segment" in input_dict.keys(): + loss = self.criteria(seg_logits, input_dict["segment"]) + return_dict["loss"] = loss + return_dict["seg_logits"] = seg_logits + # test + else: + return_dict["seg_logits"] = seg_logits + return return_dict + + +@MODELS.register_module() +class DefaultLORASegmentorV2(nn.Module): + def __init__( + self, + num_classes, + backbone_out_channels, + backbone=None, + criteria=None, + freeze_backbone=False, + use_lora=False, + lora_r=8, + lora_alpha=16, + lora_dropout=0.1, + backbone_path=None, + keywords=None, + replacements=None, + ): + super().__init__() + self.seg_head = ( + nn.Linear(backbone_out_channels, num_classes) + if num_classes > 0 + else nn.Identity() + ) + self.keywords = keywords + self.replacements = replacements + self.backbone = build_model(backbone) + backbone_weight = torch.load( + backbone_path, + map_location=lambda storage, loc: storage.cuda(), + ) + self.backbone_load(backbone_weight) + + self.criteria = build_criteria(criteria) + self.freeze_backbone = freeze_backbone + self.use_lora = use_lora + + if self.use_lora: + lora_config = LoraConfig( + r=lora_r, + lora_alpha=lora_alpha, + target_modules=["qkv"], + # target_modules=["query", "value"], + lora_dropout=lora_dropout, + bias="none", + ) + self.backbone.enc = get_peft_model(self.backbone.enc, lora_config) + + if self.freeze_backbone: + for p in self.backbone.parameters(): + p.requires_grad = False + if self.use_lora: + for name, param in self.backbone.named_parameters(): + if "lora_" in name: + param.requires_grad = True + self.backbone.enc.print_trainable_parameters() + + def backbone_load(self, checkpoint): + weight = OrderedDict() + for key, value in checkpoint["state_dict"].items(): + if not key.startswith("module."): + key = "module." + key # xxx.xxx -> module.xxx.xxx + # Now all keys contain "module." no matter DDP or not. + if self.keywords in key: + key = key.replace(self.keywords, self.replacements) + key = key[7:] # module.xxx.xxx -> xxx.xxx + if key.startswith("backbone."): + key = key[9:] + weight[key] = value + load_state_info = self.backbone.load_state_dict(weight, strict=False) + print(f"Missing keys: {load_state_info[0]}") + print(f"Unexpected keys: {load_state_info[1]}") + + def forward(self, input_dict, return_point=False): + point = Point(input_dict) + if self.freeze_backbone and not self.use_lora: + with torch.no_grad(): + point = self.backbone(point) + else: + point = self.backbone(point) + + if isinstance(point, Point): + while "pooling_parent" in point.keys(): + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + feat = point.feat + else: + feat = point + + seg_logits = self.seg_head(feat) + return_dict = dict() + if return_point: + return_dict["point"] = point + + if self.training: + loss = self.criteria(seg_logits, input_dict["segment"]) + return_dict["loss"] = loss + elif "segment" in input_dict.keys(): + loss = self.criteria(seg_logits, input_dict["segment"]) + return_dict["loss"] = loss + return_dict["seg_logits"] = seg_logits + else: + return_dict["seg_logits"] = seg_logits + return return_dict + + +@MODELS.register_module() +class DINOEnhancedSegmentor(nn.Module): + def __init__( + self, + num_classes, + backbone_out_channels, + backbone=None, + criteria=None, + freeze_backbone=False, + ): + super().__init__() + self.seg_head = ( + nn.Linear(backbone_out_channels, num_classes) + if num_classes > 0 + else nn.Identity() + ) + self.backbone = build_model(backbone) if backbone is not None else None + self.criteria = build_criteria(criteria) + self.freeze_backbone = freeze_backbone + if self.backbone is not None and self.freeze_backbone: + for p in self.backbone.parameters(): + p.requires_grad = False + + def forward(self, input_dict, return_point=False): + point = Point(input_dict) + if self.backbone is not None: + if self.freeze_backbone: + with torch.no_grad(): + point = self.backbone(point) + else: + point = self.backbone(point) + point_list = [point] + while "unpooling_parent" in point_list[-1].keys(): + point_list.append(point_list[-1].pop("unpooling_parent")) + for i in reversed(range(1, len(point_list))): + point = point_list[i] + parent = point_list[i - 1] + assert "pooling_inverse" in point.keys() + inverse = point.pooling_inverse + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = point_list[0] + while "pooling_parent" in point.keys(): + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pooling_inverse + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + feat = [point.feat] + else: + feat = [] + dino_coord = input_dict["dino_coord"] + dino_feat = input_dict["dino_feat"] + dino_offset = input_dict["dino_offset"] + idx = torch_cluster.knn( + x=dino_coord, + y=point.origin_coord, + batch_x=offset2batch(dino_offset), + batch_y=offset2batch(point.origin_offset), + k=1, + )[1] + + feat.append(dino_feat[idx]) + feat = torch.concatenate(feat, dim=-1) + seg_logits = self.seg_head(feat) + return_dict = dict() + if return_point: + # PCA evaluator parse feat and coord in point + return_dict["point"] = point + # train + if self.training: + loss = self.criteria(seg_logits, input_dict["origin_segment"]) + return_dict["loss"] = loss + # eval + elif "segment" in input_dict.keys(): + loss = self.criteria(seg_logits, input_dict["origin_segment"]) + return_dict["loss"] = loss + return_dict["seg_logits"] = seg_logits + # test + else: + return_dict["seg_logits"] = seg_logits + return return_dict + + +@MODELS.register_module() +class DefaultClassifier(nn.Module): + def __init__( + self, + backbone=None, + criteria=None, + num_classes=40, + backbone_embed_dim=256, + freeze_backbone=False, + ): + super().__init__() + self.backbone = build_model(backbone) + self.criteria = build_criteria(criteria) + self.num_classes = num_classes + self.backbone_embed_dim = backbone_embed_dim + self.freeze_backbone = freeze_backbone + self.cls_head = nn.Sequential( + nn.Linear(backbone_embed_dim, 256), + nn.BatchNorm1d(256), + nn.ReLU(inplace=True), + nn.Dropout(p=0.5), + nn.Linear(256, 128), + nn.BatchNorm1d(128), + nn.ReLU(inplace=True), + nn.Dropout(p=0.5), + nn.Linear(128, num_classes), + ) + if self.freeze_backbone: + for p in self.backbone.parameters(): + p.requires_grad = False + self.backbone.eval() + + def forward(self, input_dict): + point = Point(input_dict) + if self.freeze_backbone: + with torch.no_grad(): + point = self.backbone(point) + else: + point = self.backbone(point) + # Backbone added after v1.5.0 return Point instead of feat + # And after v1.5.0 feature aggregation for classification operated in classifier + # TODO: remove this part after make all backbone return Point only. + if isinstance(point, Point): + point.feat = torch_scatter.segment_csr( + src=point.feat, + indptr=nn.functional.pad(point.offset, (1, 0)), + reduce="mean", + ) + feat = point.feat + else: + feat = point + cls_logits = self.cls_head(feat) + if self.training: + loss = self.criteria(cls_logits, input_dict["category"]) + return dict(loss=loss) + elif "category" in input_dict.keys(): + loss = self.criteria(cls_logits, input_dict["category"]) + return dict(loss=loss, cls_logits=cls_logits) + else: + return dict(cls_logits=cls_logits) diff --git a/pointcept/models/litept/__init__.py b/pointcept/models/litept/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..be3a4f8826d9ca86f60b25a1ed7f4d2c82cdce47 --- /dev/null +++ b/pointcept/models/litept/__init__.py @@ -0,0 +1 @@ +from .litept_v1 import * diff --git a/pointcept/models/litept/litept_v1.py b/pointcept/models/litept/litept_v1.py new file mode 100644 index 0000000000000000000000000000000000000000..4f1e63c8a313901e47f377527411d071193b275e --- /dev/null +++ b/pointcept/models/litept/litept_v1.py @@ -0,0 +1,778 @@ +""" +LitePT model + +Author: Yuanwen Yue (yuayue@ethz.ch) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from addict import Dict +import torch +import torch.nn as nn +import spconv.pytorch as spconv +import torch_scatter +from timm.layers import DropPath + +try: + import flash_attn +except ImportError: + flash_attn = None + +from pointcept.models.builder import MODELS +from pointcept.models.utils.misc import offset2bincount +from pointcept.models.utils.structure import Point +from pointcept.models.modules import PointModule, PointSequential + +try: + import pointrope as _kernels + + class PointROPE_func(torch.autograd.Function): + + @staticmethod + def forward(ctx, tokens, positions, base, F0=1): + ctx.save_for_backward(positions) + ctx.saved_base = base + ctx.saved_F0 = F0 + # tokens = tokens.clone() # uncomment this if inplace doesn't work + _kernels.pointrope(tokens, positions, base, F0) + ctx.mark_dirty(tokens) + return tokens + + @staticmethod + def backward(ctx, grad_res): + positions, base, F0 = ctx.saved_tensors[0], ctx.saved_base, ctx.saved_F0 + _kernels.pointrope(grad_res, positions, base, -F0) + ctx.mark_dirty(grad_res) + return grad_res, None, None, None + + class PointROPE(torch.nn.Module): + def __init__(self, freq=100.0, F0=1.0): + super().__init__() + self.base = freq + self.F0 = F0 + + def forward(self, tokens, positions): + tokens = tokens.transpose(1, 2).contiguous() + positions = positions.contiguous() + tokens = PointROPE_func.apply(tokens, positions, self.base, self.F0) + return tokens.transpose(1, 2).contiguous() + +except Exception as e: + print( + f"[PointROPE] CUDA implementation unavailable ({type(e).__name__}: {e}). " + "Using slower Pytorch fallback." + ) + + class PointROPE(torch.nn.Module): + + def __init__(self, freq=100.0, F0=1.0): + super().__init__() + self.base = freq + self.F0 = F0 + self.cache = {} + + def get_cos_sin(self, D, seq_len, device, dtype): + if (D, seq_len, device, dtype) not in self.cache: + inv_freq = self.F0 / ( + self.base ** (torch.arange(0, D, 2).float().to(device) / D) + ) + t = torch.arange(seq_len, device=device, dtype=inv_freq.dtype) + freqs = torch.einsum("i,j->ij", t, inv_freq).to(dtype) + freqs = torch.cat((freqs, freqs), dim=-1) + cos = freqs.cos() + sin = freqs.sin() + self.cache[D, seq_len, device, dtype] = (cos, sin) + return self.cache[D, seq_len, device, dtype] + + @staticmethod + def rotate_half(x): + x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :] + return torch.cat((-x2, x1), dim=-1) + + def apply_rope1d(self, tokens, pos1d, cos, sin): + assert pos1d.ndim == 2 + cos = torch.nn.functional.embedding(pos1d, cos)[:, None, :, :] + sin = torch.nn.functional.embedding(pos1d, sin)[:, None, :, :] + return (tokens * cos) + (self.rotate_half(tokens) * sin) + + def forward(self, tokens, positions, max_seqlen=None): + """ + input: + * tokens: batch_size x nheads x ntokens x dim + * positions: batch_size x ntokens x 3 (xyz position of each token) + output: + * tokens after appplying PointROPE (batch_size x nheads x ntokens x dim) + """ + assert ( + tokens.size(3) % 3 == 0 + ), "number of dimensions should be a multiple of three" + D = tokens.size(3) // 3 + assert positions.ndim == 3 and positions.shape[-1] == 3 # Batch, Seq, 3 + if max_seqlen == None: + cos, sin = self.get_cos_sin( + D, int(positions.max()) + 1, tokens.device, tokens.dtype + ) + else: # use dynamic sequence length according to batched input + cos, sin = self.get_cos_sin( + D, max_seqlen + 1, tokens.device, tokens.dtype + ) + # split features into three parts along the feature dimension, and apply rope1d on each subspace + x, y, z = tokens.chunk(3, dim=-1) + x = self.apply_rope1d(x, positions[:, :, 0], cos, sin) + y = self.apply_rope1d(y, positions[:, :, 1], cos, sin) + z = self.apply_rope1d(z, positions[:, :, 2], cos, sin) + tokens = torch.cat((x, y, z), dim=-1) + return tokens + + +class PointROPEAttention(PointModule): + def __init__( + self, + channels, + num_heads, + patch_size, + rope_freq, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + order_index=0, + ): + super().__init__() + assert channels % num_heads == 0 + self.channels = channels + self.num_heads = num_heads + self.scale = qk_scale or (channels // num_heads) ** -0.5 + self.order_index = order_index + + self.patch_size = patch_size + self.attn_drop = attn_drop + + self.qkv = torch.nn.Linear(channels, channels * 3, bias=qkv_bias) + self.proj = torch.nn.Linear(channels, channels) + self.proj_drop = torch.nn.Dropout(proj_drop) + self.softmax = torch.nn.Softmax(dim=-1) + + # pointrope + self.rope = PointROPE(freq=rope_freq) + + @torch.no_grad() + def get_padding_and_inverse(self, point): + pad_key = "pad" + unpad_key = "unpad" + cu_seqlens_key = "cu_seqlens_key" + if ( + pad_key not in point.keys() + or unpad_key not in point.keys() + or cu_seqlens_key not in point.keys() + ): + offset = point.offset + bincount = offset2bincount(offset) + bincount_pad = ( + torch.div( + bincount + self.patch_size - 1, + self.patch_size, + rounding_mode="trunc", + ) + * self.patch_size + ) + # only pad point when num of points larger than patch_size + mask_pad = bincount > self.patch_size + bincount_pad = ~mask_pad * bincount + mask_pad * bincount_pad + _offset = nn.functional.pad(offset, (1, 0)) + _offset_pad = nn.functional.pad(torch.cumsum(bincount_pad, dim=0), (1, 0)) + pad = torch.arange(_offset_pad[-1], device=offset.device) + unpad = torch.arange(_offset[-1], device=offset.device) + cu_seqlens = [] + for i in range(len(offset)): + unpad[_offset[i] : _offset[i + 1]] += _offset_pad[i] - _offset[i] + if bincount[i] != bincount_pad[i]: + pad[ + _offset_pad[i + 1] + - self.patch_size + + (bincount[i] % self.patch_size) : _offset_pad[i + 1] + ] = pad[ + _offset_pad[i + 1] + - 2 * self.patch_size + + (bincount[i] % self.patch_size) : _offset_pad[i + 1] + - self.patch_size + ] + pad[_offset_pad[i] : _offset_pad[i + 1]] -= _offset_pad[i] - _offset[i] + cu_seqlens.append( + torch.arange( + _offset_pad[i], + _offset_pad[i + 1], + step=self.patch_size, + dtype=torch.int32, + device=offset.device, + ) + ) + point[pad_key] = pad + point[unpad_key] = unpad + point[cu_seqlens_key] = nn.functional.pad( + torch.concat(cu_seqlens), (0, 1), value=_offset_pad[-1] + ) + return point[pad_key], point[unpad_key], point[cu_seqlens_key] + + def forward(self, point): + + H = self.num_heads + K = self.patch_size + C = self.channels + + pad, unpad, cu_seqlens = self.get_padding_and_inverse(point) + + order = point.serialized_order[self.order_index][pad] + inverse = unpad[point.serialized_inverse[self.order_index]] + + # padding and reshape feat and batch for serialized point patch + qkv = self.qkv(point.feat)[order] # [N, C] + + ## apply pointrope + pos = point.grid_coord[order] # [N, 3] + pos = pos.reshape(-1, 3).unsqueeze(0) + + q, k, v = qkv.half().chunk(3, dim=-1) + q = q.reshape(-1, H, C // H).transpose(0, 1)[None] # [1, H, N, head_dim] + k = k.reshape(-1, H, C // H).transpose(0, 1)[None] # [1, H, N, head_dim] + + # workround to make pointrope cuda float32 happy + q = self.rope(q.float(), pos).to(q.dtype) # [1, H, N, head_dim] + k = self.rope(k.float(), pos).to(k.dtype) # [1, H, N, head_dim] + + # assemble input for flash attention + qkv_rotated = torch.stack( + [ + q.squeeze(0).transpose(0, 1), + k.squeeze(0).transpose(0, 1), + v.reshape(-1, H, C // H), + ], + dim=1, + ) # [N, 3, H, head_dim] + + if flash_attn is None: + raise ImportError( + "LitePT forward requires flash_attn, but it is not installed. " + "Please install flash_attn to run LitePT attention." + ) + + feat = flash_attn.flash_attn_varlen_qkvpacked_func( + qkv_rotated, + cu_seqlens, + max_seqlen=self.patch_size, + dropout_p=self.attn_drop if self.training else 0, + softmax_scale=self.scale, + ).reshape(-1, C) + + feat = feat.to(qkv.dtype) + feat = feat[inverse] + + # ffn + feat = self.proj(feat) + feat = self.proj_drop(feat) + point.feat = feat + return point + + +class MLP(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=None, + out_channels=None, + act_layer=nn.GELU, + drop=0.0, + ): + super().__init__() + out_channels = out_channels or in_channels + hidden_channels = hidden_channels or in_channels + self.fc1 = nn.Linear(in_channels, hidden_channels) + self.act = act_layer() + self.fc2 = nn.Linear(hidden_channels, out_channels) + self.drop = nn.Dropout(drop) + + def forward(self, x): + x = self.fc1(x) + x = self.act(x) + x = self.drop(x) + x = self.fc2(x) + x = self.drop(x) + return x + + +class Block(PointModule): + def __init__( + self, + channels, + num_heads, + patch_size=48, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.0, + norm_layer=nn.LayerNorm, + act_layer=nn.GELU, + pre_norm=True, + order_index=0, + cpe_indice_key=None, + enable_conv=True, + enable_attn=True, + rope_freq=100.0, + ): + super().__init__() + self.channels = channels + self.pre_norm = pre_norm + + self.enable_conv = enable_conv + self.enable_attn = enable_attn + + if self.enable_conv: + self.conv = PointSequential( + spconv.SubMConv3d( + channels, + channels, + kernel_size=3, + bias=True, + indice_key=cpe_indice_key, + ), + nn.Linear(channels, channels), + norm_layer(channels), + ) + else: + self.norm0 = PointSequential( + norm_layer(channels), + ) + + if self.enable_attn: + self.norm1 = PointSequential(norm_layer(channels)) + self.attn = PointROPEAttention( + channels=channels, + patch_size=patch_size, + rope_freq=rope_freq, + num_heads=num_heads, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + order_index=order_index, + ) + self.norm2 = PointSequential(norm_layer(channels)) + self.mlp = PointSequential( + MLP( + in_channels=channels, + hidden_channels=int(channels * mlp_ratio), + out_channels=channels, + act_layer=act_layer, + drop=proj_drop, + ) + ) + self.drop_path = PointSequential( + DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + ) + + def forward(self, point: Point): + if self.enable_conv: + shortcut = point.feat + point = self.conv(point) + point.feat = shortcut + point.feat + else: + point = self.norm0(point) + + if self.enable_attn: + shortcut = point.feat + if self.pre_norm: + point = self.norm1(point) + point = self.drop_path(self.attn(point)) + point.feat = shortcut + point.feat + if not self.pre_norm: + point = self.norm1(point) + + shortcut = point.feat + if self.pre_norm: + point = self.norm2(point) + point = self.drop_path(self.mlp(point)) + point.feat = shortcut + point.feat + if not self.pre_norm: + point = self.norm2(point) + + point.sparse_conv_feat = point.sparse_conv_feat.replace_feature(point.feat) + return point + + +class GridPooling(PointModule): + def __init__( + self, + in_channels, + out_channels, + stride=2, + norm_layer=None, + act_layer=None, + reduce="max", + shuffle_orders=True, + traceable=True, # record parent and cluster + re_serialization=False, + serialization_order="z", + ): + super().__init__() + self.in_channels = in_channels + self.out_channels = out_channels + + self.stride = stride + assert reduce in ["sum", "mean", "min", "max"] + self.reduce = reduce + self.shuffle_orders = shuffle_orders + self.traceable = traceable + + self.proj = nn.Linear(in_channels, out_channels) + if norm_layer is not None: + self.norm = PointSequential(norm_layer(out_channels)) + if act_layer is not None: + self.act = PointSequential(act_layer()) + + self.re_serialization = re_serialization + self.serialization_order = serialization_order + + def forward(self, point: Point): + if "grid_coord" in point.keys(): + grid_coord = point.grid_coord + elif {"coord", "grid_size"}.issubset(point.keys()): + grid_coord = torch.div( + point.coord - point.coord.min(0)[0], + point.grid_size, + rounding_mode="trunc", + ).int() + else: + raise AssertionError( + "[gird_coord] or [coord, grid_size] should be include in the Point" + ) + grid_coord = torch.div(grid_coord, self.stride, rounding_mode="trunc") + grid_coord = grid_coord | point.batch.view(-1, 1) << 48 + grid_coord, cluster, counts = torch.unique( + grid_coord, + sorted=True, + return_inverse=True, + return_counts=True, + dim=0, + ) + grid_coord = grid_coord & ((1 << 48) - 1) + # indices of point sorted by cluster, for torch_scatter.segment_csr + _, indices = torch.sort(cluster) + # index pointer for sorted point, for torch_scatter.segment_csr + idx_ptr = torch.cat([counts.new_zeros(1), torch.cumsum(counts, dim=0)]) + # head_indices of each cluster, for reduce attr e.g. code, batch + head_indices = indices[idx_ptr[:-1]] + point_dict = Dict( + feat=torch_scatter.segment_csr( + self.proj(point.feat)[indices], idx_ptr, reduce=self.reduce + ), + coord=torch_scatter.segment_csr( + point.coord[indices], idx_ptr, reduce="mean" + ), + grid_coord=grid_coord, + batch=point.batch[head_indices], + ) + if "origin_coord" in point.keys(): + point_dict["origin_coord"] = torch_scatter.segment_csr( + point.origin_coord[indices], idx_ptr, reduce="mean" + ) + if "condition" in point.keys(): + point_dict["condition"] = point.condition + if "context" in point.keys(): + point_dict["context"] = point.context + if "name" in point.keys(): + point_dict["name"] = point.name + if "split" in point.keys(): + point_dict["split"] = point.split + if "color" in point.keys(): + point_dict["color"] = torch_scatter.segment_csr( + point.color[indices], idx_ptr, reduce="mean" + ) + if "grid_size" in point.keys(): + point_dict["grid_size"] = point.grid_size * self.stride + if "mask" in point.keys(): + point_dict["mask"] = ( + torch_scatter.segment_csr( + point.mask[indices].float(), idx_ptr, reduce="mean" + ) + > 0.5 + ) + + if self.traceable: + point_dict["pooling_inverse"] = cluster + point_dict["pooling_parent"] = point + point = Point(point_dict) + if self.norm is not None: + point = self.norm(point) + if self.act is not None: + point = self.act(point) + + if self.re_serialization: + point.serialization( + order=self.serialization_order, shuffle_orders=self.shuffle_orders + ) + point.sparsify() + return point + + +class GridUnpooling(PointModule): + def __init__( + self, + in_channels, + skip_channels, + out_channels, + norm_layer=None, + act_layer=None, + traceable=False, # record parent and cluster + ): + super().__init__() + self.proj = PointSequential(nn.Linear(in_channels, out_channels)) + self.proj_skip = PointSequential(nn.Linear(skip_channels, out_channels)) + + if norm_layer is not None: + self.proj.add(norm_layer(out_channels)) + self.proj_skip.add(norm_layer(out_channels)) + + if act_layer is not None: + self.proj.add(act_layer()) + self.proj_skip.add(act_layer()) + + self.traceable = traceable + + def forward(self, point): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pooling_inverse + feat = point.feat + + parent = self.proj_skip(parent) + parent.feat = parent.feat + self.proj(point).feat[inverse] + parent.sparse_conv_feat = parent.sparse_conv_feat.replace_feature(parent.feat) + + if self.traceable: + point.feat = feat + parent["unpooling_parent"] = point + parent["unpooling_inverse"] = inverse + return parent + + +class Embedding(PointModule): + def __init__( + self, + in_channels, + embed_channels, + norm_layer=None, + act_layer=None, + ): + super().__init__() + self.in_channels = in_channels + self.embed_channels = embed_channels + + self.stem = PointSequential( + conv=spconv.SubMConv3d( + in_channels, + embed_channels, + kernel_size=5, + padding=1, + bias=False, + indice_key="stem", + ) + ) + if norm_layer is not None: + self.stem.add(norm_layer(embed_channels), name="norm") + if act_layer is not None: + self.stem.add(act_layer(), name="act") + + def forward(self, point: Point): + point = self.stem(point) + return point + + +@MODELS.register_module("LitePT-v1") +class LitePT(PointModule): + def __init__( + self, + in_channels=4, + order=("z", "z-trans", "hilbert", "hilbert-trans"), + stride=(2, 2, 2, 2), + enc_depths=(2, 2, 2, 6, 2), + enc_channels=(36, 72, 144, 252, 504), + enc_num_head=(2, 4, 8, 14, 28), + enc_patch_size=(1024, 1024, 1024, 1024, 1024), + enc_conv=(True, True, True, False, False), + enc_attn=(False, False, False, True, True), + enc_rope_freq=(100.0, 100.0, 100.0, 100.0, 100.0), + dec_depths=(0, 0, 0, 0), + dec_channels=(72, 72, 144, 252), + dec_num_head=(4, 4, 8, 14), + dec_patch_size=(1024, 1024, 1024, 1024), + dec_conv=(False, False, False, False), + dec_attn=(False, False, False, False), + dec_rope_freq=(100.0, 100.0, 100.0, 100.0), + mlp_ratio=4, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.3, + pre_norm=True, + shuffle_orders=True, + enc_mode=False, + ): + super().__init__() + self.num_stages = len(enc_depths) + self.order = [order] if isinstance(order, str) else order + self.enc_mode = enc_mode + self.shuffle_orders = shuffle_orders + + self.enc_conv = enc_conv + self.enc_attn = enc_attn + self.dec_conv = dec_conv + self.dec_attn = dec_attn + + assert self.num_stages == len(stride) + 1 + assert self.num_stages == len(enc_depths) + assert self.num_stages == len(enc_channels) + assert self.num_stages == len(enc_num_head) + assert self.num_stages == len(enc_patch_size) + assert self.enc_mode or self.num_stages == len(dec_depths) + 1 + assert self.enc_mode or self.num_stages == len(dec_channels) + 1 + assert self.enc_mode or self.num_stages == len(dec_num_head) + 1 + assert self.enc_mode or self.num_stages == len(dec_patch_size) + 1 + + # norm layers + bn_layer = partial(nn.BatchNorm1d, eps=1e-3, momentum=0.01) + ln_layer = nn.LayerNorm + + # activation layers + act_layer = nn.GELU + + self.embedding = Embedding( + in_channels=in_channels, + embed_channels=enc_channels[0], + norm_layer=bn_layer, + act_layer=act_layer, + ) + + # encoder + enc_drop_path = [ + x.item() for x in torch.linspace(0, drop_path, sum(enc_depths)) + ] + self.enc = PointSequential() + for s in range(self.num_stages): + enc_drop_path_ = enc_drop_path[ + sum(enc_depths[:s]) : sum(enc_depths[: s + 1]) + ] + enc = PointSequential() + if s > 0: + enc.add( + GridPooling( + in_channels=enc_channels[s - 1], + out_channels=enc_channels[s], + stride=stride[s - 1], + norm_layer=bn_layer, + act_layer=act_layer, + re_serialization=enc_attn[s], + serialization_order=self.order, + ), + name="down", + ) + for i in range(enc_depths[s]): + enc.add( + Block( + channels=enc_channels[s], + num_heads=enc_num_head[s], + patch_size=enc_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=enc_drop_path_[i], + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_conv=enc_conv[s], + enable_attn=enc_attn[s], + rope_freq=enc_rope_freq[s], + ), + name=f"block{i}", + ) + if len(enc) != 0: + self.enc.add(module=enc, name=f"enc{s}") + + # decoder + if not self.enc_mode: + dec_drop_path = [ + x.item() for x in torch.linspace(0, drop_path, sum(dec_depths)) + ] + self.dec = PointSequential() + dec_channels = list(dec_channels) + [enc_channels[-1]] + for s in reversed(range(self.num_stages - 1)): + dec_drop_path_ = dec_drop_path[ + sum(dec_depths[:s]) : sum(dec_depths[: s + 1]) + ] + dec_drop_path_.reverse() + dec = PointSequential() + dec.add( + GridUnpooling( + in_channels=dec_channels[s + 1], + skip_channels=enc_channels[s], + out_channels=dec_channels[s], + norm_layer=bn_layer, + act_layer=act_layer, + ), + name="up", + ) + for i in range(dec_depths[s]): + dec.add( + Block( + channels=dec_channels[s], + num_heads=dec_num_head[s], + patch_size=dec_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=dec_drop_path_[i], + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_conv=dec_conv[s], + enable_attn=dec_attn[s], + rope_freq=dec_rope_freq[s], + ), + name=f"block{i}", + ) + self.dec.add(module=dec, name=f"dec{s}") + + def forward(self, data_dict): + """ + data_dict is the batched input point cloud, it should contain as least: + 1. feat [N, input_dim]: input feature for the point cloud + 2. grid_coord [N, 3]: voxelized coordinate after grid sampling + or/and + coord [N, 3]: original coordinate + grid_size: grid_size used for grid sampling + 3. offset [batch_size]: separator of point clouds in batched data + or/and + batch [N]: batch index of each point + """ + point = Point(data_dict) + if self.enc_attn[0]: + point.serialization(order=self.order, shuffle_orders=self.shuffle_orders) + point.sparsify() + + point = self.embedding(point) + point = self.enc(point) + + if not self.enc_mode: + point = self.dec(point) + + return point diff --git a/pointcept/models/losses/__init__.py b/pointcept/models/losses/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0f4f29ca432b06443b4267e091066e238599c82d --- /dev/null +++ b/pointcept/models/losses/__init__.py @@ -0,0 +1,4 @@ +from .builder import build_criteria, LOSSES + +from .misc import CrossEntropyLoss, SmoothCELoss, DiceLoss, FocalLoss, BinaryFocalLoss +from .lovasz import LovaszLoss diff --git a/pointcept/models/losses/builder.py b/pointcept/models/losses/builder.py new file mode 100644 index 0000000000000000000000000000000000000000..ef642d9830622d847eb2b7e7013252a32c2b6368 --- /dev/null +++ b/pointcept/models/losses/builder.py @@ -0,0 +1,31 @@ +""" +Criteria Builder + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from pointcept.utils.registry import Registry + +LOSSES = Registry("losses") + + +class Criteria(object): + def __init__(self, cfg=None): + self.cfg = cfg if cfg is not None else [] + self.criteria = [] + for loss_cfg in self.cfg: + self.criteria.append(LOSSES.build(cfg=loss_cfg)) + + def __call__(self, pred, target): + if len(self.criteria) == 0: + # loss computation occur in model + return pred + loss = 0 + for c in self.criteria: + loss += c(pred, target) + return loss + + +def build_criteria(cfg): + return Criteria(cfg) diff --git a/pointcept/models/losses/lovasz.py b/pointcept/models/losses/lovasz.py new file mode 100644 index 0000000000000000000000000000000000000000..690c2ba507478853cf7ea827ac6e4a35d8f31709 --- /dev/null +++ b/pointcept/models/losses/lovasz.py @@ -0,0 +1,257 @@ +""" +Lovasz Loss +refer https://arxiv.org/abs/1705.08790 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from typing import Optional +from itertools import filterfalse +import torch +import torch.nn.functional as F +from torch.nn.modules.loss import _Loss + +from .builder import LOSSES + +BINARY_MODE: str = "binary" +MULTICLASS_MODE: str = "multiclass" +MULTILABEL_MODE: str = "multilabel" + + +def _lovasz_grad(gt_sorted): + """Compute gradient of the Lovasz extension w.r.t sorted errors + See Alg. 1 in paper + """ + p = len(gt_sorted) + gts = gt_sorted.sum() + intersection = gts - gt_sorted.float().cumsum(0) + union = gts + (1 - gt_sorted).float().cumsum(0) + jaccard = 1.0 - intersection / union + if p > 1: # cover 1-pixel case + jaccard[1:p] = jaccard[1:p] - jaccard[0:-1] + return jaccard + + +def _lovasz_hinge(logits, labels, per_image=True, ignore=None): + """ + Binary Lovasz hinge loss + logits: [B, H, W] Logits at each pixel (between -infinity and +infinity) + labels: [B, H, W] Tensor, binary ground truth masks (0 or 1) + per_image: compute the loss per image instead of per batch + ignore: void class id + """ + if per_image: + loss = mean( + _lovasz_hinge_flat( + *_flatten_binary_scores(log.unsqueeze(0), lab.unsqueeze(0), ignore) + ) + for log, lab in zip(logits, labels) + ) + else: + loss = _lovasz_hinge_flat(*_flatten_binary_scores(logits, labels, ignore)) + return loss + + +def _lovasz_hinge_flat(logits, labels): + """Binary Lovasz hinge loss + Args: + logits: [P] Logits at each prediction (between -infinity and +infinity) + labels: [P] Tensor, binary ground truth labels (0 or 1) + """ + if len(labels) == 0: + # only void pixels, the gradients should be 0 + return logits.sum() * 0.0 + signs = 2.0 * labels.float() - 1.0 + errors = 1.0 - logits * signs + errors_sorted, perm = torch.sort(errors, dim=0, descending=True) + perm = perm.data + gt_sorted = labels[perm] + grad = _lovasz_grad(gt_sorted) + loss = torch.dot(F.relu(errors_sorted), grad) + return loss + + +def _flatten_binary_scores(scores, labels, ignore=None): + """Flattens predictions in the batch (binary case) + Remove labels equal to 'ignore' + """ + scores = scores.view(-1) + labels = labels.view(-1) + if ignore is None: + return scores, labels + valid = labels != ignore + vscores = scores[valid] + vlabels = labels[valid] + return vscores, vlabels + + +def _lovasz_softmax( + probas, labels, classes="present", class_seen=None, per_image=False, ignore=None +): + """Multi-class Lovasz-Softmax loss + Args: + @param probas: [B, C, H, W] Class probabilities at each prediction (between 0 and 1). + Interpreted as binary (sigmoid) output with outputs of size [B, H, W]. + @param labels: [B, H, W] Tensor, ground truth labels (between 0 and C - 1) + @param classes: 'all' for all, 'present' for classes present in labels, or a list of classes to average. + @param per_image: compute the loss per image instead of per batch + @param ignore: void class labels + """ + if per_image: + loss = mean( + _lovasz_softmax_flat( + *_flatten_probas(prob.unsqueeze(0), lab.unsqueeze(0), ignore), + classes=classes + ) + for prob, lab in zip(probas, labels) + ) + else: + loss = _lovasz_softmax_flat( + *_flatten_probas(probas, labels, ignore), + classes=classes, + class_seen=class_seen + ) + return loss + + +def _lovasz_softmax_flat(probas, labels, classes="present", class_seen=None): + """Multi-class Lovasz-Softmax loss + Args: + @param probas: [P, C] Class probabilities at each prediction (between 0 and 1) + @param labels: [P] Tensor, ground truth labels (between 0 and C - 1) + @param classes: 'all' for all, 'present' for classes present in labels, or a list of classes to average. + """ + if probas.numel() == 0: + # only void pixels, the gradients should be 0 + return probas * 0.0 + C = probas.size(1) + losses = [] + class_to_sum = list(range(C)) if classes in ["all", "present"] else classes + # for c in class_to_sum: + for c in labels.unique(): + if class_seen is None: + fg = (labels == c).type_as(probas) # foreground for class c + if classes == "present" and fg.sum() == 0: + continue + if C == 1: + if len(classes) > 1: + raise ValueError("Sigmoid output possible only with 1 class") + class_pred = probas[:, 0] + else: + class_pred = probas[:, c] + errors = (fg - class_pred).abs() + errors_sorted, perm = torch.sort(errors, 0, descending=True) + perm = perm.data + fg_sorted = fg[perm] + losses.append(torch.dot(errors_sorted, _lovasz_grad(fg_sorted))) + else: + if c in class_seen: + fg = (labels == c).type_as(probas) # foreground for class c + if classes == "present" and fg.sum() == 0: + continue + if C == 1: + if len(classes) > 1: + raise ValueError("Sigmoid output possible only with 1 class") + class_pred = probas[:, 0] + else: + class_pred = probas[:, c] + errors = (fg - class_pred).abs() + errors_sorted, perm = torch.sort(errors, 0, descending=True) + perm = perm.data + fg_sorted = fg[perm] + losses.append(torch.dot(errors_sorted, _lovasz_grad(fg_sorted))) + return mean(losses) + + +def _flatten_probas(probas, labels, ignore=None): + """Flattens predictions in the batch""" + if probas.dim() == 3: + # assumes output of a sigmoid layer + B, H, W = probas.size() + probas = probas.view(B, 1, H, W) + + C = probas.size(1) + probas = torch.movedim(probas, 1, -1) # [B, C, Di, Dj, ...] -> [B, Di, Dj, ..., C] + probas = probas.contiguous().view(-1, C) # [P, C] + + labels = labels.view(-1) + if ignore is None: + return probas, labels + valid = labels != ignore + vprobas = probas[valid] + vlabels = labels[valid] + return vprobas, vlabels + + +def isnan(x): + return x != x + + +def mean(values, ignore_nan=False, empty=0): + """Nan-mean compatible with generators.""" + values = iter(values) + if ignore_nan: + values = filterfalse(isnan, values) + try: + n = 1 + acc = next(values) + except StopIteration: + if empty == "raise": + raise ValueError("Empty mean") + return empty + for n, v in enumerate(values, 2): + acc += v + if n == 1: + return acc + return acc / n + + +@LOSSES.register_module() +class LovaszLoss(_Loss): + def __init__( + self, + mode: str, + class_seen: Optional[int] = None, + per_image: bool = False, + ignore_index: Optional[int] = None, + loss_weight: float = 1.0, + ): + """Lovasz loss for segmentation task. + It supports binary, multiclass and multilabel cases + Args: + mode: Loss mode 'binary', 'multiclass' or 'multilabel' + ignore_index: Label that indicates ignored pixels (does not contribute to loss) + per_image: If True loss computed per each image and then averaged, else computed per whole batch + Shape + - **y_pred** - torch.Tensor of shape (N, C, H, W) + - **y_true** - torch.Tensor of shape (N, H, W) or (N, C, H, W) + Reference + https://github.com/BloodAxe/pytorch-toolbelt + """ + assert mode in {BINARY_MODE, MULTILABEL_MODE, MULTICLASS_MODE} + super().__init__() + + self.mode = mode + self.ignore_index = ignore_index + self.per_image = per_image + self.class_seen = class_seen + self.loss_weight = loss_weight + + def forward(self, y_pred, y_true): + if self.mode in {BINARY_MODE, MULTILABEL_MODE}: + loss = _lovasz_hinge( + y_pred, y_true, per_image=self.per_image, ignore=self.ignore_index + ) + elif self.mode == MULTICLASS_MODE: + y_pred = y_pred.softmax(dim=1) + loss = _lovasz_softmax( + y_pred, + y_true, + class_seen=self.class_seen, + per_image=self.per_image, + ignore=self.ignore_index, + ) + else: + raise ValueError("Wrong mode {}.".format(self.mode)) + return loss * self.loss_weight diff --git a/pointcept/models/losses/misc.py b/pointcept/models/losses/misc.py new file mode 100644 index 0000000000000000000000000000000000000000..ec300a54b2d920d37882d25e8c7771bce01db97c --- /dev/null +++ b/pointcept/models/losses/misc.py @@ -0,0 +1,223 @@ +""" +Misc Losses + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +import torch.nn as nn +import torch.nn.functional as F +from .builder import LOSSES + + +@LOSSES.register_module() +class CrossEntropyLoss(nn.Module): + def __init__( + self, + weight=None, + size_average=None, + reduce=None, + reduction="mean", + label_smoothing=0.0, + loss_weight=1.0, + ignore_index=-1, + ): + super(CrossEntropyLoss, self).__init__() + weight = torch.tensor(weight).cuda() if weight is not None else None + self.loss_weight = loss_weight + self.loss = nn.CrossEntropyLoss( + weight=weight, + size_average=size_average, + ignore_index=ignore_index, + reduce=reduce, + reduction=reduction, + label_smoothing=label_smoothing, + ) + + def forward(self, pred, target): + return self.loss(pred, target) * self.loss_weight + + +@LOSSES.register_module() +class SmoothCELoss(nn.Module): + def __init__(self, smoothing_ratio=0.1): + super(SmoothCELoss, self).__init__() + self.smoothing_ratio = smoothing_ratio + + def forward(self, pred, target): + eps = self.smoothing_ratio + n_class = pred.size(1) + one_hot = torch.zeros_like(pred).scatter(1, target.view(-1, 1), 1) + one_hot = one_hot * (1 - eps) + (1 - one_hot) * eps / (n_class - 1) + log_prb = F.log_softmax(pred, dim=1) + loss = -(one_hot * log_prb).total(dim=1) + loss = loss[torch.isfinite(loss)].mean() + return loss + + +@LOSSES.register_module() +class BinaryFocalLoss(nn.Module): + def __init__(self, gamma=2.0, alpha=0.5, logits=True, reduce=True, loss_weight=1.0): + """Binary Focal Loss + ` + """ + super(BinaryFocalLoss, self).__init__() + assert 0 < alpha < 1 + self.gamma = gamma + self.alpha = alpha + self.logits = logits + self.reduce = reduce + self.loss_weight = loss_weight + + def forward(self, pred, target, **kwargs): + """Forward function. + Args: + pred (torch.Tensor): The prediction with shape (N) + target (torch.Tensor): The ground truth. If containing class + indices, shape (N) where each value is 0≤targets[i]≤1, If containing class probabilities, + same shape as the input. + Returns: + torch.Tensor: The calculated loss + """ + if self.logits: + bce = F.binary_cross_entropy_with_logits(pred, target, reduction="none") + else: + bce = F.binary_cross_entropy(pred, target, reduction="none") + pt = torch.exp(-bce) + alpha = self.alpha * target + (1 - self.alpha) * (1 - target) + focal_loss = alpha * (1 - pt) ** self.gamma * bce + + if self.reduce: + focal_loss = torch.mean(focal_loss) + return focal_loss * self.loss_weight + + +@LOSSES.register_module() +class FocalLoss(nn.Module): + def __init__( + self, gamma=2.0, alpha=0.5, reduction="mean", loss_weight=1.0, ignore_index=-1 + ): + """Focal Loss + ` + """ + super(FocalLoss, self).__init__() + assert reduction in ( + "mean", + "sum", + ), "AssertionError: reduction should be 'mean' or 'sum'" + assert isinstance( + alpha, (float, list) + ), "AssertionError: alpha should be of type float" + assert isinstance(gamma, float), "AssertionError: gamma should be of type float" + assert isinstance( + loss_weight, float + ), "AssertionError: loss_weight should be of type float" + assert isinstance(ignore_index, int), "ignore_index must be of type int" + self.gamma = gamma + self.alpha = alpha + self.reduction = reduction + self.loss_weight = loss_weight + self.ignore_index = ignore_index + + def forward(self, pred, target, **kwargs): + """Forward function. + Args: + pred (torch.Tensor): The prediction with shape (N, C) where C = number of classes. + target (torch.Tensor): The ground truth. If containing class + indices, shape (N) where each value is 0≤targets[i]≤C−1, If containing class probabilities, + same shape as the input. + Returns: + torch.Tensor: The calculated loss + """ + # [B, C, d_1, d_2, ..., d_k] -> [C, B, d_1, d_2, ..., d_k] + pred = pred.transpose(0, 1) + # [C, B, d_1, d_2, ..., d_k] -> [C, N] + pred = pred.reshape(pred.size(0), -1) + # [C, N] -> [N, C] + pred = pred.transpose(0, 1).contiguous() + # (B, d_1, d_2, ..., d_k) --> (B * d_1 * d_2 * ... * d_k,) + target = target.view(-1).contiguous() + assert pred.size(0) == target.size( + 0 + ), "The shape of pred doesn't match the shape of target" + valid_mask = target != self.ignore_index + target = target[valid_mask] + pred = pred[valid_mask] + + if len(target) == 0: + return 0.0 + + num_classes = pred.size(1) + target = F.one_hot(target, num_classes=num_classes) + + alpha = self.alpha + if isinstance(alpha, list): + alpha = pred.new_tensor(alpha) + pred_sigmoid = pred.sigmoid() + target = target.type_as(pred) + one_minus_pt = (1 - pred_sigmoid) * target + pred_sigmoid * (1 - target) + focal_weight = (alpha * target + (1 - alpha) * (1 - target)) * one_minus_pt.pow( + self.gamma + ) + + loss = ( + F.binary_cross_entropy_with_logits(pred, target, reduction="none") + * focal_weight + ) + if self.reduction == "mean": + loss = loss.mean() + elif self.reduction == "sum": + loss = loss.total() + return self.loss_weight * loss + + +@LOSSES.register_module() +class DiceLoss(nn.Module): + def __init__(self, smooth=1, exponent=2, loss_weight=1.0, ignore_index=-1): + """DiceLoss. + This loss is proposed in `V-Net: Fully Convolutional Neural Networks for + Volumetric Medical Image Segmentation `_. + """ + super(DiceLoss, self).__init__() + self.smooth = smooth + self.exponent = exponent + self.loss_weight = loss_weight + self.ignore_index = ignore_index + + def forward(self, pred, target, **kwargs): + # [B, C, d_1, d_2, ..., d_k] -> [C, B, d_1, d_2, ..., d_k] + pred = pred.transpose(0, 1) + # [C, B, d_1, d_2, ..., d_k] -> [C, N] + pred = pred.reshape(pred.size(0), -1) + # [C, N] -> [N, C] + pred = pred.transpose(0, 1).contiguous() + # (B, d_1, d_2, ..., d_k) --> (B * d_1 * d_2 * ... * d_k,) + target = target.view(-1).contiguous() + assert pred.size(0) == target.size( + 0 + ), "The shape of pred doesn't match the shape of target" + valid_mask = target != self.ignore_index + target = target[valid_mask] + pred = pred[valid_mask] + + pred = F.softmax(pred, dim=1) + num_classes = pred.shape[1] + target = F.one_hot( + torch.clamp(target.long(), 0, num_classes - 1), num_classes=num_classes + ) + + total_loss = 0 + for i in range(num_classes): + if i != self.ignore_index: + num = torch.sum(torch.mul(pred[:, i], target[:, i])) * 2 + self.smooth + den = ( + torch.sum( + pred[:, i].pow(self.exponent) + target[:, i].pow(self.exponent) + ) + + self.smooth + ) + dice_loss = 1 - num / den + total_loss += dice_loss + loss = total_loss / num_classes + return self.loss_weight * loss diff --git a/pointcept/models/masked_scene_contrast/__init__.py b/pointcept/models/masked_scene_contrast/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..18733d94acc5c7ed0feb7c52c77cb040d53bee7e --- /dev/null +++ b/pointcept/models/masked_scene_contrast/__init__.py @@ -0,0 +1,2 @@ +from .masked_scene_contrast_v1m1_base import MaskedSceneContrast +from .masked_scene_contrast_v1m2_csc import MaskedSceneContrast diff --git a/pointcept/models/masked_scene_contrast/masked_scene_contrast_v1m1_base.py b/pointcept/models/masked_scene_contrast/masked_scene_contrast_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..e42b6c0b9863a6de06fe004d37316ca032874a9f --- /dev/null +++ b/pointcept/models/masked_scene_contrast/masked_scene_contrast_v1m1_base.py @@ -0,0 +1,310 @@ +""" +Masked Scene Contrast +https://arxiv.org/abs/2303.14191 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import random +from itertools import chain +import torch +import torch.nn as nn +import torch.distributed as dist +from torch_geometric.nn.pool import voxel_grid + +from timm.layers import trunc_normal_ +import pointops + +from pointcept.models.builder import MODELS, build_model +from pointcept.models.utils import offset2batch +from pointcept.utils.comm import get_world_size + + +@MODELS.register_module("MSC-v1m1") +class MaskedSceneContrast(nn.Module): + def __init__( + self, + backbone, + backbone_in_channels, + backbone_out_channels, + mask_grid_size=0.1, + mask_rate=0.4, + view1_mix_prob=0, + view2_mix_prob=0, + matching_max_k=8, + matching_max_radius=0.03, + matching_max_pair=8192, + nce_t=0.4, + contrast_weight=1, + reconstruct_weight=1, + reconstruct_color=True, + reconstruct_normal=True, + ): + super().__init__() + self.backbone = build_model(backbone) + self.mask_grid_size = mask_grid_size + self.mask_rate = mask_rate + self.view1_mix_prob = view1_mix_prob + self.view2_mix_prob = view2_mix_prob + self.matching_max_k = matching_max_k + self.matching_max_radius = matching_max_radius + self.matching_max_pair = matching_max_pair + self.nce_t = nce_t + self.contrast_weight = contrast_weight + self.reconstruct_weight = reconstruct_weight + self.reconstruct_color = reconstruct_color + self.reconstruct_normal = reconstruct_normal + + self.mask_token = nn.Parameter(torch.zeros(1, backbone_in_channels)) + trunc_normal_(self.mask_token, mean=0.0, std=0.02) + self.color_head = ( + nn.Linear(backbone_out_channels, 3) if reconstruct_color else None + ) + self.normal_head = ( + nn.Linear(backbone_out_channels, 3) if reconstruct_normal else None + ) + self.nce_criteria = torch.nn.CrossEntropyLoss(reduction="mean") + + @torch.no_grad() + def generate_cross_masks( + self, view1_origin_coord, view1_offset, view2_origin_coord, view2_offset + ): + # union origin coord + view1_batch = offset2batch(view1_offset) + view2_batch = offset2batch(view2_offset) + + view1_batch_count = view1_batch.bincount() + view2_batch_count = view2_batch.bincount() + view1_origin_coord_split = view1_origin_coord.split(list(view1_batch_count)) + view2_origin_coord_split = view2_origin_coord.split(list(view2_batch_count)) + union_origin_coord = torch.cat( + list( + chain.from_iterable( + zip(view1_origin_coord_split, view2_origin_coord_split) + ) + ) + ) + union_offset = torch.cat( + [view1_offset.unsqueeze(-1), view2_offset.unsqueeze(-1)], dim=-1 + ).sum(-1) + union_batch = offset2batch(union_offset) + + # grid partition + mask_patch_coord = union_origin_coord.div(self.mask_grid_size) + mask_patch_grid_coord = torch.floor(mask_patch_coord) + mask_patch_cluster = voxel_grid( + pos=mask_patch_grid_coord, size=1, batch=union_batch, start=0 + ) + unique, cluster, counts = torch.unique( + mask_patch_cluster, sorted=True, return_inverse=True, return_counts=True + ) + patch_num = unique.shape[0] + patch_max_point = counts.max().item() + patch2point_map = cluster.new_zeros(patch_num, patch_max_point) + patch2point_mask = torch.lt( + torch.arange(patch_max_point).cuda().unsqueeze(0), counts.unsqueeze(-1) + ) + sorted_cluster_value, sorted_cluster_indices = torch.sort(cluster) + patch2point_map[patch2point_mask] = sorted_cluster_indices + + # generate cross masks + assert self.mask_rate <= 0.5 + patch_mask = torch.zeros(patch_num, device=union_origin_coord.device).int() + rand_perm = torch.randperm(patch_num) + mask_patch_num = int(patch_num * self.mask_rate) + + # mask1 tag with 1, mask2 tag with 2 + patch_mask[rand_perm[0:mask_patch_num]] = 1 + patch_mask[rand_perm[mask_patch_num : mask_patch_num * 2]] = 2 + point_mask = torch.zeros( + union_origin_coord.shape[0], device=union_origin_coord.device + ).int() + point_mask[ + patch2point_map[patch_mask == 1][patch2point_mask[patch_mask == 1]] + ] = 1 + point_mask[ + patch2point_map[patch_mask == 2][patch2point_mask[patch_mask == 2]] + ] = 2 + + # separate mask to view1 and view2 + point_mask_split = point_mask.split( + list( + torch.cat( + [view1_batch_count.unsqueeze(-1), view2_batch_count.unsqueeze(-1)], + dim=-1, + ).flatten() + ) + ) + view1_point_mask = torch.cat(point_mask_split[0::2]) == 1 + view2_point_mask = torch.cat(point_mask_split[1::2]) == 2 + return view1_point_mask, view2_point_mask + + @torch.no_grad() + def match_contrastive_pair( + self, view1_coord, view1_offset, view2_coord, view2_offset, max_k, max_radius + ): + index, distance = pointops.knn_query( + max_k, + view2_coord.float(), + view2_offset.int(), + view1_coord.float(), + view1_offset.int(), + ) + index = torch.cat( + [ + torch.arange(index.shape[0], device=index.device, dtype=torch.long) + .view(-1, 1, 1) + .expand(-1, max_k, 1), + index.view(-1, max_k, 1), + ], + dim=-1, + )[distance.squeeze(-1) < max_radius] + unique, count = index[:, 0].unique(return_counts=True) + select = ( + torch.cumsum(count, dim=0) + - torch.randint(count.max(), count.shape, device=count.device) % count + - 1 + ) + index = index[select] + if index.shape[0] > self.matching_max_pair: + index = index[torch.randperm(index.shape[0])[: self.matching_max_pair]] + return index + + def compute_contrastive_loss( + self, view1_feat, view1_offset, view2_feat, view2_offset, match_index + ): + assert view1_offset.shape == view2_offset.shape + + view1_feat = view1_feat[match_index[:, 0]] + view2_feat = view2_feat[match_index[:, 1]] + view1_feat = view1_feat / ( + torch.norm(view1_feat, p=2, dim=1, keepdim=True) + 1e-7 + ) + view2_feat = view2_feat / ( + torch.norm(view2_feat, p=2, dim=1, keepdim=True) + 1e-7 + ) + sim = torch.mm(view1_feat, view2_feat.transpose(1, 0)) + + with torch.no_grad(): + pos_sim = torch.diagonal(sim).mean() + neg_sim = sim.mean(dim=-1).mean() - pos_sim / match_index.shape[0] + labels = torch.arange(sim.shape[0], device=view1_feat.device).long() + loss = self.nce_criteria(torch.div(sim, self.nce_t), labels) + + if get_world_size() > 1: + dist.all_reduce(loss) + dist.all_reduce(pos_sim) + dist.all_reduce(neg_sim) + return ( + loss / get_world_size(), + pos_sim / get_world_size(), + neg_sim / get_world_size(), + ) + + def forward(self, data_dict): + view1_origin_coord = data_dict["view1_origin_coord"] + view1_coord = data_dict["view1_coord"] + view1_feat = data_dict["view1_feat"] + view1_offset = data_dict["view1_offset"].int() + + view2_origin_coord = data_dict["view2_origin_coord"] + view2_coord = data_dict["view2_coord"] + view2_feat = data_dict["view2_feat"] + view2_offset = data_dict["view2_offset"].int() + + # mask generation by union original coord (without spatial augmentation) + view1_point_mask, view2_point_mask = self.generate_cross_masks( + view1_origin_coord, view1_offset, view2_origin_coord, view2_offset + ) + + view1_mask_tokens = self.mask_token.expand(view1_coord.shape[0], -1) + view1_weight = view1_point_mask.unsqueeze(-1).type_as(view1_mask_tokens) + view1_feat = view1_feat * (1 - view1_weight) + view1_mask_tokens * view1_weight + + view2_mask_tokens = self.mask_token.expand(view2_coord.shape[0], -1) + view2_weight = view2_point_mask.unsqueeze(-1).type_as(view2_mask_tokens) + view2_feat = view2_feat * (1 - view2_weight) + view2_mask_tokens * view2_weight + + view1_data_dict = dict( + origin_coord=view1_origin_coord, + coord=view1_coord, + feat=view1_feat, + offset=view1_offset, + ) + view2_data_dict = dict( + origin_coord=view2_origin_coord, + coord=view2_coord, + feat=view2_feat, + offset=view2_offset, + ) + + # SparseConv based method need grid coord + if "view1_grid_coord" in data_dict.keys(): + view1_data_dict["grid_coord"] = data_dict["view1_grid_coord"] + if "view2_grid_coord" in data_dict.keys(): + view2_data_dict["grid_coord"] = data_dict["view2_grid_coord"] + + # view mixing strategy + if random.random() < self.view1_mix_prob: + view1_data_dict["offset"] = torch.cat( + [view1_offset[1:-1:2], view1_offset[-1].unsqueeze(0)], dim=0 + ) + if random.random() < self.view2_mix_prob: + view2_data_dict["offset"] = torch.cat( + [view2_offset[1:-1:2], view2_offset[-1].unsqueeze(0)], dim=0 + ) + + view1_feat = self.backbone(view1_data_dict) + view2_feat = self.backbone(view2_data_dict) + match_index = self.match_contrastive_pair( + view1_origin_coord, + view1_offset, + view2_origin_coord, + view2_offset, + max_k=self.matching_max_k, + max_radius=self.matching_max_radius, + ) + nce_loss, pos_sim, neg_sim = self.compute_contrastive_loss( + view1_feat, view1_offset, view2_feat, view2_offset, match_index + ) + loss = nce_loss * self.contrast_weight + result_dict = dict(nce_loss=nce_loss, pos_sim=pos_sim, neg_sim=neg_sim) + + if self.color_head is not None: + assert "view1_color" in data_dict.keys() + assert "view2_color" in data_dict.keys() + view1_color = data_dict["view1_color"] + view2_color = data_dict["view2_color"] + view1_color_pred = self.color_head(view1_feat[view1_point_mask]) + view2_color_pred = self.color_head(view2_feat[view2_point_mask]) + color_loss = ( + torch.sum((view1_color_pred - view1_color[view1_point_mask]) ** 2) + + torch.sum((view2_color_pred - view2_color[view2_point_mask]) ** 2) + ) / (view1_color_pred.shape[0] + view2_color_pred.shape[0]) + loss = loss + color_loss * self.reconstruct_weight + result_dict["color_loss"] = color_loss + + if self.normal_head is not None: + assert "view1_normal" in data_dict.keys() + assert "view2_normal" in data_dict.keys() + view1_normal = data_dict["view1_normal"] + view2_normal = data_dict["view2_normal"] + view1_normal_pred = self.normal_head(view1_feat[view1_point_mask]) + view2_normal_pred = self.normal_head(view2_feat[view2_point_mask]) + + view1_normal_pred = view1_normal_pred / ( + torch.norm(view1_normal_pred, p=2, dim=1, keepdim=True) + 1e-10 + ) + view2_normal_pred = view2_normal_pred / ( + torch.norm(view2_normal_pred, p=2, dim=1, keepdim=True) + 1e-10 + ) + normal_loss = ( + torch.sum(view1_normal_pred * view1_normal[view1_point_mask]) + + torch.sum(view2_normal_pred * view2_normal[view2_point_mask]) + ) / (view1_normal_pred.shape[0] + view2_normal_pred.shape[0]) + loss = loss + normal_loss * self.reconstruct_weight + result_dict["normal_loss"] = normal_loss + + result_dict["loss"] = loss + return result_dict diff --git a/pointcept/models/masked_scene_contrast/masked_scene_contrast_v1m2_csc.py b/pointcept/models/masked_scene_contrast/masked_scene_contrast_v1m2_csc.py new file mode 100644 index 0000000000000000000000000000000000000000..435689e027632485f6ca8aad6ce403bac39129fe --- /dev/null +++ b/pointcept/models/masked_scene_contrast/masked_scene_contrast_v1m2_csc.py @@ -0,0 +1,377 @@ +""" +Masked Scene Contrast v1m2 +contrastive learning backend with CSC (https://arxiv.org/abs/2012.09165) + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Chengyao Wang (cywang22@cse.cuhk.edu.hk) +Please cite our work if the code is helpful to you. +""" + +import random +from itertools import chain +import torch +import torch.nn as nn +import torch.distributed as dist +from torch_geometric.nn.pool import voxel_grid + +from timm.layers import trunc_normal_ +import pointops + +from pointcept.models.builder import MODELS, build_model +from pointcept.models.utils import offset2batch +from pointcept.utils.comm import get_world_size + + +@MODELS.register_module("MSC-v1m2") +class MaskedSceneContrast(nn.Module): + def __init__( + self, + backbone, + backbone_in_channels, + backbone_out_channels, + mask_grid_size=0.1, + mask_rate=0.4, + view1_mix_prob=0, + view2_mix_prob=0, + matching_max_k=8, + matching_max_radius=0.03, + matching_max_pair=8192, + nce_t=0.4, + contrast_weight=1, + reconstruct_weight=1, + reconstruct_color=True, + reconstruct_normal=True, + partitions=4, + r1=0.125, + r2=2, + ): + super().__init__() + self.backbone = build_model(backbone) + self.mask_grid_size = mask_grid_size + self.mask_rate = mask_rate + self.view1_mix_prob = view1_mix_prob + self.view2_mix_prob = view2_mix_prob + self.matching_max_k = matching_max_k + self.matching_max_radius = matching_max_radius + self.matching_max_pair = matching_max_pair + self.nce_t = nce_t + self.contrast_weight = contrast_weight + self.reconstruct_weight = reconstruct_weight + self.reconstruct_color = reconstruct_color + self.reconstruct_normal = reconstruct_normal + + # csc partition + self.partitions = partitions + self.r1 = r1 + self.r2 = r2 + + self.mask_token = nn.Parameter(torch.zeros(1, backbone_in_channels)) + trunc_normal_(self.mask_token, mean=0.0, std=0.02) + self.color_head = ( + nn.Linear(backbone_out_channels, 3) if reconstruct_color else None + ) + self.normal_head = ( + nn.Linear(backbone_out_channels, 3) if reconstruct_normal else None + ) + self.nce_criteria = torch.nn.CrossEntropyLoss(reduction="mean") + + @torch.no_grad() + def generate_cross_masks( + self, view1_origin_coord, view1_offset, view2_origin_coord, view2_offset + ): + # union origin coord + view1_batch = offset2batch(view1_offset) + view2_batch = offset2batch(view2_offset) + + view1_batch_count = view1_batch.bincount() + view2_batch_count = view2_batch.bincount() + view1_origin_coord_split = view1_origin_coord.split(list(view1_batch_count)) + view2_origin_coord_split = view2_origin_coord.split(list(view2_batch_count)) + union_origin_coord = torch.cat( + list( + chain.from_iterable( + zip(view1_origin_coord_split, view2_origin_coord_split) + ) + ) + ) + union_offset = torch.cat( + [view1_offset.unsqueeze(-1), view2_offset.unsqueeze(-1)], dim=-1 + ).sum(-1) + union_batch = offset2batch(union_offset) + + # grid partition + mask_patch_coord = union_origin_coord.div(self.mask_grid_size) + mask_patch_grid_coord = torch.floor(mask_patch_coord) + mask_patch_cluster = voxel_grid( + pos=mask_patch_grid_coord, size=1, batch=union_batch, start=0 + ) + unique, cluster, counts = torch.unique( + mask_patch_cluster, sorted=True, return_inverse=True, return_counts=True + ) + patch_num = unique.shape[0] + patch_max_point = counts.max().item() + patch2point_map = cluster.new_zeros(patch_num, patch_max_point) + patch2point_mask = torch.lt( + torch.arange(patch_max_point).cuda().unsqueeze(0), counts.unsqueeze(-1) + ) + sorted_cluster_value, sorted_cluster_indices = torch.sort(cluster) + patch2point_map[patch2point_mask] = sorted_cluster_indices + + # generate cross masks + assert self.mask_rate <= 0.5 + patch_mask = torch.zeros(patch_num, device=union_origin_coord.device).int() + rand_perm = torch.randperm(patch_num) + mask_patch_num = int(patch_num * self.mask_rate) + + # mask1 tag with 1, mask2 tag with 2 + patch_mask[rand_perm[0:mask_patch_num]] = 1 + patch_mask[rand_perm[mask_patch_num : mask_patch_num * 2]] = 2 + point_mask = torch.zeros( + union_origin_coord.shape[0], device=union_origin_coord.device + ).int() + point_mask[ + patch2point_map[patch_mask == 1][patch2point_mask[patch_mask == 1]] + ] = 1 + point_mask[ + patch2point_map[patch_mask == 2][patch2point_mask[patch_mask == 2]] + ] = 2 + + # separate mask to view1 and view2 + point_mask_split = point_mask.split( + list( + torch.cat( + [view1_batch_count.unsqueeze(-1), view2_batch_count.unsqueeze(-1)], + dim=-1, + ).flatten() + ) + ) + view1_point_mask = torch.cat(point_mask_split[0::2]) == 1 + view2_point_mask = torch.cat(point_mask_split[1::2]) == 2 + return view1_point_mask, view2_point_mask + + @torch.no_grad() + def match_contrastive_pair( + self, view1_coord, view1_offset, view2_coord, view2_offset, max_k, max_radius + ): + index, distance = pointops.knn_query( + max_k, + view2_coord.float(), + view2_offset.int(), + view1_coord.float(), + view1_offset.int(), + ) + index = torch.cat( + [ + torch.arange(index.shape[0], device=index.device, dtype=torch.long) + .view(-1, 1, 1) + .expand(-1, max_k, 1), + index.view(-1, max_k, 1), + ], + dim=-1, + )[distance.squeeze(-1) < max_radius] + unique, count = index[:, 0].unique(return_counts=True) + select = ( + torch.cumsum(count, dim=0) + - torch.randint(count.max(), count.shape, device=count.device) % count + - 1 + ) + index = index[select] + if index.shape[0] > self.matching_max_pair: + index = index[torch.randperm(index.shape[0])[: self.matching_max_pair]] + return index + + def compute_partitions(self, coord1, coord2): + partition_matrix = torch.zeros((coord1.shape[0], coord2.shape[0])) + partition_matrix = partition_matrix.cuda() - 1e7 + + rel_trans = coord1.unsqueeze(0) - coord2.unsqueeze(1) + mask_up = rel_trans[:, :, 2] > 0.0 + mask_down = rel_trans[:, :, 2] < 0.0 + + distance_matrix = torch.sqrt(torch.sum(rel_trans.pow(2), 2).add(1e-7)) + + mask = (distance_matrix[:, :] > self.r1) & (distance_matrix[:, :] <= self.r2) + partition_matrix[mask & mask_up] = 0 + partition_matrix[mask & mask_down] = 1 + + mask = distance_matrix[:, :] > self.r2 + partition_matrix[mask & mask_up] = 2 + partition_matrix[mask & mask_down] = 3 + + return partition_matrix + + def compute_contrastive_loss( + self, + view1_feat, + view1_coord, + view1_offset, + view2_feat, + view2_coord, + view2_offset, + match_index, + ): + assert view1_offset.shape == view2_offset.shape + device = view1_feat.device + loss = torch.tensor(0.0, device=device) + pos_sim = torch.tensor(0.0, device=device) + neg_sim = torch.tensor(0.0, device=device) + large_num = 1e9 + + view1_feat = view1_feat[match_index[:, 0]] + view2_feat = view2_feat[match_index[:, 1]] + view1_feat = view1_feat / ( + torch.norm(view1_feat, p=2, dim=1, keepdim=True) + 1e-7 + ) + view2_feat = view2_feat / ( + torch.norm(view2_feat, p=2, dim=1, keepdim=True) + 1e-7 + ) + + view1_coord = view1_coord[match_index[:, 0]] + view2_coord = view2_coord[match_index[:, 1]] + + batch = offset2batch(view1_offset)[match_index[:, 0]] + for batch_id in batch.unique(): + batch_mask = batch == batch_id + sim = torch.mm(view1_feat[batch_mask], view2_feat[batch_mask].T) + + with torch.no_grad(): + pos_sim += torch.diagonal(sim).mean() + neg_sim += sim.mean(dim=-1).mean() - pos_sim / batch_mask.sum() + + labels = torch.arange(sim.shape[0], device=view1_feat.device).long() + part = self.compute_partitions( + view1_coord[batch_mask], view2_coord[batch_mask] + ) + for part_id in part.unique(): + part_mask = part == part_id + part_mask.fill_diagonal_(True) + loss += self.nce_criteria( + torch.div(sim, self.nce_t) - large_num * (~part_mask).float(), + labels, + ) + + loss /= len(view1_offset) * self.partitions + pos_sim /= len(view1_offset) + neg_sim /= len(view1_offset) + + if get_world_size() > 1: + dist.all_reduce(loss) + dist.all_reduce(pos_sim) + dist.all_reduce(neg_sim) + return ( + loss / get_world_size(), + pos_sim / get_world_size(), + neg_sim / get_world_size(), + ) + + def forward(self, data_dict): + view1_origin_coord = data_dict["view1_origin_coord"] + view1_coord = data_dict["view1_coord"] + view1_feat = data_dict["view1_feat"] + view1_offset = data_dict["view1_offset"].int() + + view2_origin_coord = data_dict["view2_origin_coord"] + view2_coord = data_dict["view2_coord"] + view2_feat = data_dict["view2_feat"] + view2_offset = data_dict["view2_offset"].int() + + # mask generation by union original coord (without spatial augmentation) + view1_point_mask, view2_point_mask = self.generate_cross_masks( + view1_origin_coord, view1_offset, view2_origin_coord, view2_offset + ) + + view1_mask_tokens = self.mask_token.expand(view1_coord.shape[0], -1) + view1_weight = view1_point_mask.unsqueeze(-1).type_as(view1_mask_tokens) + view1_feat = view1_feat * (1 - view1_weight) + view1_mask_tokens * view1_weight + + view2_mask_tokens = self.mask_token.expand(view2_coord.shape[0], -1) + view2_weight = view2_point_mask.unsqueeze(-1).type_as(view2_mask_tokens) + view2_feat = view2_feat * (1 - view2_weight) + view2_mask_tokens * view2_weight + + view1_data_dict = dict( + origin_coord=view1_origin_coord, + coord=view1_coord, + feat=view1_feat, + offset=view1_offset, + ) + view2_data_dict = dict( + origin_coord=view2_origin_coord, + coord=view2_coord, + feat=view2_feat, + offset=view2_offset, + ) + + # SparseConv based method need grid coord + if "view1_grid_coord" in data_dict.keys(): + view1_data_dict["grid_coord"] = data_dict["view1_grid_coord"] + if "view2_grid_coord" in data_dict.keys(): + view2_data_dict["grid_coord"] = data_dict["view2_grid_coord"] + + # view mixing strategy + if random.random() < self.view1_mix_prob: + view1_data_dict["offset"] = torch.cat( + [view1_offset[1:-1:2], view1_offset[-1].unsqueeze(0)], dim=0 + ) + if random.random() < self.view2_mix_prob: + view2_data_dict["offset"] = torch.cat( + [view2_offset[1:-1:2], view2_offset[-1].unsqueeze(0)], dim=0 + ) + + view1_feat = self.backbone(view1_data_dict) + view2_feat = self.backbone(view2_data_dict) + match_index = self.match_contrastive_pair( + view1_origin_coord, + view1_offset, + view2_origin_coord, + view2_offset, + max_k=self.matching_max_k, + max_radius=self.matching_max_radius, + ) + nce_loss, pos_sim, neg_sim = self.compute_contrastive_loss( + view1_feat, + view1_origin_coord, + view1_offset, + view2_feat, + view2_origin_coord, + view2_offset, + match_index, + ) + loss = nce_loss * self.contrast_weight + result_dict = dict(nce_loss=nce_loss, pos_sim=pos_sim, neg_sim=neg_sim) + + if self.color_head is not None: + assert "view1_color" in data_dict.keys() + assert "view2_color" in data_dict.keys() + view1_color = data_dict["view1_color"] + view2_color = data_dict["view2_color"] + view1_color_pred = self.color_head(view1_feat[view1_point_mask]) + view2_color_pred = self.color_head(view2_feat[view2_point_mask]) + color_loss = ( + torch.sum((view1_color_pred - view1_color[view1_point_mask]) ** 2) + + torch.sum((view2_color_pred - view2_color[view2_point_mask]) ** 2) + ) / (view1_color_pred.shape[0] + view2_color_pred.shape[0]) + loss = loss + color_loss * self.reconstruct_weight + result_dict["color_loss"] = color_loss + + if self.normal_head is not None: + assert "view1_normal" in data_dict.keys() + assert "view2_normal" in data_dict.keys() + view1_normal = data_dict["view1_normal"] + view2_normal = data_dict["view2_normal"] + view1_normal_pred = self.normal_head(view1_feat[view1_point_mask]) + view2_normal_pred = self.normal_head(view2_feat[view2_point_mask]) + + view1_normal_pred = view1_normal_pred / ( + torch.norm(view1_normal_pred, p=2, dim=1, keepdim=True) + 1e-10 + ) + view2_normal_pred = view2_normal_pred / ( + torch.norm(view2_normal_pred, p=2, dim=1, keepdim=True) + 1e-10 + ) + normal_loss = ( + torch.sum(view1_normal_pred * view1_normal[view1_point_mask]) + + torch.sum(view2_normal_pred * view2_normal[view2_point_mask]) + ) / (view1_normal_pred.shape[0] + view2_normal_pred.shape[0]) + loss = loss + normal_loss * self.reconstruct_weight + result_dict["normal_loss"] = normal_loss + + result_dict["loss"] = loss + return result_dict diff --git a/pointcept/models/modules.py b/pointcept/models/modules.py new file mode 100644 index 0000000000000000000000000000000000000000..0ec8fbdb0c730c665adab0f3e2ef570b45a198f7 --- /dev/null +++ b/pointcept/models/modules.py @@ -0,0 +1,120 @@ +import sys +import torch.nn as nn +import spconv.pytorch as spconv + +try: + import ocnn +except ImportError: + ocnn = None + +from collections import OrderedDict +from pointcept.models.utils.structure import Point +from pointcept.engines.hooks import HookBase + + +def is_ocnn_module(module): + if ocnn is not None: + ocnn_modules = ( + ocnn.nn.OctreeConv, + ocnn.nn.OctreeDeconv, + ocnn.nn.OctreeGroupConv, + ocnn.nn.OctreeDWConv, + ) + return isinstance(module, ocnn_modules) + else: + return False + + +class PointModule(nn.Module): + r"""PointModule + placeholder, all module subclass from this will take Point in PointSequential. + """ + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + +class PointSequential(PointModule): + r"""A sequential container. + Modules will be added to it in the order they are passed in the constructor. + Alternatively, an ordered dict of modules can also be passed in. + """ + + def __init__(self, *args, **kwargs): + super().__init__() + if len(args) == 1 and isinstance(args[0], OrderedDict): + for key, module in args[0].items(): + self.add_module(key, module) + else: + for idx, module in enumerate(args): + self.add_module(str(idx), module) + for name, module in kwargs.items(): + if sys.version_info < (3, 6): + raise ValueError("kwargs only supported in py36+") + if name in self._modules: + raise ValueError("name exists.") + self.add_module(name, module) + + def __getitem__(self, idx): + if not (-len(self) <= idx < len(self)): + raise IndexError("index {} is out of range".format(idx)) + if idx < 0: + idx += len(self) + it = iter(self._modules.values()) + for i in range(idx): + next(it) + return next(it) + + def __len__(self): + return len(self._modules) + + def add(self, module, name=None): + if name is None: + name = str(len(self._modules)) + if name in self._modules: + raise KeyError("name exists") + self.add_module(name, module) + + def forward(self, input): + for k, module in self._modules.items(): + # Point module + if isinstance(module, PointModule): + input = module(input) + # Spconv module + elif spconv.modules.is_spconv_module(module): + if isinstance(input, Point): + input.sparse_conv_feat = module(input.sparse_conv_feat) + input.feat = input.sparse_conv_feat.features + else: + input = module(input) + elif is_ocnn_module(module): + if isinstance(input, Point): + input.octree.features[-1] = module( + input.feat[input.octree_order], input.octree, input.octree.depth + ) + input.feat = input.octree.features[-1][input.octree_inverse] + else: + input = module(input) + # PyTorch module + else: + if isinstance(input, Point): + input.feat = module(input.feat) + if "sparse_conv_feat" in input.keys(): + input.sparse_conv_feat = input.sparse_conv_feat.replace_feature( + input.feat + ) + elif isinstance(input, spconv.SparseConvTensor): + if input.indices.shape[0] != 0: + input = input.replace_feature(module(input.features)) + else: + input = module(input) + return input + + +class PointModel(PointModule, HookBase): + r"""PointModel + placeholder, PointModel can be customized as a Pointcept hook. + """ + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) diff --git a/pointcept/models/oacnns/__init__.py b/pointcept/models/oacnns/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..654b767080457f3814142159a121a2bef9c682b7 --- /dev/null +++ b/pointcept/models/oacnns/__init__.py @@ -0,0 +1 @@ +from .oacnns_v1m1_base import OACNNs diff --git a/pointcept/models/oacnns/oacnns_v1m1_base.py b/pointcept/models/oacnns/oacnns_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..d990bdc782a12fa5c9acbb0c340b687b7017effc --- /dev/null +++ b/pointcept/models/oacnns/oacnns_v1m1_base.py @@ -0,0 +1,344 @@ +from functools import partial +import torch +import torch.nn as nn +import spconv.pytorch as spconv +from timm.layers import trunc_normal_ +from ..builder import MODELS +from ..utils import offset2batch +from torch_geometric.nn.pool import voxel_grid +from torch_geometric.utils import scatter + + +class BasicBlock(nn.Module): + def __init__( + self, + in_channels, + embed_channels, + norm_fn=None, + indice_key=None, + depth=4, + groups=None, + grid_size=None, + bias=False, + ): + super().__init__() + assert embed_channels % groups == 0 + self.groups = groups + self.embed_channels = embed_channels + self.proj = nn.ModuleList() + self.grid_size = grid_size + self.weight = nn.ModuleList() + self.l_w = nn.ModuleList() + self.proj.append( + nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=False), + norm_fn(embed_channels), + nn.ReLU(), + ) + ) + for _ in range(depth - 1): + self.proj.append( + nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=False), + norm_fn(embed_channels), + nn.ReLU(), + ) + ) + self.l_w.append( + nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=False), + norm_fn(embed_channels), + nn.ReLU(), + ) + ) + self.weight.append(nn.Linear(embed_channels, embed_channels, bias=False)) + + self.adaptive = nn.Linear(embed_channels, depth - 1, bias=False) + self.fuse = nn.Sequential( + nn.Linear(embed_channels * 2, embed_channels, bias=False), + norm_fn(embed_channels), + nn.ReLU(), + ) + self.voxel_block = spconv.SparseSequential( + spconv.SubMConv3d( + embed_channels, + embed_channels, + kernel_size=3, + stride=1, + padding=1, + indice_key=indice_key, + bias=bias, + ), + norm_fn(embed_channels), + nn.ReLU(), + spconv.SubMConv3d( + embed_channels, + embed_channels, + kernel_size=3, + stride=1, + padding=1, + indice_key=indice_key, + bias=bias, + ), + norm_fn(embed_channels), + ) + self.act = nn.ReLU() + + def forward(self, x, clusters): + feat = x.features + feats = [] + for i, cluster in enumerate(clusters): + pw = self.l_w[i](feat) + pw = pw - scatter(pw, cluster, reduce="mean")[cluster] + pw = self.weight[i](pw) + pw = torch.exp(pw - pw.max()) + pw = pw / (scatter(pw, cluster, reduce="sum", dim=0)[cluster] + 1e-6) + pfeat = self.proj[i](feat) * pw + pfeat = scatter(pfeat, cluster, reduce="sum")[cluster] + feats.append(pfeat) + adp = self.adaptive(feat) + adp = torch.softmax(adp, dim=1) + feats = torch.stack(feats, dim=1) + feats = torch.einsum("l n, l n c -> l c", adp, feats) + feat = self.proj[-1](feat) + feat = torch.cat([feat, feats], dim=1) + feat = self.fuse(feat) + x.features + res = feat + x = x.replace_feature(feat) + x = self.voxel_block(x) + x = x.replace_feature(self.act(x.features + res)) + return x + + +class DonwBlock(nn.Module): + def __init__( + self, + in_channels, + embed_channels, + depth, + sp_indice_key, + point_grid_size, + num_ref=16, + groups=None, + norm_fn=None, + sub_indice_key=None, + ): + super().__init__() + self.num_ref = num_ref + self.depth = depth + self.point_grid_size = point_grid_size + self.down = spconv.SparseSequential( + spconv.SparseConv3d( + in_channels, + embed_channels, + kernel_size=2, + stride=2, + indice_key=sp_indice_key, + bias=False, + ), + norm_fn(embed_channels), + nn.ReLU(), + ) + self.blocks = nn.ModuleList() + for _ in range(depth): + self.blocks.append( + BasicBlock( + in_channels=embed_channels, + embed_channels=embed_channels, + depth=len(point_grid_size) + 1, + groups=groups, + grid_size=point_grid_size, + norm_fn=norm_fn, + indice_key=sub_indice_key, + ) + ) + + def forward(self, x): + x = self.down(x) + coord = x.indices[:, 1:].float() + batch = x.indices[:, 0] + clusters = [] + for grid_size in self.point_grid_size: + cluster = voxel_grid(pos=coord, size=grid_size, batch=batch) + _, cluster = torch.unique(cluster, return_inverse=True) + clusters.append(cluster) + for block in self.blocks: + x = block(x, clusters) + return x + + +class UpBlock(nn.Module): + def __init__( + self, + in_channels, + skip_channels, + embed_channels, + depth, + sp_indice_key, + norm_fn=None, + down_ratio=2, + sub_indice_key=None, + ): + super().__init__() + assert depth > 0 + self.up = spconv.SparseSequential( + spconv.SparseInverseConv3d( + in_channels, + embed_channels, + kernel_size=down_ratio, + indice_key=sp_indice_key, + bias=False, + ), + norm_fn(embed_channels), + nn.ReLU(), + ) + self.blocks = nn.ModuleList() + self.fuse = nn.Sequential( + nn.Linear(skip_channels + embed_channels, embed_channels), + norm_fn(embed_channels), + nn.ReLU(), + nn.Linear(embed_channels, embed_channels), + norm_fn(embed_channels), + nn.ReLU(), + ) + + def forward(self, x, skip_x): + x = self.up(x) + x = x.replace_feature( + self.fuse(torch.cat([x.features, skip_x.features], dim=1)) + x.features + ) + return x + + +@MODELS.register_module() +class OACNNs(nn.Module): + def __init__( + self, + in_channels, + num_classes, + embed_channels=64, + enc_num_ref=[16, 16, 16, 16], + enc_channels=[64, 64, 128, 256], + groups=[2, 4, 8, 16], + enc_depth=[2, 3, 6, 4], + down_ratio=[2, 2, 2, 2], + dec_channels=[96, 96, 128, 256], + point_grid_size=[[16, 32, 64], [8, 16, 24], [4, 8, 12], [2, 4, 6]], + dec_depth=[2, 2, 2, 2], + ): + super().__init__() + self.in_channels = in_channels + self.num_classes = num_classes + self.num_stages = len(enc_channels) + self.embed_channels = embed_channels + norm_fn = partial(nn.BatchNorm1d, eps=1e-3, momentum=0.01) + + self.stem = spconv.SparseSequential( + spconv.SubMConv3d( + in_channels, + embed_channels, + kernel_size=3, + padding=1, + indice_key="stem", + bias=False, + ), + norm_fn(embed_channels), + nn.ReLU(), + spconv.SubMConv3d( + embed_channels, + embed_channels, + kernel_size=3, + padding=1, + indice_key="stem", + bias=False, + ), + norm_fn(embed_channels), + nn.ReLU(), + spconv.SubMConv3d( + embed_channels, + embed_channels, + kernel_size=3, + padding=1, + indice_key="stem", + bias=False, + ), + norm_fn(embed_channels), + nn.ReLU(), + ) + + self.enc = nn.ModuleList() + self.dec = nn.ModuleList() + for i in range(self.num_stages): + self.enc.append( + DonwBlock( + in_channels=embed_channels if i == 0 else enc_channels[i - 1], + embed_channels=enc_channels[i], + depth=enc_depth[i], + norm_fn=norm_fn, + groups=groups[i], + point_grid_size=point_grid_size[i], + num_ref=enc_num_ref[i], + sp_indice_key=f"spconv{i}", + sub_indice_key=f"subm{i + 1}", + ) + ) + self.dec.append( + UpBlock( + in_channels=( + enc_channels[-1] + if i == self.num_stages - 1 + else dec_channels[i + 1] + ), + skip_channels=embed_channels if i == 0 else enc_channels[i - 1], + embed_channels=dec_channels[i], + depth=dec_depth[i], + norm_fn=norm_fn, + sp_indice_key=f"spconv{i}", + sub_indice_key=f"subm{i}", + ) + ) + + self.final = spconv.SubMConv3d(dec_channels[0], num_classes, kernel_size=1) + self.apply(self._init_weights) + + def forward(self, input_dict): + discrete_coord = input_dict["grid_coord"] + feat = input_dict["feat"] + offset = input_dict["offset"] + batch = offset2batch(offset) + x = spconv.SparseConvTensor( + features=feat, + indices=torch.cat([batch.unsqueeze(-1), discrete_coord], dim=1) + .int() + .contiguous(), + spatial_shape=torch.add( + torch.max(discrete_coord, dim=0).values, 1 + ).tolist(), + batch_size=batch[-1].tolist() + 1, + ) + + x = self.stem(x) + skips = [x] + for i in range(self.num_stages): + x = self.enc[i](x) + skips.append(x) + x = skips.pop(-1) + for i in reversed(range(self.num_stages)): + skip = skips.pop(-1) + x = self.dec[i](x, skip) + x = self.final(x) + return x.features + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, spconv.SubMConv3d): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.BatchNorm1d): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) diff --git a/pointcept/models/octformer/__init__.py b/pointcept/models/octformer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..06bea370d10808b0bad2b68189e957765fc46081 --- /dev/null +++ b/pointcept/models/octformer/__init__.py @@ -0,0 +1 @@ +from .octformer_v1m1_base import OctFormer diff --git a/pointcept/models/octformer/octformer_v1m1_base.py b/pointcept/models/octformer/octformer_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..7c0faf700126b0d589867811e871025dc9782b76 --- /dev/null +++ b/pointcept/models/octformer/octformer_v1m1_base.py @@ -0,0 +1,629 @@ +""" +Octree Transformer + +Modified from https://github.com/octree-nn/octformer + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from typing import Optional, List, Dict +import torch +import torch.nn as nn +from torch.utils.checkpoint import checkpoint + +try: + import ocnn + from ocnn.octree import Octree, Points +except ImportError: + from pointcept.utils.misc import DummyClass + + ocnn = None + Octree = DummyClass + Points = DummyClass + +try: + import dwconv +except ImportError: + dwconv = None + +from pointcept.models.builder import MODELS +from pointcept.models.utils import offset2batch + + +class OctreeT(Octree): + def __init__( + self, + octree: Octree, + patch_size: int = 24, + dilation: int = 4, + nempty: bool = True, + max_depth: Optional[int] = None, + start_depth: Optional[int] = None, + **kwargs + ): + super().__init__(octree.depth, octree.full_depth) + self.__dict__.update(octree.__dict__) + + self.patch_size = patch_size + self.dilation = dilation + self.nempty = nempty + self.max_depth = max_depth or self.depth + self.start_depth = start_depth or self.full_depth + self.invalid_mask_value = -1e3 + assert self.start_depth > 1 + + self.block_num = patch_size * dilation + self.nnum_t = self.nnum_nempty if nempty else self.nnum + self.nnum_a = ((self.nnum_t / self.block_num).ceil() * self.block_num).int() + + num = self.max_depth + 1 + self.batch_idx = [None] * num + self.patch_mask = [None] * num + self.dilate_mask = [None] * num + self.rel_pos = [None] * num + self.dilate_pos = [None] * num + self.build_t() + + def build_t(self): + for d in range(self.start_depth, self.max_depth + 1): + self.build_batch_idx(d) + self.build_attn_mask(d) + self.build_rel_pos(d) + + def build_batch_idx(self, depth: int): + batch = self.batch_id(depth, self.nempty) + self.batch_idx[depth] = self.patch_partition(batch, depth, self.batch_size) + + def build_attn_mask(self, depth: int): + batch = self.batch_idx[depth] + mask = batch.view(-1, self.patch_size) + self.patch_mask[depth] = self._calc_attn_mask(mask) + + mask = batch.view(-1, self.patch_size, self.dilation) + mask = mask.transpose(1, 2).reshape(-1, self.patch_size) + self.dilate_mask[depth] = self._calc_attn_mask(mask) + + def _calc_attn_mask(self, mask: torch.Tensor): + attn_mask = mask.unsqueeze(2) - mask.unsqueeze(1) + attn_mask = attn_mask.masked_fill(attn_mask != 0, self.invalid_mask_value) + return attn_mask + + def build_rel_pos(self, depth: int): + key = self.key(depth, self.nempty) + key = self.patch_partition(key, depth) + x, y, z, _ = ocnn.octree.key2xyz(key, depth) + xyz = torch.stack([x, y, z], dim=1) + + xyz = xyz.view(-1, self.patch_size, 3) + self.rel_pos[depth] = xyz.unsqueeze(2) - xyz.unsqueeze(1) + + xyz = xyz.view(-1, self.patch_size, self.dilation, 3) + xyz = xyz.transpose(1, 2).reshape(-1, self.patch_size, 3) + self.dilate_pos[depth] = xyz.unsqueeze(2) - xyz.unsqueeze(1) + + def patch_partition(self, data: torch.Tensor, depth: int, fill_value=0): + num = self.nnum_a[depth] - self.nnum_t[depth] + tail = data.new_full((num,) + data.shape[1:], fill_value) + return torch.cat([data, tail], dim=0) + + def patch_reverse(self, data: torch.Tensor, depth: int): + return data[: self.nnum_t[depth]] + + +class MLP(torch.nn.Module): + def __init__( + self, + in_features: int, + hidden_features: Optional[int] = None, + out_features: Optional[int] = None, + activation=torch.nn.GELU, + drop: float = 0.0, + **kwargs + ): + super().__init__() + self.in_features = in_features + self.out_features = out_features or in_features + self.hidden_features = hidden_features or in_features + + self.fc1 = torch.nn.Linear(self.in_features, self.hidden_features) + self.act = activation() + self.fc2 = torch.nn.Linear(self.hidden_features, self.out_features) + self.drop = torch.nn.Dropout(drop, inplace=True) + + def forward(self, data: torch.Tensor): + data = self.fc1(data) + data = self.act(data) + data = self.drop(data) + data = self.fc2(data) + data = self.drop(data) + return data + + +class OctreeDWConvBn(torch.nn.Module): + def __init__( + self, + in_channels: int, + kernel_size: List[int] = [3], + stride: int = 1, + nempty: bool = False, + ): + super().__init__() + self.conv = dwconv.OctreeDWConv( + in_channels, kernel_size, nempty, use_bias=False + ) + self.bn = torch.nn.BatchNorm1d(in_channels) + + def forward(self, data: torch.Tensor, octree: Octree, depth: int): + out = self.conv(data, octree, depth) + out = self.bn(out) + return out + + +class RPE(torch.nn.Module): + def __init__(self, patch_size: int, num_heads: int, dilation: int = 1): + super().__init__() + self.patch_size = patch_size + self.num_heads = num_heads + self.dilation = dilation + self.pos_bnd = self.get_pos_bnd(patch_size) + self.rpe_num = 2 * self.pos_bnd + 1 + self.rpe_table = torch.nn.Parameter(torch.zeros(3 * self.rpe_num, num_heads)) + torch.nn.init.trunc_normal_(self.rpe_table, std=0.02) + + def get_pos_bnd(self, patch_size: int): + return int(0.8 * patch_size * self.dilation**0.5) + + def xyz2idx(self, xyz: torch.Tensor): + mul = torch.arange(3, device=xyz.device) * self.rpe_num + xyz = xyz.clamp(-self.pos_bnd, self.pos_bnd) + idx = xyz + (self.pos_bnd + mul) + return idx + + def forward(self, xyz): + idx = self.xyz2idx(xyz) + out = self.rpe_table.index_select(0, idx.reshape(-1)) + out = out.view(idx.shape + (-1,)).sum(3) + out = out.permute(0, 3, 1, 2) # (N, K, K, H) -> (N, H, K, K) + return out + + def extra_repr(self) -> str: + return "num_heads={}, pos_bnd={}, dilation={}".format( + self.num_heads, self.pos_bnd, self.dilation + ) # noqa + + +class OctreeAttention(torch.nn.Module): + def __init__( + self, + dim: int, + patch_size: int, + num_heads: int, + qkv_bias: bool = True, + qk_scale: Optional[float] = None, + attn_drop: float = 0.0, + proj_drop: float = 0.0, + dilation: int = 1, + use_rpe: bool = True, + ): + super().__init__() + self.dim = dim + self.patch_size = patch_size + self.num_heads = num_heads + self.dilation = dilation + self.use_rpe = use_rpe + self.scale = qk_scale or (dim // num_heads) ** -0.5 + + self.qkv = torch.nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = torch.nn.Dropout(attn_drop) + self.proj = torch.nn.Linear(dim, dim) + self.proj_drop = torch.nn.Dropout(proj_drop) + self.softmax = torch.nn.Softmax(dim=-1) + self.rpe = RPE(patch_size, num_heads, dilation) if use_rpe else None + + def forward(self, data: torch.Tensor, octree: OctreeT, depth: int): + H = self.num_heads + K = self.patch_size + C = self.dim + D = self.dilation + + # patch partition + data = octree.patch_partition(data, depth) + if D > 1: # dilation + rel_pos = octree.dilate_pos[depth] + mask = octree.dilate_mask[depth] + data = data.view(-1, K, D, C).transpose(1, 2).reshape(-1, C) + else: + rel_pos = octree.rel_pos[depth] + mask = octree.patch_mask[depth] + data = data.view(-1, K, C) + + # qkv + qkv = self.qkv(data).reshape(-1, K, 3, H, C // H).permute(2, 0, 3, 1, 4) + q, k, v = qkv[0], qkv[1], qkv[2] # (N, H, K, C') + q = q * self.scale + + # attn + attn = q @ k.transpose(-2, -1) # (N, H, K, K) + attn = self.apply_rpe(attn, rel_pos) # (N, H, K, K) + attn = attn + mask.unsqueeze(1) + attn = self.softmax(attn) + attn = self.attn_drop(attn) + data = (attn @ v).transpose(1, 2).reshape(-1, C) + + # patch reverse + if D > 1: # dilation + data = data.view(-1, D, K, C).transpose(1, 2).reshape(-1, C) + data = octree.patch_reverse(data, depth) + + # ffn + data = self.proj(data) + data = self.proj_drop(data) + return data + + def apply_rpe(self, attn, rel_pos): + if self.use_rpe: + attn = attn + self.rpe(rel_pos) + return attn + + def extra_repr(self) -> str: + return "dim={}, patch_size={}, num_heads={}, dilation={}".format( + self.dim, self.patch_size, self.num_heads, self.dilation + ) # noqa + + +class OctFormerBlock(torch.nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + patch_size: int = 32, + dilation: int = 0, + mlp_ratio: float = 4.0, + qkv_bias: bool = True, + qk_scale: Optional[float] = None, + attn_drop: float = 0.0, + proj_drop: float = 0.0, + drop_path: float = 0.0, + nempty: bool = True, + activation: torch.nn.Module = torch.nn.GELU, + **kwargs + ): + super().__init__() + self.norm1 = torch.nn.LayerNorm(dim) + self.attention = OctreeAttention( + dim, + patch_size, + num_heads, + qkv_bias, + qk_scale, + attn_drop, + proj_drop, + dilation, + ) + self.norm2 = torch.nn.LayerNorm(dim) + self.mlp = MLP(dim, int(dim * mlp_ratio), dim, activation, proj_drop) + self.drop_path = ocnn.nn.OctreeDropPath(drop_path, nempty) + self.cpe = OctreeDWConvBn(dim, nempty=nempty) + + def forward(self, data: torch.Tensor, octree: OctreeT, depth: int): + data = self.cpe(data, octree, depth) + data + attn = self.attention(self.norm1(data), octree, depth) + data = data + self.drop_path(attn, octree, depth) + ffn = self.mlp(self.norm2(data)) + data = data + self.drop_path(ffn, octree, depth) + return data + + +class OctFormerStage(torch.nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + patch_size: int = 32, + dilation: int = 0, + mlp_ratio: float = 4.0, + qkv_bias: bool = True, + qk_scale: Optional[float] = None, + attn_drop: float = 0.0, + proj_drop: float = 0.0, + drop_path: float = 0.0, + nempty: bool = True, + activation: torch.nn.Module = torch.nn.GELU, + interval: int = 6, + use_checkpoint: bool = True, + num_blocks: int = 2, + octformer_block=OctFormerBlock, + **kwargs + ): + super().__init__() + self.num_blocks = num_blocks + self.use_checkpoint = use_checkpoint + self.interval = interval # normalization interval + self.num_norms = (num_blocks - 1) // self.interval + + self.blocks = torch.nn.ModuleList( + [ + octformer_block( + dim=dim, + num_heads=num_heads, + patch_size=patch_size, + dilation=1 if (i % 2 == 0) else dilation, + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=( + drop_path[i] if isinstance(drop_path, list) else drop_path + ), + nempty=nempty, + activation=activation, + ) + for i in range(num_blocks) + ] + ) + # self.norms = torch.nn.ModuleList([ + # torch.nn.BatchNorm1d(dim) for _ in range(self.num_norms)]) + + def forward(self, data: torch.Tensor, octree: OctreeT, depth: int): + for i in range(self.num_blocks): + if self.use_checkpoint and self.training: + data = checkpoint(self.blocks[i], data, octree, depth) + else: + data = self.blocks[i](data, octree, depth) + # if i % self.interval == 0 and i != 0: + # data = self.norms[(i - 1) // self.interval](data) + return data + + +class OctFormerDecoder(torch.nn.Module): + def __init__( + self, channels: List[int], fpn_channel: int, nempty: bool, head_up: int = 1 + ): + super().__init__() + self.head_up = head_up + self.num_stages = len(channels) + self.conv1x1 = torch.nn.ModuleList( + [ + torch.nn.Linear(channels[i], fpn_channel) + for i in range(self.num_stages - 1, -1, -1) + ] + ) + self.upsample = ocnn.nn.OctreeUpsample("nearest", nempty) + self.conv3x3 = torch.nn.ModuleList( + [ + ocnn.modules.OctreeConvBnRelu( + fpn_channel, fpn_channel, kernel_size=[3], stride=1, nempty=nempty + ) + for _ in range(self.num_stages) + ] + ) + self.up_conv = torch.nn.ModuleList( + [ + ocnn.modules.OctreeDeconvBnRelu( + fpn_channel, fpn_channel, kernel_size=[3], stride=2, nempty=nempty + ) + for _ in range(self.head_up) + ] + ) + + def forward(self, features: Dict[int, torch.Tensor], octree: Octree): + depth = min(features.keys()) + depth_max = max(features.keys()) + assert self.num_stages == len(features) + + feature = self.conv1x1[0](features[depth]) + conv_out = self.conv3x3[0](feature, octree, depth) + out = self.upsample(conv_out, octree, depth, depth_max) + for i in range(1, self.num_stages): + depth_i = depth + i + feature = self.upsample(feature, octree, depth_i - 1) + feature = self.conv1x1[i](features[depth_i]) + feature + conv_out = self.conv3x3[i](feature, octree, depth_i) + out = out + self.upsample(conv_out, octree, depth_i, depth_max) + for i in range(self.head_up): + out = self.up_conv[i](out, octree, depth_max + i) + return out + + +class PatchEmbed(torch.nn.Module): + def __init__( + self, + in_channels: int = 3, + dim: int = 96, + num_down: int = 2, + nempty: bool = True, + **kwargs + ): + super().__init__() + self.num_stages = num_down + self.delta_depth = -num_down + channels = [int(dim * 2**i) for i in range(-self.num_stages, 1)] + + self.convs = torch.nn.ModuleList( + [ + ocnn.modules.OctreeConvBnRelu( + in_channels if i == 0 else channels[i], + channels[i], + kernel_size=[3], + stride=1, + nempty=nempty, + ) + for i in range(self.num_stages) + ] + ) + self.downsamples = torch.nn.ModuleList( + [ + ocnn.modules.OctreeConvBnRelu( + channels[i], + channels[i + 1], + kernel_size=[2], + stride=2, + nempty=nempty, + ) + for i in range(self.num_stages) + ] + ) + self.proj = ocnn.modules.OctreeConvBnRelu( + channels[-1], dim, kernel_size=[3], stride=1, nempty=nempty + ) + + def forward(self, data: torch.Tensor, octree: Octree, depth: int): + # TODO: reduce to single input + for i in range(self.num_stages): + depth_i = depth - i + data = self.convs[i](data, octree, depth_i) + data = self.downsamples[i](data, octree, depth_i) + data = self.proj(data, octree, depth_i - 1) + return data + + +class Downsample(torch.nn.Module): + def __init__( + self, + in_channels: int, + out_channels: int, + kernel_size: List[int] = (2,), + nempty: bool = True, + ): + super().__init__() + self.norm = torch.nn.BatchNorm1d(out_channels) + self.conv = ocnn.nn.OctreeConv( + in_channels, + out_channels, + kernel_size, + stride=2, + nempty=nempty, + use_bias=True, + ) + + def forward(self, data: torch.Tensor, octree: Octree, depth: int): + data = self.conv(data, octree, depth) + data = self.norm(data) + return data + + +@MODELS.register_module("OctFormer-v1m1") +class OctFormer(torch.nn.Module): + def __init__( + self, + in_channels, + num_classes, + fpn_channels=168, + channels=(96, 192, 384, 384), + num_blocks=(2, 2, 18, 2), + num_heads=(6, 12, 24, 24), + patch_size=26, + stem_down=2, + head_up=2, + dilation=4, + drop_path=0.5, + nempty=True, + octree_scale_factor=10.24, + octree_depth=11, + octree_full_depth=2, + ): + super().__init__() + assert ocnn is not None, "Please follow `README.md` to install ocnn.`" + assert dwconv is not None, "Please follow `README.md` to install dwconv.`" + + self.patch_size = patch_size + self.dilation = dilation + self.nempty = nempty + self.num_stages = len(num_blocks) + self.stem_down = stem_down + self.octree_scale_factor = octree_scale_factor + self.octree_depth = octree_depth + self.octree_full_depth = octree_full_depth + drop_ratio = torch.linspace(0, drop_path, sum(num_blocks)).tolist() + + self.patch_embed = PatchEmbed(in_channels, channels[0], stem_down, nempty) + self.layers = torch.nn.ModuleList( + [ + OctFormerStage( + dim=channels[i], + num_heads=num_heads[i], + patch_size=patch_size, + drop_path=drop_ratio[ + sum(num_blocks[:i]) : sum(num_blocks[: i + 1]) + ], + dilation=dilation, + nempty=nempty, + num_blocks=num_blocks[i], + ) + for i in range(self.num_stages) + ] + ) + self.downsamples = torch.nn.ModuleList( + [ + Downsample(channels[i], channels[i + 1], kernel_size=[2], nempty=nempty) + for i in range(self.num_stages - 1) + ] + ) + self.decoder = OctFormerDecoder( + channels=channels, fpn_channel=fpn_channels, nempty=nempty, head_up=head_up + ) + self.interp = ocnn.nn.OctreeInterp("nearest", nempty) + self.seg_head = ( + nn.Sequential( + nn.Linear(fpn_channels, fpn_channels), + torch.nn.BatchNorm1d(fpn_channels), + nn.ReLU(inplace=True), + nn.Linear(fpn_channels, num_classes), + ) + if num_classes > 0 + else nn.Identity() + ) + + def points2octree(self, points): + octree = ocnn.octree.Octree(self.octree_depth, self.octree_full_depth) + octree.build_octree(points) + return octree + + def forward(self, data_dict): + coord = data_dict["coord"] + normal = data_dict["normal"] + feat = data_dict["feat"] + offset = data_dict["offset"] + batch = offset2batch(offset) + + point = Points( + points=coord / self.octree_scale_factor, + normals=normal, + features=feat, + batch_id=batch.unsqueeze(-1), + batch_size=len(offset), + ) + octree = ocnn.octree.Octree( + depth=self.octree_depth, + full_depth=self.octree_full_depth, + batch_size=len(offset), + device=coord.device, + ) + octree.build_octree(point) + octree.construct_all_neigh() + + feat = self.patch_embed(octree.features[octree.depth], octree, octree.depth) + depth = octree.depth - self.stem_down # current octree depth + octree = OctreeT( + octree, + self.patch_size, + self.dilation, + self.nempty, + max_depth=depth, + start_depth=depth - self.num_stages + 1, + ) + features = {} + for i in range(self.num_stages): + depth_i = depth - i + feat = self.layers[i](feat, octree, depth_i) + features[depth_i] = feat + if i < self.num_stages - 1: + feat = self.downsamples[i](feat, octree, depth_i) + out = self.decoder(features, octree) + # interp representation to points before Octreeization + query_pts = torch.cat([point.points, point.batch_id], dim=1).contiguous() + out = self.interp(out, octree, octree.depth, query_pts) + out = self.seg_head(out) + return out diff --git a/pointcept/models/point_group/__init__.py b/pointcept/models/point_group/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e4735084d4758ca2167a0145adb5d10a72b3a680 --- /dev/null +++ b/pointcept/models/point_group/__init__.py @@ -0,0 +1,2 @@ +from .point_group_v1m1_base import * +from .point_group_v1m2_custom_criteria import * diff --git a/pointcept/models/point_group/point_group_v1m1_base.py b/pointcept/models/point_group/point_group_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..00fa93a4242875774dd297e4245783cbf09f2283 --- /dev/null +++ b/pointcept/models/point_group/point_group_v1m1_base.py @@ -0,0 +1,179 @@ +""" +PointGroup for instance segmentation + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Chengyao Wang +Please cite our work if the code is helpful to you. +""" + +from functools import partial +import torch +import torch.nn as nn +import torch.nn.functional as F + +try: + from pointgroup_ops import ballquery_batch_p, bfs_cluster +except ImportError: + ballquery_batch_p, bfs_cluster = None, None + +from pointcept.models.utils import offset2batch, batch2offset + +from pointcept.models.builder import MODELS, build_model + + +@MODELS.register_module("PG-v1m1") +class PointGroup(nn.Module): + def __init__( + self, + backbone, + backbone_out_channels=64, + semantic_num_classes=20, + semantic_ignore_index=-1, + segment_ignore_index=(-1, 0, 1), + instance_ignore_index=-1, + cluster_thresh=1.5, + cluster_closed_points=300, + cluster_propose_points=100, + cluster_min_points=50, + voxel_size=0.02, + ): + super().__init__() + norm_fn = partial(nn.BatchNorm1d, eps=1e-3, momentum=0.01) + self.semantic_num_classes = semantic_num_classes + self.segment_ignore_index = segment_ignore_index + self.semantic_ignore_index = semantic_ignore_index + self.instance_ignore_index = instance_ignore_index + self.cluster_thresh = cluster_thresh + self.cluster_closed_points = cluster_closed_points + self.cluster_propose_points = cluster_propose_points + self.cluster_min_points = cluster_min_points + self.voxel_size = voxel_size + self.backbone = build_model(backbone) + self.bias_head = nn.Sequential( + nn.Linear(backbone_out_channels, backbone_out_channels), + norm_fn(backbone_out_channels), + nn.ReLU(), + nn.Linear(backbone_out_channels, 3), + ) + self.seg_head = nn.Linear(backbone_out_channels, semantic_num_classes) + self.ce_criteria = torch.nn.CrossEntropyLoss(ignore_index=semantic_ignore_index) + + def forward(self, data_dict): + coord = data_dict["coord"] + instance_centroid = data_dict["instance_centroid"] + offset = data_dict["offset"] + + feat = self.backbone(data_dict) + bias_pred = self.bias_head(feat) + logit_pred = self.seg_head(feat) + + # compute loss + if "segment" in data_dict.keys() and "instance" in data_dict.keys(): + segment = data_dict["segment"] + instance = data_dict["instance"] + seg_loss = self.ce_criteria(logit_pred, segment) + + mask = (instance != self.instance_ignore_index).float() + bias_gt = instance_centroid - coord + bias_dist = torch.sum(torch.abs(bias_pred - bias_gt), dim=-1) + bias_l1_loss = torch.sum(bias_dist * mask) / (torch.sum(mask) + 1e-8) + + bias_pred_norm = bias_pred / ( + torch.norm(bias_pred, p=2, dim=1, keepdim=True) + 1e-8 + ) + bias_gt_norm = bias_gt / ( + torch.norm(bias_gt, p=2, dim=1, keepdim=True) + 1e-8 + ) + cosine_similarity = -(bias_pred_norm * bias_gt_norm).sum(-1) + bias_cosine_loss = torch.sum(cosine_similarity * mask) / ( + torch.sum(mask) + 1e-8 + ) + + loss = seg_loss + bias_l1_loss + bias_cosine_loss + return_dict = dict( + loss=loss, + seg_loss=seg_loss, + bias_l1_loss=bias_l1_loss, + bias_cosine_loss=bias_cosine_loss, + ) + else: + return_dict = dict() + + if not self.training: + center_pred = coord + bias_pred + center_pred /= self.voxel_size + logit_pred = F.softmax(logit_pred, dim=-1) + segment_pred = torch.max(logit_pred, 1)[1] # [n] + # cluster + mask = ( + ~torch.concat( + [ + (segment_pred == index).unsqueeze(-1) + for index in self.segment_ignore_index + ], + dim=1, + ) + .sum(-1) + .bool() + ) + + if mask.sum() == 0: + proposals_idx = torch.zeros(0).int() + proposals_offset = torch.zeros(1).int() + else: + center_pred_ = center_pred[mask] + segment_pred_ = segment_pred[mask] + + batch_ = offset2batch(offset)[mask] + offset_ = nn.ConstantPad1d((1, 0), 0)(batch2offset(batch_)) + idx, start_len = ballquery_batch_p( + center_pred_, + batch_.int(), + offset_.int(), + self.cluster_thresh, + self.cluster_closed_points, + ) + proposals_idx, proposals_offset = bfs_cluster( + segment_pred_.int().cpu(), + idx.cpu(), + start_len.cpu(), + self.cluster_min_points, + ) + proposals_idx[:, 1] = ( + mask.nonzero().view(-1)[proposals_idx[:, 1].long()].int() + ) + + # get proposal + proposals_pred = torch.zeros( + (proposals_offset.shape[0] - 1, center_pred.shape[0]), dtype=torch.int + ) + proposals_pred[proposals_idx[:, 0].long(), proposals_idx[:, 1].long()] = 1 + instance_pred = segment_pred[ + proposals_idx[:, 1][proposals_offset[:-1].long()].long() + ] + proposals_point_num = proposals_pred.sum(1) + proposals_mask = proposals_point_num > self.cluster_propose_points + proposals_pred = proposals_pred[proposals_mask] + instance_pred = instance_pred[proposals_mask] + + pred_scores = [] + pred_classes = [] + pred_masks = proposals_pred.detach().cpu() + for proposal_id in range(len(proposals_pred)): + segment_ = proposals_pred[proposal_id] + confidence_ = logit_pred[ + segment_.bool(), instance_pred[proposal_id] + ].mean() + object_ = instance_pred[proposal_id] + pred_scores.append(confidence_) + pred_classes.append(object_) + if len(pred_scores) > 0: + pred_scores = torch.stack(pred_scores).cpu() + pred_classes = torch.stack(pred_classes).cpu() + else: + pred_scores = torch.tensor([]) + pred_classes = torch.tensor([]) + + return_dict["pred_scores"] = pred_scores + return_dict["pred_masks"] = pred_masks + return_dict["pred_classes"] = pred_classes + return return_dict diff --git a/pointcept/models/point_group/point_group_v1m2_custom_criteria.py b/pointcept/models/point_group/point_group_v1m2_custom_criteria.py new file mode 100644 index 0000000000000000000000000000000000000000..17930202bf22fd92716df5dec314e4309a45a036 --- /dev/null +++ b/pointcept/models/point_group/point_group_v1m2_custom_criteria.py @@ -0,0 +1,200 @@ +""" +PointGroup for instance segmentation + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Chengyao Wang +Please cite our work if the code is helpful to you. +""" + +from functools import partial +import torch +import torch.nn as nn +import torch.nn.functional as F + +try: + from pointgroup_ops import ballquery_batch_p, bfs_cluster +except ImportError: + ballquery_batch_p, bfs_cluster = None, None + +from pointcept.models.utils import offset2batch, batch2offset +from pointcept.models.utils.structure import Point + +from pointcept.models.builder import MODELS, build_model +from pointcept.models.losses import build_criteria + + +@MODELS.register_module("PG-v1m2") +class PointGroup(nn.Module): + def __init__( + self, + backbone, + backbone_out_channels=64, + semantic_num_classes=20, + semantic_ignore_index=-1, + segment_ignore_index=(-1, 0, 1), + instance_ignore_index=-1, + cluster_thresh=1.5, + cluster_closed_points=300, + cluster_propose_points=100, + cluster_min_points=50, + voxel_size=0.02, + criteria=None, + freeze_backbone=False, + ): + super().__init__() + norm_fn = partial(nn.BatchNorm1d, eps=1e-3, momentum=0.01) + self.semantic_num_classes = semantic_num_classes + self.segment_ignore_index = segment_ignore_index + self.semantic_ignore_index = semantic_ignore_index + self.instance_ignore_index = instance_ignore_index + self.cluster_thresh = cluster_thresh + self.cluster_closed_points = cluster_closed_points + self.cluster_propose_points = cluster_propose_points + self.cluster_min_points = cluster_min_points + self.voxel_size = voxel_size + self.backbone = build_model(backbone) + self.bias_head = nn.Sequential( + nn.Linear(backbone_out_channels, backbone_out_channels), + norm_fn(backbone_out_channels), + nn.ReLU(), + nn.Linear(backbone_out_channels, 3), + ) + self.seg_head = nn.Linear(backbone_out_channels, semantic_num_classes) + self.seg_criteria = build_criteria(criteria) + self.freeze_backbone = freeze_backbone + if self.freeze_backbone: + for p in self.backbone.parameters(): + p.requires_grad = False + + def forward(self, data_dict, return_point=False): + if return_point: + return dict(point=self.backbone(data_dict)) + coord = data_dict["coord"] + instance_centroid = data_dict["instance_centroid"] + offset = data_dict["offset"] + + point = self.backbone(data_dict) + if isinstance(point, Point): + while "pooling_parent" in point.keys(): + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + feat = point.feat + else: + feat = point + bias_pred = self.bias_head(feat) + logit_pred = self.seg_head(feat) + + # compute loss + if "segment" in data_dict.keys() and "instance" in data_dict.keys(): + segment = data_dict["segment"] + instance = data_dict["instance"] + seg_loss = self.seg_criteria(logit_pred, segment) + + mask = (instance != self.instance_ignore_index).float() + bias_gt = instance_centroid - coord + bias_dist = torch.sum(torch.abs(bias_pred - bias_gt), dim=-1) + bias_l1_loss = torch.sum(bias_dist * mask) / (torch.sum(mask) + 1e-8) + + bias_pred_norm = bias_pred / ( + torch.norm(bias_pred, p=2, dim=1, keepdim=True) + 1e-8 + ) + bias_gt_norm = bias_gt / ( + torch.norm(bias_gt, p=2, dim=1, keepdim=True) + 1e-8 + ) + cosine_similarity = -(bias_pred_norm * bias_gt_norm).sum(-1) + bias_cosine_loss = torch.sum(cosine_similarity * mask) / ( + torch.sum(mask) + 1e-8 + ) + + loss = seg_loss + bias_l1_loss + bias_cosine_loss + return_dict = dict( + loss=loss, + seg_loss=seg_loss, + bias_l1_loss=bias_l1_loss, + bias_cosine_loss=bias_cosine_loss, + ) + else: + # skip for test split + return_dict = dict() + + if not self.training: + center_pred = coord + bias_pred + center_pred /= self.voxel_size + logit_pred = F.softmax(logit_pred, dim=-1) + segment_pred = torch.max(logit_pred, 1)[1] # [n] + # cluster + mask = ( + ~torch.concat( + [ + (segment_pred == index).unsqueeze(-1) + for index in self.segment_ignore_index + ], + dim=1, + ) + .sum(-1) + .bool() + ) + + if mask.sum() == 0: + proposals_idx = torch.zeros(0).int() + proposals_offset = torch.zeros(1).int() + else: + center_pred_ = center_pred[mask] + segment_pred_ = segment_pred[mask] + + batch_ = offset2batch(offset)[mask] + offset_ = nn.ConstantPad1d((1, 0), 0)(batch2offset(batch_)) + idx, start_len = ballquery_batch_p( + center_pred_, + batch_.int(), + offset_.int(), + self.cluster_thresh, + self.cluster_closed_points, + ) + proposals_idx, proposals_offset = bfs_cluster( + segment_pred_.int().cpu(), + idx.cpu(), + start_len.cpu(), + self.cluster_min_points, + ) + proposals_idx[:, 1] = ( + mask.nonzero().view(-1)[proposals_idx[:, 1].long()].int() + ) + + # get proposal + proposals_pred = torch.zeros( + (proposals_offset.shape[0] - 1, center_pred.shape[0]), dtype=torch.int + ) + proposals_pred[proposals_idx[:, 0].long(), proposals_idx[:, 1].long()] = 1 + instance_pred = segment_pred[ + proposals_idx[:, 1][proposals_offset[:-1].long()].long() + ] + proposals_point_num = proposals_pred.sum(1) + proposals_mask = proposals_point_num > self.cluster_propose_points + proposals_pred = proposals_pred[proposals_mask] + instance_pred = instance_pred[proposals_mask] + + pred_scores = [] + pred_classes = [] + pred_masks = proposals_pred.detach().cpu() + for proposal_id in range(len(proposals_pred)): + segment_ = proposals_pred[proposal_id] + confidence_ = logit_pred[ + segment_.bool(), instance_pred[proposal_id] + ].mean() + object_ = instance_pred[proposal_id] + pred_scores.append(confidence_) + pred_classes.append(object_) + if len(pred_scores) > 0: + pred_scores = torch.stack(pred_scores).cpu() + pred_classes = torch.stack(pred_classes).cpu() + else: + pred_scores = torch.tensor([]) + pred_classes = torch.tensor([]) + + return_dict["pred_scores"] = pred_scores + return_dict["pred_masks"] = pred_masks + return_dict["pred_classes"] = pred_classes + return return_dict diff --git a/pointcept/models/point_group/utils.py b/pointcept/models/point_group/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..3fef63dca6644cd775b81d517ecae3c77bdb8a83 --- /dev/null +++ b/pointcept/models/point_group/utils.py @@ -0,0 +1,176 @@ +import torch +from torch.autograd import Function +import pointgroup_ops + + +class BallQueryBatchP(Function): + @staticmethod + def forward(ctx, coords, batch_idxs, batch_offsets, radius, meanActive): + """ + :param ctx: + :param coords: (n, 3) float + :param batch_idxs: (n) int + :param batch_offsets: (B+1) int + :param radius: float + :param meanActive: int + :return: idx (nActive), int + :return: start_len (n, 2), int + """ + + n = coords.size(0) + + assert coords.is_contiguous() and coords.is_cuda + assert batch_idxs.is_contiguous() and batch_idxs.is_cuda + assert batch_offsets.is_contiguous() and batch_offsets.is_cuda + + while True: + idx = torch.zeros(n * meanActive, dtype=torch.int, device=coords.device) + start_len = torch.zeros((n, 2), dtype=torch.int, device=coords.device) + nActive = pointgroup_ops.ballquery_batch_p( + coords, batch_idxs, batch_offsets, idx, start_len, n, meanActive, radius + ) + if nActive <= n * meanActive: + break + meanActive = int(nActive // n + 1) + idx = idx[:nActive] + + return idx, start_len + + @staticmethod + def backward(ctx, a=None, b=None): + return None, None, None + + +ballquery_batch_p = BallQueryBatchP.apply + + +class Clustering: + def __init__( + self, + ignored_labels, + class_mapping, + thresh=0.03, + closed_points=300, + min_points=50, + propose_points=100, + score_func=torch.max, + ) -> None: + self.ignored_labels = ignored_labels + self.thresh = thresh + self.closed_points = closed_points + self.min_points = min_points + self.class_mapping = class_mapping + self.propose_points = propose_points + self.score_func = score_func + + def cluster(self, vertices, scores): + labels = torch.max(scores, 1)[1] # (N) long, cuda + proposals_idx, proposals_offset = self.cluster_(vertices, labels) + + ## debug + # import ipdb; ipdb.set_trace() + # colors = np.array(create_color_palette())[labels.cpu()] + # write_triangle_mesh(vertices, colors, None, 'semantics.ply') + + # scatter + proposals_pred = torch.zeros( + (proposals_offset.shape[0] - 1, vertices.shape[0]), dtype=torch.int + ) # (nProposal, N), int, cuda + proposals_pred[proposals_idx[:, 0].long(), proposals_idx[:, 1].long()] = 1 + labels = labels[proposals_idx[:, 1][proposals_offset[:-1].long()].long()] + + proposals_pointnum = proposals_pred.sum(1) + npoint_mask = proposals_pointnum > self.propose_points + + proposals_pred = proposals_pred[npoint_mask] + labels = labels[npoint_mask] + return proposals_pred, labels + + def cluster_(self, vertices, labels): + """ + :param batch_idxs: (N), int, cuda + :labels: 0-19 + """ + batch_idxs = torch.zeros_like(labels) + + mask_non_ignored = torch.ones_like(labels).bool() + for ignored_label in self.ignored_labels: + mask_non_ignored = mask_non_ignored & ( + self.class_mapping[labels] != ignored_label + ) + object_idxs = mask_non_ignored.nonzero().view(-1) + + vertices_ = vertices[object_idxs].float() + labels_ = labels[object_idxs].int() + + if vertices_.numel() == 0: + return torch.zeros((0, 2)).int(), torch.zeros(1).int() + + batch_idxs_ = batch_idxs[object_idxs].int() + batch_offsets_ = torch.FloatTensor([0, object_idxs.shape[0]]).int().cuda() + + idx, start_len = ballquery_batch_p( + vertices_, batch_idxs_, batch_offsets_, self.thresh, self.closed_points + ) + proposals_idx, proposals_offset = bfs_cluster( + labels_.cpu(), idx.cpu(), start_len.cpu(), self.min_points + ) + proposals_idx[:, 1] = object_idxs[proposals_idx[:, 1].long()].int() + + return proposals_idx, proposals_offset + + def get_instances(self, vertices, scores): + proposals_pred, labels = self.cluster(vertices, scores) + instances = {} + for proposal_id in range(len(proposals_pred)): + clusters_i = proposals_pred[proposal_id] + score = scores[clusters_i.bool(), labels[proposal_id]] + score = self.score_func(score) + instances[proposal_id] = {} + instances[proposal_id]["conf"] = score.cpu().numpy() + instances[proposal_id]["label_id"] = self.class_mapping.cpu()[ + labels[proposal_id] + ] + instances[proposal_id]["pred_mask"] = clusters_i.cpu().numpy() + return instances + + +class BFSCluster(Function): + @staticmethod + def forward(ctx, semantic_label, ball_query_idxs, start_len, threshold): + """ + :param ctx: + :param semantic_label: (N), int + :param ball_query_idxs: (nActive), int + :param start_len: (N, 2), int + :return: cluster_idxs: int (sumNPoint, 2), dim 0 for cluster_id, dim 1 for corresponding point idxs in N + :return: cluster_offsets: int (nCluster + 1) + """ + + N = start_len.size(0) + + assert semantic_label.is_contiguous() + assert ball_query_idxs.is_contiguous() + assert start_len.is_contiguous() + + cluster_idxs = semantic_label.new() + cluster_offsets = semantic_label.new() + + pointgroup_ops.bfs_cluster( + semantic_label, + ball_query_idxs, + start_len, + cluster_idxs, + cluster_offsets, + N, + threshold, + ) + + return cluster_idxs, cluster_offsets + + @staticmethod + def backward(ctx, a=None): + return None + + +bfs_cluster = BFSCluster.apply diff --git a/pointcept/models/point_prompt_training/__init__.py b/pointcept/models/point_prompt_training/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..83701f2ffe14c8479c5fcbfd7a05e670e680ec4d --- /dev/null +++ b/pointcept/models/point_prompt_training/__init__.py @@ -0,0 +1,5 @@ +from .point_prompt_training_v1m1_language_guided import * +from .point_prompt_training_v1m2_decoupled import * +from .point_prompt_training_v1m3_neo import * + +from .prompt_driven_normalization import PDNorm diff --git a/pointcept/models/point_prompt_training/point_prompt_training_v1m1_language_guided.py b/pointcept/models/point_prompt_training/point_prompt_training_v1m1_language_guided.py new file mode 100644 index 0000000000000000000000000000000000000000..443fc482ba02c319905a387c58fa7ec5032f6a07 --- /dev/null +++ b/pointcept/models/point_prompt_training/point_prompt_training_v1m1_language_guided.py @@ -0,0 +1,118 @@ +""" +Point Prompt Training + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from collections import OrderedDict + +import torch +import torch.nn as nn +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS +from pointcept.models.losses import build_criteria + + +@MODELS.register_module("PPT-v1m1") +class PointPromptTraining(nn.Module): + """ + PointPromptTraining provides Data-driven Context and enables multi-dataset training with + Language-driven Categorical Alignment. PDNorm is supported by SpUNet-v1m3 to adapt the + backbone to a specific dataset with a given dataset condition and context. + """ + + def __init__( + self, + backbone=None, + criteria=None, + backbone_out_channels=96, + context_channels=256, + conditions=("Structured3D", "ScanNet", "S3DIS"), + template="[x]", + clip_model="ViT-B/16", + # fmt: off + class_name=( + "wall", "floor", "cabinet", "bed", "chair", "sofa", "table", "door", + "window", "bookshelf", "bookcase", "picture", "counter", "desk", "shelves", "curtain", + "dresser", "pillow", "mirror", "ceiling", "refrigerator", "television", "shower curtain", "nightstand", + "toilet", "sink", "lamp", "bathtub", "garbagebin", "board", "beam", "column", + "clutter", "otherstructure", "otherfurniture", "otherprop", + ), + valid_index=( + (0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 25, 26, 33, 34, 35), + (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 20, 22, 24, 25, 27, 34), + (0, 1, 4, 5, 6, 7, 8, 10, 19, 29, 30, 31, 32), + ), + # fmt: on + backbone_mode=False, + ): + super().__init__() + assert len(conditions) == len(valid_index) + assert backbone.type in ["SpUNet-v1m3", "PT-v2m3", "PT-v3m1"] + self.backbone = MODELS.build(backbone) + self.criteria = build_criteria(criteria) + self.conditions = conditions + self.valid_index = valid_index + self.embedding_table = nn.Embedding(len(conditions), context_channels) + self.backbone_mode = backbone_mode + if not self.backbone_mode: + import clip + + clip_model, _ = clip.load( + clip_model, device="cpu", download_root="./.cache/clip" + ) + clip_model.requires_grad_(False) + class_prompt = [template.replace("[x]", name) for name in class_name] + class_token = clip.tokenize(class_prompt) + class_embedding = clip_model.encode_text(class_token) + class_embedding = class_embedding / class_embedding.norm( + dim=-1, keepdim=True + ) + self.register_buffer("class_embedding", class_embedding) + self.proj_head = nn.Linear( + backbone_out_channels, clip_model.text_projection.shape[1] + ) + self.logit_scale = clip_model.logit_scale + + def forward(self, data_dict): + condition = data_dict["condition"][0] + assert condition in self.conditions + context = self.embedding_table( + torch.tensor( + [self.conditions.index(condition)], device=data_dict["coord"].device + ) + ) + data_dict["context"] = context + point = self.backbone(data_dict) + # Backbone added after v1.5.0 return Point instead of feat and use DefaultSegmentorV2 + # TODO: remove this part after make all backbone return Point only. + if isinstance(point, Point): + feat = point.feat + else: + feat = point + if self.backbone_mode: + # PPT serve as a multi-dataset backbone when enable backbone mode + return feat + feat = self.proj_head(feat) + feat = feat / feat.norm(dim=-1, keepdim=True) + sim = ( + feat + @ self.class_embedding[ + self.valid_index[self.conditions.index(condition)], : + ].t() + ) + logit_scale = self.logit_scale.exp() + seg_logits = logit_scale * sim + # train + if self.training: + loss = self.criteria(seg_logits, data_dict["segment"]) + return dict(loss=loss) + # eval + elif "segment" in data_dict.keys(): + loss = self.criteria(seg_logits, data_dict["segment"]) + return dict(loss=loss, seg_logits=seg_logits) + # test + else: + return dict(seg_logits=seg_logits) diff --git a/pointcept/models/point_prompt_training/point_prompt_training_v1m2_decoupled.py b/pointcept/models/point_prompt_training/point_prompt_training_v1m2_decoupled.py new file mode 100644 index 0000000000000000000000000000000000000000..9ad9c6bf1fcb5e17b139d6c06b44b589fcee816f --- /dev/null +++ b/pointcept/models/point_prompt_training/point_prompt_training_v1m2_decoupled.py @@ -0,0 +1,79 @@ +""" +Point Prompt Training with decoupled segmentation head + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from collections import OrderedDict + +import torch +import torch.nn as nn +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS +from pointcept.models.losses import build_criteria + + +@MODELS.register_module("PPT-v1m2") +class PointPromptTraining(nn.Module): + """ + PointPromptTraining v1m2 provides Data-driven Context and enables multi-dataset training with + Decoupled Segmentation Head. PDNorm is supported by SpUNet-v1m3 to adapt the + backbone to a specific dataset with a given dataset condition and context. + """ + + def __init__( + self, + backbone=None, + criteria=None, + backbone_out_channels=96, + context_channels=256, + conditions=("Structured3D", "ScanNet", "S3DIS"), + num_classes=(25, 20, 13), + backbone_mode=False, + ): + super().__init__() + assert len(conditions) == len(num_classes) + assert backbone.type in ["SpUNet-v1m3", "PT-v2m3", "PT-v3m1"] + self.backbone = MODELS.build(backbone) + self.criteria = build_criteria(criteria) + self.conditions = conditions + self.embedding_table = nn.Embedding(len(conditions), context_channels) + self.backbone_mode = backbone_mode + self.seg_heads = nn.ModuleList( + [nn.Linear(backbone_out_channels, num_cls) for num_cls in num_classes] + ) + + def forward(self, data_dict): + condition = data_dict["condition"][0] + assert condition in self.conditions + context = self.embedding_table( + torch.tensor( + [self.conditions.index(condition)], device=data_dict["coord"].device + ) + ) + data_dict["context"] = context + point = self.backbone(data_dict) + # Backbone added after v1.5.0 return Point instead of feat and use DefaultSegmentorV2 + # TODO: remove this part after make all backbone return Point only. + if isinstance(point, Point): + feat = point.feat + else: + feat = point + if self.backbone_mode: + # PPT serve as a multi-dataset backbone when enable backbone mode + return feat + seg_head = self.seg_heads[self.conditions.index(condition)] + seg_logits = seg_head(feat) + # train + if self.training: + loss = self.criteria(seg_logits, data_dict["segment"]) + return dict(loss=loss) + # eval + elif "segment" in data_dict.keys(): + loss = self.criteria(seg_logits, data_dict["segment"]) + return dict(loss=loss, seg_logits=seg_logits) + # test + else: + return dict(seg_logits=seg_logits) diff --git a/pointcept/models/point_prompt_training/point_prompt_training_v1m3_neo.py b/pointcept/models/point_prompt_training/point_prompt_training_v1m3_neo.py new file mode 100644 index 0000000000000000000000000000000000000000..21bd4d41680d515cf57c6bf5d2242ddfd14cb7cd --- /dev/null +++ b/pointcept/models/point_prompt_training/point_prompt_training_v1m3_neo.py @@ -0,0 +1,140 @@ +""" +Point Prompt Training specific for Sonata + +In Sonata, as we identify the criminal that restricts domain adaptive capacity is BN +and successfully replaces all BN in PTv3 to LN. I think we don't need PDNorm +anymore. Hence, I remove the domain prompting and turns it into a pure multi-dataset +joint training framework. + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from collections import OrderedDict + +import torch +import torch.nn as nn +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS +from pointcept.models.losses import build_criteria + + +@MODELS.register_module("PPT-v1m3") +class PointPromptTraining(nn.Module): + """ + PointPromptTraining provides Data-driven Context and enables multi-dataset training with + Language-driven Categorical Alignment. PDNorm is supported by SpUNet-v1m3 to adapt the + backbone to a specific dataset with a given dataset condition and context. + """ + + def __init__( + self, + backbone=None, + criteria=None, + backbone_out_channels=96, + conditions=("Structured3D", "ScanNet", "S3DIS"), + template="[x]", + clip_model="ViT-B/16", + class_names=None, + freeze_backbone=False, + backbone_mode=False, + ): + super().__init__() + if class_names is None: + # fmt: off + class_names = [ + ("wall", "floor", "cabinet", "bed", "chair", + "sofa", "table", "door", "window", "picture", + "desk", "shelves", "curtain", "dresser", "pillow", + "mirror", "ceiling", "refrigerator", "television", "nightstand", + "sink", "lamp", "otherstructure", "otherfurniture", "otherprop"), + ("wall", "floor", "cabinet", "bed", "chair", + "sofa", "table", "door", "window", "bookshelf", + "picture", "counter", "desk", "curtain", "refridgerator", + "shower curtain", "toilet", "sink", "bathtub", "otherfurniture"), + ("ceiling", "floor", "wall", "beam", "column", + "window", "door", "table", "chair", "sofa", + "bookcase", "board", "clutter"), + ] + # fmt: on + assert len(conditions) == len(class_names) + # assert backbone.type in ["SpUNet-v1m3", "PT-v2m3", "PT-v3m1"] + self.backbone = MODELS.build(backbone) + self.criteria = build_criteria(criteria) + self.conditions = conditions + self.freeze_backbone = freeze_backbone + if self.freeze_backbone: + for p in self.backbone.parameters(): + p.requires_grad = False + self.backbone_mode = backbone_mode + if not self.backbone_mode: + import clip + + clip_model, _ = clip.load( + clip_model, device="cpu", download_root="./.cache/clip" + ) + clip_model.requires_grad_(False) + class_embeddings = [] + num_classes = [] + for i in range(len(conditions)): + class_prompt = [ + template.replace("[x]", name) for name in class_names[i] + ] + class_token = clip.tokenize(class_prompt) + class_embedding = clip_model.encode_text(class_token) + class_embedding = class_embedding / class_embedding.norm( + dim=-1, keepdim=True + ) + class_embeddings.append(class_embedding) + num_classes.append(len(class_prompt)) + class_embeddings = torch.cat(class_embeddings, dim=0) + self.register_buffer("class_embeddings", class_embeddings, persistent=False) + self.num_classes = num_classes + self.proj_head = nn.Linear( + backbone_out_channels, clip_model.text_projection.shape[1] + ) + self.logit_scale = clip_model.logit_scale + + def forward(self, data_dict): + condition = data_dict["condition"][0] + if self.freeze_backbone: + with torch.no_grad(): + point = self.backbone(data_dict) + else: + point = self.backbone(data_dict) + while "pooling_parent" in point.keys(): + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + + if self.backbone_mode: + # PPT serve as a multi-dataset backbone when enable backbone mode + return point.feat + + feat = self.proj_head(point.feat) + + eps = 1e-6 if feat.dtype == torch.float16 else 1e-12 + feat = nn.functional.normalize(feat, dim=-1, p=2, eps=eps) + + sim = ( + feat + @ self.class_embeddings.split(self.num_classes)[ + self.conditions.index(condition) + ].t() + ) + logit_scale = self.logit_scale.exp() + seg_logits = logit_scale * sim + # train + if self.training: + loss = self.criteria(seg_logits, data_dict["segment"]) + return dict(loss=loss) + # eval + elif "segment" in data_dict.keys(): + loss = self.criteria(seg_logits, data_dict["segment"]) + return dict(loss=loss, seg_logits=seg_logits) + # test + else: + return dict(seg_logits=seg_logits) diff --git a/pointcept/models/point_prompt_training/prompt_driven_normalization.py b/pointcept/models/point_prompt_training/prompt_driven_normalization.py new file mode 100644 index 0000000000000000000000000000000000000000..5d7d0d0c01dd4ccf939afeff870b5d72cab403a3 --- /dev/null +++ b/pointcept/models/point_prompt_training/prompt_driven_normalization.py @@ -0,0 +1,47 @@ +import torch.nn as nn + +from pointcept.models.modules import PointModule, PointSequential +from pointcept.models.builder import MODULES + + +@MODULES.register_module() +class PDNorm(PointModule): + def __init__( + self, + num_features, + norm_layer, + context_channels=256, + conditions=("ScanNet", "S3DIS", "Structured3D"), + decouple=True, + adaptive=False, + ): + super().__init__() + self.conditions = conditions + self.decouple = decouple + self.adaptive = adaptive + if self.decouple: + self.norm = nn.ModuleList([norm_layer(num_features) for _ in conditions]) + else: + self.norm = norm_layer + if self.adaptive: + self.modulation = nn.Sequential( + nn.SiLU(), nn.Linear(context_channels, 2 * num_features, bias=True) + ) + + def forward(self, point): + assert {"feat", "condition"}.issubset(point.keys()) + if isinstance(point.condition, str): + condition = point.condition + else: + condition = point.condition[0] + if self.decouple: + assert condition in self.conditions + norm = self.norm[self.conditions.index(condition)] + else: + norm = self.norm + point.feat = norm(point.feat) + if self.adaptive: + assert "context" in point.keys() + shift, scale = self.modulation(point.context).chunk(2, dim=1) + point.feat = point.feat * (1.0 + scale) + shift + return point diff --git a/pointcept/models/point_transformer/__init__.py b/pointcept/models/point_transformer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..d6493a312bfcf559642e6a2cc77d96c3770f0dd1 --- /dev/null +++ b/pointcept/models/point_transformer/__init__.py @@ -0,0 +1,3 @@ +from .point_transformer_seg import * +from .point_transformer_partseg import * +from .point_transformer_cls import * diff --git a/pointcept/models/point_transformer/point_transformer_cls.py b/pointcept/models/point_transformer/point_transformer_cls.py new file mode 100644 index 0000000000000000000000000000000000000000..8e12746fef73e9b3ee75b72942fbc8dc96e6e1bf --- /dev/null +++ b/pointcept/models/point_transformer/point_transformer_cls.py @@ -0,0 +1,131 @@ +""" +Point Transformer V1 for Object Classification + +Might be a bit different from the original paper + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +import torch.nn as nn + +from .point_transformer_seg import TransitionDown, Bottleneck +from pointcept.models.builder import MODELS + + +class PointTransformerCls(nn.Module): + def __init__(self, block, blocks, in_channels=6, num_classes=40): + super().__init__() + self.in_channels = in_channels + self.in_planes, planes = in_channels, [32, 64, 128, 256, 512] + fpn_planes, fpnhead_planes, share_planes = 128, 64, 8 + stride, nsample = [1, 4, 4, 4, 4], [8, 16, 16, 16, 16] + self.enc1 = self._make_enc( + block, + planes[0], + blocks[0], + share_planes, + stride=stride[0], + nsample=nsample[0], + ) # N/1 + self.enc2 = self._make_enc( + block, + planes[1], + blocks[1], + share_planes, + stride=stride[1], + nsample=nsample[1], + ) # N/4 + self.enc3 = self._make_enc( + block, + planes[2], + blocks[2], + share_planes, + stride=stride[2], + nsample=nsample[2], + ) # N/16 + self.enc4 = self._make_enc( + block, + planes[3], + blocks[3], + share_planes, + stride=stride[3], + nsample=nsample[3], + ) # N/64 + self.enc5 = self._make_enc( + block, + planes[4], + blocks[4], + share_planes, + stride=stride[4], + nsample=nsample[4], + ) # N/256 + self.cls = nn.Sequential( + nn.Linear(planes[4], 256), + nn.BatchNorm1d(256), + nn.ReLU(inplace=True), + nn.Dropout(p=0.5), + nn.Linear(256, 128), + nn.BatchNorm1d(128), + nn.ReLU(inplace=True), + nn.Dropout(p=0.5), + nn.Linear(128, num_classes), + ) + + def _make_enc(self, block, planes, blocks, share_planes=8, stride=1, nsample=16): + layers = [ + TransitionDown(self.in_planes, planes * block.expansion, stride, nsample) + ] + self.in_planes = planes * block.expansion + for _ in range(1, blocks): + layers.append( + block(self.in_planes, self.in_planes, share_planes, nsample=nsample) + ) + return nn.Sequential(*layers) + + def forward(self, data_dict): + p0 = data_dict["coord"] + x0 = data_dict["feat"] + o0 = data_dict["offset"].int() + x0 = p0 if self.in_channels == 3 else torch.cat((p0, x0), 1) + p1, x1, o1 = self.enc1([p0, x0, o0]) + p2, x2, o2 = self.enc2([p1, x1, o1]) + p3, x3, o3 = self.enc3([p2, x2, o2]) + p4, x4, o4 = self.enc4([p3, x3, o3]) + p5, x5, o5 = self.enc5([p4, x4, o4]) + x = [] + for i in range(o5.shape[0]): + if i == 0: + s_i, e_i, cnt = 0, o5[0], o5[0] + else: + s_i, e_i, cnt = o5[i - 1], o5[i], o5[i] - o5[i - 1] + x_b = x5[s_i:e_i, :].sum(0, True) / cnt + x.append(x_b) + x = torch.cat(x, 0) + x = self.cls(x) + return x + + +@MODELS.register_module("PointTransformer-Cls26") +class PointTransformerCls26(PointTransformerCls): + def __init__(self, **kwargs): + super(PointTransformerCls26, self).__init__( + Bottleneck, [1, 1, 1, 1, 1], **kwargs + ) + + +@MODELS.register_module("PointTransformer-Cls38") +class PointTransformerCls38(PointTransformerCls): + def __init__(self, **kwargs): + super(PointTransformerCls38, self).__init__( + Bottleneck, [1, 2, 2, 2, 2], **kwargs + ) + + +@MODELS.register_module("PointTransformer-Cls50") +class PointTransformerCls50(PointTransformerCls): + def __init__(self, **kwargs): + super(PointTransformerCls50, self).__init__( + Bottleneck, [1, 2, 3, 5, 2], **kwargs + ) diff --git a/pointcept/models/point_transformer/point_transformer_partseg.py b/pointcept/models/point_transformer/point_transformer_partseg.py new file mode 100644 index 0000000000000000000000000000000000000000..3326a9f7d6fd62a9394e434135615339a3c679f8 --- /dev/null +++ b/pointcept/models/point_transformer/point_transformer_partseg.py @@ -0,0 +1,374 @@ +""" +Point Transformer V1 for Part Segmentation + +Might be a bit different from the original paper + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +import torch.nn as nn +import einops +import pointops + +from pointcept.models.builder import MODELS +from .utils import LayerNorm1d + + +class PointTransformerLayer(nn.Module): + def __init__(self, in_planes, out_planes, share_planes=8, nsample=16): + super().__init__() + self.mid_planes = mid_planes = out_planes // 1 + self.out_planes = out_planes + self.share_planes = share_planes + self.nsample = nsample + self.linear_q = nn.Linear(in_planes, mid_planes) + self.linear_k = nn.Linear(in_planes, mid_planes) + self.linear_v = nn.Linear(in_planes, out_planes) + self.linear_p = nn.Sequential( + nn.Linear(3, 3), + LayerNorm1d(3), + nn.ReLU(inplace=True), + nn.Linear(3, out_planes), + ) + self.linear_w = nn.Sequential( + LayerNorm1d(mid_planes), + nn.ReLU(inplace=True), + nn.Linear(mid_planes, out_planes // share_planes), + LayerNorm1d(out_planes // share_planes), + nn.ReLU(inplace=True), + nn.Linear(out_planes // share_planes, out_planes // share_planes), + ) + self.softmax = nn.Softmax(dim=1) + + def forward(self, pxo) -> torch.Tensor: + p, x, o = pxo # (n, 3), (n, c), (b) + x_q, x_k, x_v = self.linear_q(x), self.linear_k(x), self.linear_v(x) + x_k, idx = pointops.knn_query_and_group( + x_k, p, o, new_xyz=p, new_offset=o, nsample=self.nsample, with_xyz=True + ) + x_v, _ = pointops.knn_query_and_group( + x_v, + p, + o, + new_xyz=p, + new_offset=o, + idx=idx, + nsample=self.nsample, + with_xyz=False, + ) + p_r, x_k = x_k[:, :, 0:3], x_k[:, :, 3:] + p_r = self.linear_p(p_r) + r_qk = ( + x_k + - x_q.unsqueeze(1) + + einops.reduce( + p_r, "n ns (i j) -> n ns j", reduction="sum", j=self.mid_planes + ) + ) + w = self.linear_w(r_qk) # (n, nsample, c) + w = self.softmax(w) + x = torch.einsum( + "n t s i, n t i -> n s i", + einops.rearrange(x_v + p_r, "n ns (s i) -> n ns s i", s=self.share_planes), + w, + ) + x = einops.rearrange(x, "n s i -> n (s i)") + return x + + +class TransitionDown(nn.Module): + def __init__(self, in_planes, out_planes, stride=1, nsample=16): + super().__init__() + self.stride, self.nsample = stride, nsample + if stride != 1: + self.linear = nn.Linear(3 + in_planes, out_planes, bias=False) + self.pool = nn.MaxPool1d(nsample) + else: + self.linear = nn.Linear(in_planes, out_planes, bias=False) + self.bn = nn.BatchNorm1d(out_planes) + self.relu = nn.ReLU(inplace=True) + + def forward(self, pxo): + p, x, o = pxo # (n, 3), (n, c), (b) + if self.stride != 1: + n_o, count = [o[0].item() // self.stride], o[0].item() // self.stride + for i in range(1, o.shape[0]): + count += (o[i].item() - o[i - 1].item()) // self.stride + n_o.append(count) + n_o = torch.cuda.IntTensor(n_o) + idx = pointops.farthest_point_sampling(p, o, n_o) # (m) + n_p = p[idx.long(), :] # (m, 3) + x, _ = pointops.knn_query_and_group( + x, + p, + offset=o, + new_xyz=n_p, + new_offset=n_o, + nsample=self.nsample, + with_xyz=True, + ) + x = self.relu( + self.bn(self.linear(x).transpose(1, 2).contiguous()) + ) # (m, c, nsample) + x = self.pool(x).squeeze(-1) # (m, c) + p, o = n_p, n_o + else: + x = self.relu(self.bn(self.linear(x))) # (n, c) + return [p, x, o] + + +class TransitionUp(nn.Module): + def __init__(self, in_planes, out_planes=None, num_shape_class=None): + super().__init__() + if out_planes is None: + self.num_shape_class = num_shape_class + if num_shape_class is not None: + self.linear1 = nn.Sequential( + nn.Linear(2 * in_planes + 1024, in_planes), + nn.BatchNorm1d(in_planes), + nn.ReLU(inplace=True), + ) + else: + self.linear1 = nn.Sequential( + nn.Linear(2 * in_planes, in_planes), + nn.BatchNorm1d(in_planes), + nn.ReLU(inplace=True), + ) + + self.linear2 = nn.Sequential( + nn.Linear(in_planes, in_planes), nn.ReLU(inplace=True) + ) + if num_shape_class is not None: + self.linear3 = nn.Sequential( + nn.Linear(num_shape_class, 1024), nn.ReLU(inplace=True) + ) + else: + self.linear1 = nn.Sequential( + nn.Linear(out_planes, out_planes), + nn.BatchNorm1d(out_planes), + nn.ReLU(inplace=True), + ) + self.linear2 = nn.Sequential( + nn.Linear(in_planes, out_planes), + nn.BatchNorm1d(out_planes), + nn.ReLU(inplace=True), + ) + + def forward(self, pxo1, pxo2=None, y=None): + if pxo2 is None: + _, x, o = pxo1 # (n, 3), (n, c), (b) + x_tmp = [] + for i in range(o.shape[0]): + if i == 0: + s_i, e_i, cnt = 0, o[0], o[0] + else: + s_i, e_i, cnt = o[i - 1], o[i], o[i] - o[i - 1] + x_b = x[s_i:e_i, :] + y_b = y[i].unsqueeze(-1).unsqueeze(-1).long() + y_onehot = torch.zeros(1, self.num_shape_class).cuda() # (1, l) + y_onehot.scatter_(1, y_b, 1) # (1, l) + x_b = torch.cat( + ( + x_b, + self.linear2(x_b.sum(0, True) / cnt).repeat(cnt, 1), + self.linear3(y_onehot).repeat(cnt, 1), + ), + dim=1, + ) + x_tmp.append(x_b) + x = torch.cat(x_tmp, 0) + x = self.linear1(x) + else: + p1, x1, o1 = pxo1 + p2, x2, o2 = pxo2 + x = self.linear1(x1) + pointops.interpolation( + p2, p1, self.linear2(x2), o2, o1 + ) + return x + + +class Bottleneck(nn.Module): + expansion = 1 + + def __init__(self, in_planes, planes, share_planes=8, nsample=16): + super(Bottleneck, self).__init__() + self.linear1 = nn.Linear(in_planes, planes, bias=False) + self.bn1 = nn.BatchNorm1d(planes) + self.transformer = PointTransformerLayer(planes, planes, share_planes, nsample) + self.bn2 = nn.BatchNorm1d(planes) + self.linear3 = nn.Linear(planes, planes * self.expansion, bias=False) + self.bn3 = nn.BatchNorm1d(planes * self.expansion) + self.relu = nn.ReLU(inplace=True) + + def forward(self, pxo): + p, x, o = pxo # (n, 3), (n, c), (b) + identity = x + x = self.relu(self.bn1(self.linear1(x))) + x = self.relu(self.bn2(self.transformer([p, x, o]))) + x = self.bn3(self.linear3(x)) + x += identity + x = self.relu(x) + return [p, x, o] + + +class PointTransformerSeg(nn.Module): + def __init__( + self, block, blocks, in_channels=6, num_classes=50, num_shape_classes=None + ): + super().__init__() + self.in_channels = in_channels + self.num_classes = num_classes + self.num_shape_classes = num_shape_classes + self.in_planes, planes = in_channels, [32, 64, 128, 256, 512] + fpn_planes, fpnhead_planes, share_planes = 128, 64, 8 + stride, nsample = [1, 4, 4, 4, 4], [8, 16, 16, 16, 16] + self.enc1 = self._make_enc( + block, + planes[0], + blocks[0], + share_planes, + stride=stride[0], + nsample=nsample[0], + ) # N/1 + self.enc2 = self._make_enc( + block, + planes[1], + blocks[1], + share_planes, + stride=stride[1], + nsample=nsample[1], + ) # N/4 + self.enc3 = self._make_enc( + block, + planes[2], + blocks[2], + share_planes, + stride=stride[2], + nsample=nsample[2], + ) # N/16 + self.enc4 = self._make_enc( + block, + planes[3], + blocks[3], + share_planes, + stride=stride[3], + nsample=nsample[3], + ) # N/64 + self.enc5 = self._make_enc( + block, + planes[4], + blocks[4], + share_planes, + stride=stride[4], + nsample=nsample[4], + ) # N/256 + self.dec5 = self._make_dec( + block, + planes[4], + 1, + share_planes, + num_shape_classes=num_shape_classes, + nsample=nsample[4], + is_head=True, + ) # transform p5 + self.dec4 = self._make_dec( + block, planes[3], 1, share_planes, nsample=nsample[3] + ) # fusion p5 and p4 + self.dec3 = self._make_dec( + block, planes[2], 1, share_planes, nsample=nsample[2] + ) # fusion p4 and p3 + self.dec2 = self._make_dec( + block, planes[1], 1, share_planes, nsample=nsample[1] + ) # fusion p3 and p2 + self.dec1 = self._make_dec( + block, planes[0], 1, share_planes, nsample=nsample[0] + ) # fusion p2 and p1 + self.cls = nn.Sequential( + nn.Linear(planes[0], planes[0]), + nn.BatchNorm1d(planes[0]), + nn.ReLU(inplace=True), + nn.Linear(planes[0], num_classes), + ) + + def _make_enc(self, block, planes, blocks, share_planes=8, stride=1, nsample=16): + layers = [ + TransitionDown(self.in_planes, planes * block.expansion, stride, nsample) + ] + self.in_planes = planes * block.expansion + for _ in range(blocks): + layers.append( + block(self.in_planes, self.in_planes, share_planes, nsample=nsample) + ) + return nn.Sequential(*layers) + + def _make_dec( + self, + block, + planes, + blocks, + share_planes=8, + num_shape_classes=None, + nsample=16, + is_head=False, + ): + layers = [ + TransitionUp( + self.in_planes, + None if is_head else planes * block.expansion, + num_shape_classes, + ) + ] + self.in_planes = planes * block.expansion + for _ in range(blocks): + layers.append( + block(self.in_planes, self.in_planes, share_planes, nsample=nsample) + ) + return nn.Sequential(*layers) + + def forward(self, data_dict): + p0 = data_dict["coord"] + x0 = data_dict["feat"] + o0 = data_dict["offset"].int() + if self.num_shape_classes is not None: + y = data_dict["cls_token"] + p1, x1, o1 = self.enc1([p0, x0, o0]) + p2, x2, o2 = self.enc2([p1, x1, o1]) + p3, x3, o3 = self.enc3([p2, x2, o2]) + p4, x4, o4 = self.enc4([p3, x3, o3]) + p5, x5, o5 = self.enc5([p4, x4, o4]) + if self.num_shape_classes is not None: + x5 = self.dec5[1:]([p5, self.dec5[0]([p5, x5, o5], y=y), o5])[1] + else: + x5 = self.dec5[1:]([p5, self.dec5[0]([p5, x5, o5]), o5])[1] + x4 = self.dec4[1:]([p4, self.dec4[0]([p4, x4, o4], [p5, x5, o5]), o4])[1] + x3 = self.dec3[1:]([p3, self.dec3[0]([p3, x3, o3], [p4, x4, o4]), o3])[1] + x2 = self.dec2[1:]([p2, self.dec2[0]([p2, x2, o2], [p3, x3, o3]), o2])[1] + x1 = self.dec1[1:]([p1, self.dec1[0]([p1, x1, o1], [p2, x2, o2]), o1])[1] + x = self.cls(x1) + return x + + +@MODELS.register_module("PointTransformer-PartSeg26") +class PointTransformerSeg26(PointTransformerSeg): + def __init__(self, **kwargs): + super(PointTransformerSeg26, self).__init__( + Bottleneck, [1, 1, 1, 1, 1], **kwargs + ) + + +@MODELS.register_module("PointTransformer-PartSeg38") +class PointTransformerSeg38(PointTransformerSeg): + def __init__(self, **kwargs): + super(PointTransformerSeg38, self).__init__( + Bottleneck, [1, 2, 2, 2, 2], **kwargs + ) + + +@MODELS.register_module("PointTransformer-PartSeg50") +class PointTransformerSeg50(PointTransformerSeg): + def __init__(self, **kwargs): + super(PointTransformerSeg50, self).__init__( + Bottleneck, [1, 2, 3, 5, 2], **kwargs + ) diff --git a/pointcept/models/point_transformer/point_transformer_seg.py b/pointcept/models/point_transformer/point_transformer_seg.py new file mode 100644 index 0000000000000000000000000000000000000000..248cacad1ade65e48fa4686560fb40617a0ea449 --- /dev/null +++ b/pointcept/models/point_transformer/point_transformer_seg.py @@ -0,0 +1,327 @@ +""" +Point Transformer V1 for Semantic Segmentation + +Might be a bit different from the original paper + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +import torch.nn as nn +import einops +import pointops + +from pointcept.models.builder import MODELS +from .utils import LayerNorm1d + + +class PointTransformerLayer(nn.Module): + def __init__(self, in_planes, out_planes, share_planes=8, nsample=16): + super().__init__() + self.mid_planes = mid_planes = out_planes // 1 + self.out_planes = out_planes + self.share_planes = share_planes + self.nsample = nsample + self.linear_q = nn.Linear(in_planes, mid_planes) + self.linear_k = nn.Linear(in_planes, mid_planes) + self.linear_v = nn.Linear(in_planes, out_planes) + self.linear_p = nn.Sequential( + nn.Linear(3, 3), + LayerNorm1d(3), + nn.ReLU(inplace=True), + nn.Linear(3, out_planes), + ) + self.linear_w = nn.Sequential( + LayerNorm1d(mid_planes), + nn.ReLU(inplace=True), + nn.Linear(mid_planes, out_planes // share_planes), + LayerNorm1d(out_planes // share_planes), + nn.ReLU(inplace=True), + nn.Linear(out_planes // share_planes, out_planes // share_planes), + ) + self.softmax = nn.Softmax(dim=1) + + def forward(self, pxo) -> torch.Tensor: + p, x, o = pxo # (n, 3), (n, c), (b) + x_q, x_k, x_v = self.linear_q(x), self.linear_k(x), self.linear_v(x) + x_k, idx = pointops.knn_query_and_group( + x_k, p, o, new_xyz=p, new_offset=o, nsample=self.nsample, with_xyz=True + ) + x_v, _ = pointops.knn_query_and_group( + x_v, + p, + o, + new_xyz=p, + new_offset=o, + idx=idx, + nsample=self.nsample, + with_xyz=False, + ) + p_r, x_k = x_k[:, :, 0:3], x_k[:, :, 3:] + p_r = self.linear_p(p_r) + r_qk = ( + x_k + - x_q.unsqueeze(1) + + einops.reduce( + p_r, "n ns (i j) -> n ns j", reduction="sum", j=self.mid_planes + ) + ) + w = self.linear_w(r_qk) # (n, nsample, c) + w = self.softmax(w) + x = torch.einsum( + "n t s i, n t i -> n s i", + einops.rearrange(x_v + p_r, "n ns (s i) -> n ns s i", s=self.share_planes), + w, + ) + x = einops.rearrange(x, "n s i -> n (s i)") + return x + + +class TransitionDown(nn.Module): + def __init__(self, in_planes, out_planes, stride=1, nsample=16): + super().__init__() + self.stride, self.nsample = stride, nsample + if stride != 1: + self.linear = nn.Linear(3 + in_planes, out_planes, bias=False) + self.pool = nn.MaxPool1d(nsample) + else: + self.linear = nn.Linear(in_planes, out_planes, bias=False) + self.bn = nn.BatchNorm1d(out_planes) + self.relu = nn.ReLU(inplace=True) + + def forward(self, pxo): + p, x, o = pxo # (n, 3), (n, c), (b) + if self.stride != 1: + n_o, count = [o[0].item() // self.stride], o[0].item() // self.stride + for i in range(1, o.shape[0]): + count += (o[i].item() - o[i - 1].item()) // self.stride + n_o.append(count) + n_o = torch.cuda.IntTensor(n_o) + idx = pointops.farthest_point_sampling(p, o, n_o) # (m) + n_p = p[idx.long(), :] # (m, 3) + x, _ = pointops.knn_query_and_group( + x, + p, + offset=o, + new_xyz=n_p, + new_offset=n_o, + nsample=self.nsample, + with_xyz=True, + ) + x = self.relu( + self.bn(self.linear(x).transpose(1, 2).contiguous()) + ) # (m, c, nsample) + x = self.pool(x).squeeze(-1) # (m, c) + p, o = n_p, n_o + else: + x = self.relu(self.bn(self.linear(x))) # (n, c) + return [p, x, o] + + +class TransitionUp(nn.Module): + def __init__(self, in_planes, out_planes=None): + super().__init__() + if out_planes is None: + self.linear1 = nn.Sequential( + nn.Linear(2 * in_planes, in_planes), + nn.BatchNorm1d(in_planes), + nn.ReLU(inplace=True), + ) + self.linear2 = nn.Sequential( + nn.Linear(in_planes, in_planes), nn.ReLU(inplace=True) + ) + else: + self.linear1 = nn.Sequential( + nn.Linear(out_planes, out_planes), + nn.BatchNorm1d(out_planes), + nn.ReLU(inplace=True), + ) + self.linear2 = nn.Sequential( + nn.Linear(in_planes, out_planes), + nn.BatchNorm1d(out_planes), + nn.ReLU(inplace=True), + ) + + def forward(self, pxo1, pxo2=None): + if pxo2 is None: + _, x, o = pxo1 # (n, 3), (n, c), (b) + x_tmp = [] + for i in range(o.shape[0]): + if i == 0: + s_i, e_i, cnt = 0, o[0], o[0] + else: + s_i, e_i, cnt = o[i - 1], o[i], o[i] - o[i - 1] + x_b = x[s_i:e_i, :] + x_b = torch.cat( + (x_b, self.linear2(x_b.sum(0, True) / cnt).repeat(cnt, 1)), 1 + ) + x_tmp.append(x_b) + x = torch.cat(x_tmp, 0) + x = self.linear1(x) + else: + p1, x1, o1 = pxo1 + p2, x2, o2 = pxo2 + x = self.linear1(x1) + pointops.interpolation( + p2, p1, self.linear2(x2), o2, o1 + ) + return x + + +class Bottleneck(nn.Module): + expansion = 1 + + def __init__(self, in_planes, planes, share_planes=8, nsample=16): + super(Bottleneck, self).__init__() + self.linear1 = nn.Linear(in_planes, planes, bias=False) + self.bn1 = nn.BatchNorm1d(planes) + self.transformer = PointTransformerLayer(planes, planes, share_planes, nsample) + self.bn2 = nn.BatchNorm1d(planes) + self.linear3 = nn.Linear(planes, planes * self.expansion, bias=False) + self.bn3 = nn.BatchNorm1d(planes * self.expansion) + self.relu = nn.ReLU(inplace=True) + + def forward(self, pxo): + p, x, o = pxo # (n, 3), (n, c), (b) + identity = x + x = self.relu(self.bn1(self.linear1(x))) + x = self.relu(self.bn2(self.transformer([p, x, o]))) + x = self.bn3(self.linear3(x)) + x += identity + x = self.relu(x) + return [p, x, o] + + +class PointTransformerSeg(nn.Module): + def __init__(self, block, blocks, in_channels=6, num_classes=13): + super().__init__() + self.in_channels = in_channels + self.in_planes, planes = in_channels, [32, 64, 128, 256, 512] + fpn_planes, fpnhead_planes, share_planes = 128, 64, 8 + stride, nsample = [1, 4, 4, 4, 4], [8, 16, 16, 16, 16] + self.enc1 = self._make_enc( + block, + planes[0], + blocks[0], + share_planes, + stride=stride[0], + nsample=nsample[0], + ) # N/1 + self.enc2 = self._make_enc( + block, + planes[1], + blocks[1], + share_planes, + stride=stride[1], + nsample=nsample[1], + ) # N/4 + self.enc3 = self._make_enc( + block, + planes[2], + blocks[2], + share_planes, + stride=stride[2], + nsample=nsample[2], + ) # N/16 + self.enc4 = self._make_enc( + block, + planes[3], + blocks[3], + share_planes, + stride=stride[3], + nsample=nsample[3], + ) # N/64 + self.enc5 = self._make_enc( + block, + planes[4], + blocks[4], + share_planes, + stride=stride[4], + nsample=nsample[4], + ) # N/256 + self.dec5 = self._make_dec( + block, planes[4], 1, share_planes, nsample=nsample[4], is_head=True + ) # transform p5 + self.dec4 = self._make_dec( + block, planes[3], 1, share_planes, nsample=nsample[3] + ) # fusion p5 and p4 + self.dec3 = self._make_dec( + block, planes[2], 1, share_planes, nsample=nsample[2] + ) # fusion p4 and p3 + self.dec2 = self._make_dec( + block, planes[1], 1, share_planes, nsample=nsample[1] + ) # fusion p3 and p2 + self.dec1 = self._make_dec( + block, planes[0], 1, share_planes, nsample=nsample[0] + ) # fusion p2 and p1 + self.cls = nn.Sequential( + nn.Linear(planes[0], planes[0]), + nn.BatchNorm1d(planes[0]), + nn.ReLU(inplace=True), + nn.Linear(planes[0], num_classes), + ) + + def _make_enc(self, block, planes, blocks, share_planes=8, stride=1, nsample=16): + layers = [ + TransitionDown(self.in_planes, planes * block.expansion, stride, nsample) + ] + self.in_planes = planes * block.expansion + for _ in range(blocks): + layers.append( + block(self.in_planes, self.in_planes, share_planes, nsample=nsample) + ) + return nn.Sequential(*layers) + + def _make_dec( + self, block, planes, blocks, share_planes=8, nsample=16, is_head=False + ): + layers = [ + TransitionUp(self.in_planes, None if is_head else planes * block.expansion) + ] + self.in_planes = planes * block.expansion + for _ in range(blocks): + layers.append( + block(self.in_planes, self.in_planes, share_planes, nsample=nsample) + ) + return nn.Sequential(*layers) + + def forward(self, data_dict): + p0 = data_dict["coord"] + x0 = data_dict["feat"] + o0 = data_dict["offset"].int() + p1, x1, o1 = self.enc1([p0, x0, o0]) + p2, x2, o2 = self.enc2([p1, x1, o1]) + p3, x3, o3 = self.enc3([p2, x2, o2]) + p4, x4, o4 = self.enc4([p3, x3, o3]) + p5, x5, o5 = self.enc5([p4, x4, o4]) + x5 = self.dec5[1:]([p5, self.dec5[0]([p5, x5, o5]), o5])[1] + x4 = self.dec4[1:]([p4, self.dec4[0]([p4, x4, o4], [p5, x5, o5]), o4])[1] + x3 = self.dec3[1:]([p3, self.dec3[0]([p3, x3, o3], [p4, x4, o4]), o3])[1] + x2 = self.dec2[1:]([p2, self.dec2[0]([p2, x2, o2], [p3, x3, o3]), o2])[1] + x1 = self.dec1[1:]([p1, self.dec1[0]([p1, x1, o1], [p2, x2, o2]), o1])[1] + x = self.cls(x1) + return x + + +@MODELS.register_module("PointTransformer-Seg26") +class PointTransformerSeg26(PointTransformerSeg): + def __init__(self, **kwargs): + super(PointTransformerSeg26, self).__init__( + Bottleneck, [1, 1, 1, 1, 1], **kwargs + ) + + +@MODELS.register_module("PointTransformer-Seg38") +class PointTransformerSeg38(PointTransformerSeg): + def __init__(self, **kwargs): + super(PointTransformerSeg38, self).__init__( + Bottleneck, [1, 2, 2, 2, 2], **kwargs + ) + + +@MODELS.register_module("PointTransformer-Seg50") +class PointTransformerSeg50(PointTransformerSeg): + def __init__(self, **kwargs): + super(PointTransformerSeg50, self).__init__( + Bottleneck, [1, 2, 3, 5, 2], **kwargs + ) diff --git a/pointcept/models/point_transformer/utils.py b/pointcept/models/point_transformer/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..c5687701835bb1f8a8936ea5ae5d52285567dc77 --- /dev/null +++ b/pointcept/models/point_transformer/utils.py @@ -0,0 +1,14 @@ +import torch +import torch.nn as nn + +torch.nn.LayerNorm + + +class LayerNorm1d(nn.BatchNorm1d): + def forward(self, input: torch.Tensor) -> torch.Tensor: + return ( + super() + .forward(input.transpose(1, 2).contiguous()) + .transpose(1, 2) + .contiguous() + ) diff --git a/pointcept/models/point_transformer_v2/__init__.py b/pointcept/models/point_transformer_v2/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e9689fa2518b599bc6f94e6f8d0ea461859b8909 --- /dev/null +++ b/pointcept/models/point_transformer_v2/__init__.py @@ -0,0 +1,10 @@ +""" +Point Transformer V2 + +Copyright (c) Xiaoyang Wu (xiaoyang.wu@connect.hku.hk). All Rights Reserved. +Please cite our work if you use any part of the code. +""" + +from .point_transformer_v2m1_origin import * +from .point_transformer_v2m2_base import * +from .point_transformer_v2m3_pdnorm import * diff --git a/pointcept/models/point_transformer_v2/point_transformer_v2m1_origin.py b/pointcept/models/point_transformer_v2/point_transformer_v2m1_origin.py new file mode 100644 index 0000000000000000000000000000000000000000..24b6386895b37b5fb09fde38afe6f013852e2162 --- /dev/null +++ b/pointcept/models/point_transformer_v2/point_transformer_v2m1_origin.py @@ -0,0 +1,614 @@ +""" +Point Transformer V2 mode 1 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from copy import deepcopy +import math +import torch +import torch.nn as nn +from torch.utils.checkpoint import checkpoint +from torch_geometric.nn.pool import voxel_grid +from torch_scatter import segment_csr + +import einops +from timm.layers import DropPath +import pointops + +from pointcept.models.builder import MODELS +from pointcept.models.utils import offset2batch, batch2offset + + +class GroupedLinear(nn.Module): + __constants__ = ["in_features", "out_features", "groups"] + in_features: int + out_features: int + groups: int + weight: torch.Tensor + + def __init__( + self, in_features: int, out_features: int, groups: int, device=None, dtype=None + ) -> None: + factory_kwargs = {"device": device, "dtype": dtype} + super(GroupedLinear, self).__init__() + self.in_features = in_features + self.out_features = out_features + self.groups = groups + assert in_features & groups == 0 + assert out_features % groups == 0 + # for convenient, currently only support out_features == groups, one output + assert out_features == groups + self.weight = nn.Parameter(torch.empty((1, in_features), **factory_kwargs)) + self.reset_parameters() + + def reset_parameters(self) -> None: + nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5)) + + def forward(self, input: torch.Tensor) -> torch.Tensor: + return ( + (input * self.weight) + .reshape( + list(input.shape[:-1]) + [self.groups, input.shape[-1] // self.groups] + ) + .sum(-1) + ) + + def extra_repr(self) -> str: + return "in_features={}, out_features={}, bias={}".format( + self.in_features, self.out_features, self.bias is not None + ) + + +class PointBatchNorm(nn.Module): + """ + Batch Normalization for Point Clouds data in shape of [B*N, C], [B*N, L, C] + """ + + def __init__(self, embed_channels): + super().__init__() + self.norm = nn.BatchNorm1d(embed_channels) + + def forward(self, input: torch.Tensor) -> torch.Tensor: + if input.dim() == 3: + return ( + self.norm(input.transpose(1, 2).contiguous()) + .transpose(1, 2) + .contiguous() + ) + elif input.dim() == 2: + return self.norm(input) + else: + raise NotImplementedError + + +class GroupedVectorAttention(nn.Module): + def __init__( + self, + embed_channels, + groups, + attn_drop_rate=0.0, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + ): + super(GroupedVectorAttention, self).__init__() + self.embed_channels = embed_channels + self.groups = groups + assert embed_channels % groups == 0 + self.attn_drop_rate = attn_drop_rate + self.qkv_bias = qkv_bias + self.pe_multiplier = pe_multiplier + self.pe_bias = pe_bias + + self.linear_q = nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=qkv_bias), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + ) + self.linear_k = nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=qkv_bias), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + ) + + self.linear_v = nn.Linear(embed_channels, embed_channels, bias=qkv_bias) + + if self.pe_multiplier: + self.linear_p_multiplier = nn.Sequential( + nn.Linear(3, embed_channels), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + nn.Linear(embed_channels, embed_channels), + ) + if self.pe_bias: + self.linear_p_bias = nn.Sequential( + nn.Linear(3, embed_channels), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + nn.Linear(embed_channels, embed_channels), + ) + self.weight_encoding = nn.Sequential( + GroupedLinear(embed_channels, groups, groups), + PointBatchNorm(groups), + nn.ReLU(inplace=True), + nn.Linear(groups, groups), + ) + self.softmax = nn.Softmax(dim=1) + self.attn_drop = nn.Dropout(attn_drop_rate) + + def forward(self, feat, coord, reference_index): + query, key, value = ( + self.linear_q(feat), + self.linear_k(feat), + self.linear_v(feat), + ) + key = pointops.grouping(reference_index, key, coord, with_xyz=True) + value = pointops.grouping(reference_index, value, coord, with_xyz=False) + pos, key = key[:, :, 0:3], key[:, :, 3:] + relation_qk = key - query.unsqueeze(1) + if self.pe_multiplier: + pem = self.linear_p_multiplier(pos) + relation_qk = relation_qk * pem + if self.pe_bias: + peb = self.linear_p_bias(pos) + relation_qk = relation_qk + peb + value = value + peb + + weight = self.weight_encoding(relation_qk) + weight = self.attn_drop(self.softmax(weight)) + + mask = torch.sign(reference_index + 1) + weight = torch.einsum("n s g, n s -> n s g", weight, mask) + value = einops.rearrange(value, "n ns (g i) -> n ns g i", g=self.groups) + feat = torch.einsum("n s g i, n s g -> n g i", value, weight) + feat = einops.rearrange(feat, "n g i -> n (g i)") + return feat + + +class Block(nn.Module): + def __init__( + self, + embed_channels, + groups, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(Block, self).__init__() + self.attn = GroupedVectorAttention( + embed_channels=embed_channels, + groups=groups, + qkv_bias=qkv_bias, + attn_drop_rate=attn_drop_rate, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + ) + self.fc1 = nn.Linear(embed_channels, embed_channels, bias=False) + self.fc3 = nn.Linear(embed_channels, embed_channels, bias=False) + self.norm1 = PointBatchNorm(embed_channels) + self.norm2 = PointBatchNorm(embed_channels) + self.norm3 = PointBatchNorm(embed_channels) + self.act = nn.ReLU(inplace=True) + self.enable_checkpoint = enable_checkpoint + self.drop_path = ( + DropPath(drop_path_rate) if drop_path_rate > 0.0 else nn.Identity() + ) + + def forward(self, points, reference_index): + coord, feat, offset = points + identity = feat + feat = self.act(self.norm1(self.fc1(feat))) + feat = ( + self.attn(feat, coord, reference_index) + if not self.enable_checkpoint + else checkpoint(self.attn, feat, coord, reference_index) + ) + feat = self.act(self.norm2(feat)) + feat = self.norm3(self.fc3(feat)) + feat = identity + self.drop_path(feat) + feat = self.act(feat) + return [coord, feat, offset] + + +class BlockSequence(nn.Module): + def __init__( + self, + depth, + embed_channels, + groups, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(BlockSequence, self).__init__() + + if isinstance(drop_path_rate, list): + drop_path_rates = drop_path_rate + assert len(drop_path_rates) == depth + elif isinstance(drop_path_rate, float): + drop_path_rates = [deepcopy(drop_path_rate) for _ in range(depth)] + else: + drop_path_rates = [0.0 for _ in range(depth)] + + self.neighbours = neighbours + self.blocks = nn.ModuleList() + for i in range(depth): + block = Block( + embed_channels=embed_channels, + groups=groups, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=drop_path_rates[i], + enable_checkpoint=enable_checkpoint, + ) + self.blocks.append(block) + + def forward(self, points): + coord, feat, offset = points + # reference index query of neighbourhood attention + # for windows attention, modify reference index query method + reference_index, _ = pointops.knn_query(self.neighbours, coord, offset) + for block in self.blocks: + points = block(points, reference_index) + return points + + +class GridPool(nn.Module): + """ + Partition-based Pooling (Grid Pooling) + """ + + def __init__(self, in_channels, out_channels, grid_size, bias=False): + super(GridPool, self).__init__() + self.in_channels = in_channels + self.out_channels = out_channels + self.grid_size = grid_size + + self.fc = nn.Linear(in_channels, out_channels, bias=bias) + self.norm = PointBatchNorm(out_channels) + self.act = nn.ReLU(inplace=True) + + def forward(self, points, start=None): + coord, feat, offset = points + batch = offset2batch(offset) + feat = self.act(self.norm(self.fc(feat))) + start = ( + segment_csr( + coord, + torch.cat([batch.new_zeros(1), torch.cumsum(batch.bincount(), dim=0)]), + reduce="min", + ) + if start is None + else start + ) + cluster = voxel_grid( + pos=coord - start[batch], size=self.grid_size, batch=batch, start=0 + ) + unique, cluster, counts = torch.unique( + cluster, sorted=True, return_inverse=True, return_counts=True + ) + _, sorted_cluster_indices = torch.sort(cluster) + idx_ptr = torch.cat([counts.new_zeros(1), torch.cumsum(counts, dim=0)]) + coord = segment_csr(coord[sorted_cluster_indices], idx_ptr, reduce="mean") + feat = segment_csr(feat[sorted_cluster_indices], idx_ptr, reduce="max") + batch = batch[idx_ptr[:-1]] + offset = batch2offset(batch) + return [coord, feat, offset], cluster + + +class UnpoolWithSkip(nn.Module): + """ + Map Unpooling with skip connection + """ + + def __init__( + self, + in_channels, + skip_channels, + out_channels, + bias=True, + skip=True, + backend="map", + ): + super(UnpoolWithSkip, self).__init__() + self.in_channels = in_channels + self.skip_channels = skip_channels + self.out_channels = out_channels + self.skip = skip + self.backend = backend + assert self.backend in ["map", "interp"] + + self.proj = nn.Sequential( + nn.Linear(in_channels, out_channels, bias=bias), + PointBatchNorm(out_channels), + nn.ReLU(inplace=True), + ) + self.proj_skip = nn.Sequential( + nn.Linear(skip_channels, out_channels, bias=bias), + PointBatchNorm(out_channels), + nn.ReLU(inplace=True), + ) + + def forward(self, points, skip_points, cluster=None): + coord, feat, offset = points + skip_coord, skip_feat, skip_offset = skip_points + if self.backend == "map" and cluster is not None: + feat = self.proj(feat)[cluster] + else: + feat = pointops.interpolation( + coord, skip_coord, self.proj(feat), offset, skip_offset + ) + if self.skip: + feat = feat + self.proj_skip(skip_feat) + return [skip_coord, feat, skip_offset] + + +class Encoder(nn.Module): + def __init__( + self, + depth, + in_channels, + embed_channels, + groups, + grid_size=None, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=None, + drop_path_rate=None, + enable_checkpoint=False, + ): + super(Encoder, self).__init__() + + self.down = GridPool( + in_channels=in_channels, + out_channels=embed_channels, + grid_size=grid_size, + ) + + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate if attn_drop_rate is not None else 0.0, + drop_path_rate=drop_path_rate if drop_path_rate is not None else 0.0, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points): + points, cluster = self.down(points) + return self.blocks(points), cluster + + +class Decoder(nn.Module): + def __init__( + self, + in_channels, + skip_channels, + embed_channels, + groups, + depth, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=None, + drop_path_rate=None, + enable_checkpoint=False, + unpool_backend="map", + ): + super(Decoder, self).__init__() + + self.up = UnpoolWithSkip( + in_channels=in_channels, + out_channels=embed_channels, + skip_channels=skip_channels, + backend=unpool_backend, + ) + + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate if attn_drop_rate is not None else 0.0, + drop_path_rate=drop_path_rate if drop_path_rate is not None else 0.0, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points, skip_points, cluster): + points = self.up(points, skip_points, cluster) + return self.blocks(points) + + +class GVAPatchEmbed(nn.Module): + def __init__( + self, + depth, + in_channels, + embed_channels, + groups, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(GVAPatchEmbed, self).__init__() + self.in_channels = in_channels + self.embed_channels = embed_channels + self.proj = nn.Sequential( + nn.Linear(in_channels, embed_channels, bias=False), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + ) + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=drop_path_rate, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points): + coord, feat, offset = points + feat = self.proj(feat) + return self.blocks([coord, feat, offset]) + + +@MODELS.register_module("PT-v2m1") +class PointTransformerV2(nn.Module): + def __init__( + self, + in_channels, + num_classes, + patch_embed_depth=1, + patch_embed_channels=48, + patch_embed_groups=6, + patch_embed_neighbours=8, + enc_depths=(2, 2, 6, 2), + enc_channels=(96, 192, 384, 512), + enc_groups=(12, 24, 48, 64), + enc_neighbours=(16, 16, 16, 16), + dec_depths=(1, 1, 1, 1), + dec_channels=(48, 96, 192, 384), + dec_groups=(6, 12, 24, 48), + dec_neighbours=(16, 16, 16, 16), + grid_sizes=(0.06, 0.12, 0.24, 0.48), + attn_qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0, + enable_checkpoint=False, + unpool_backend="map", + ): + super(PointTransformerV2, self).__init__() + self.in_channels = in_channels + self.num_classes = num_classes + self.num_stages = len(enc_depths) + assert self.num_stages == len(dec_depths) + assert self.num_stages == len(enc_channels) + assert self.num_stages == len(dec_channels) + assert self.num_stages == len(enc_groups) + assert self.num_stages == len(dec_groups) + assert self.num_stages == len(enc_neighbours) + assert self.num_stages == len(dec_neighbours) + assert self.num_stages == len(grid_sizes) + self.patch_embed = GVAPatchEmbed( + in_channels=in_channels, + embed_channels=patch_embed_channels, + groups=patch_embed_groups, + depth=patch_embed_depth, + neighbours=patch_embed_neighbours, + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + enable_checkpoint=enable_checkpoint, + ) + + enc_dp_rates = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(enc_depths)) + ] + dec_dp_rates = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(dec_depths)) + ] + enc_channels = [patch_embed_channels] + list(enc_channels) + dec_channels = list(dec_channels) + [enc_channels[-1]] + self.enc_stages = nn.ModuleList() + self.dec_stages = nn.ModuleList() + for i in range(self.num_stages): + enc = Encoder( + depth=enc_depths[i], + in_channels=enc_channels[i], + embed_channels=enc_channels[i + 1], + groups=enc_groups[i], + grid_size=grid_sizes[i], + neighbours=enc_neighbours[i], + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=enc_dp_rates[ + sum(enc_depths[:i]) : sum(enc_depths[: i + 1]) + ], + enable_checkpoint=enable_checkpoint, + ) + dec = Decoder( + depth=dec_depths[i], + in_channels=dec_channels[i + 1], + skip_channels=enc_channels[i], + embed_channels=dec_channels[i], + groups=dec_groups[i], + neighbours=dec_neighbours[i], + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=dec_dp_rates[ + sum(dec_depths[:i]) : sum(dec_depths[: i + 1]) + ], + enable_checkpoint=enable_checkpoint, + unpool_backend=unpool_backend, + ) + self.enc_stages.append(enc) + self.dec_stages.append(dec) + self.seg_head = ( + nn.Sequential( + nn.Linear(dec_channels[0], dec_channels[0]), + PointBatchNorm(dec_channels[0]), + nn.ReLU(inplace=True), + nn.Linear(dec_channels[0], num_classes), + ) + if num_classes > 0 + else nn.Identity() + ) + + def forward(self, data_dict): + coord = data_dict["coord"] + feat = data_dict["feat"] + offset = data_dict["offset"].int() + + # a batch of point cloud is a list of coord, feat and offset + points = [coord, feat, offset] + points = self.patch_embed(points) + skips = [[points]] + for i in range(self.num_stages): + points, cluster = self.enc_stages[i](points) + skips[-1].append(cluster) # record grid cluster of pooling + skips.append([points]) # record points info of current stage + + points = skips.pop(-1)[0] # unpooling points info in the last enc stage + for i in reversed(range(self.num_stages)): + skip_points, cluster = skips.pop(-1) + points = self.dec_stages[i](points, skip_points, cluster) + coord, feat, offset = points + seg_logits = self.seg_head(feat) + return seg_logits diff --git a/pointcept/models/point_transformer_v2/point_transformer_v2m2_base.py b/pointcept/models/point_transformer_v2/point_transformer_v2m2_base.py new file mode 100644 index 0000000000000000000000000000000000000000..db117590a9cd111fc9e5838117439b15c98115c3 --- /dev/null +++ b/pointcept/models/point_transformer_v2/point_transformer_v2m2_base.py @@ -0,0 +1,576 @@ +""" +Point Transformer V2 Mode 2 (recommend) + +Disable Grouped Linear + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from copy import deepcopy +import math +import torch +import torch.nn as nn +from torch.utils.checkpoint import checkpoint +from torch_geometric.nn.pool import voxel_grid +from torch_scatter import segment_csr + +import einops +from timm.layers import DropPath +import pointops + +from pointcept.models.builder import MODELS +from pointcept.models.utils import offset2batch, batch2offset + + +class PointBatchNorm(nn.Module): + """ + Batch Normalization for Point Clouds data in shape of [B*N, C], [B*N, L, C] + """ + + def __init__(self, embed_channels): + super().__init__() + self.norm = nn.BatchNorm1d(embed_channels) + + def forward(self, input: torch.Tensor) -> torch.Tensor: + if input.dim() == 3: + return ( + self.norm(input.transpose(1, 2).contiguous()) + .transpose(1, 2) + .contiguous() + ) + elif input.dim() == 2: + return self.norm(input) + else: + raise NotImplementedError + + +class GroupedVectorAttention(nn.Module): + def __init__( + self, + embed_channels, + groups, + attn_drop_rate=0.0, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + ): + super(GroupedVectorAttention, self).__init__() + self.embed_channels = embed_channels + self.groups = groups + assert embed_channels % groups == 0 + self.attn_drop_rate = attn_drop_rate + self.qkv_bias = qkv_bias + self.pe_multiplier = pe_multiplier + self.pe_bias = pe_bias + + self.linear_q = nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=qkv_bias), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + ) + self.linear_k = nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=qkv_bias), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + ) + + self.linear_v = nn.Linear(embed_channels, embed_channels, bias=qkv_bias) + + if self.pe_multiplier: + self.linear_p_multiplier = nn.Sequential( + nn.Linear(3, embed_channels), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + nn.Linear(embed_channels, embed_channels), + ) + if self.pe_bias: + self.linear_p_bias = nn.Sequential( + nn.Linear(3, embed_channels), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + nn.Linear(embed_channels, embed_channels), + ) + self.weight_encoding = nn.Sequential( + nn.Linear(embed_channels, groups), + PointBatchNorm(groups), + nn.ReLU(inplace=True), + nn.Linear(groups, groups), + ) + self.softmax = nn.Softmax(dim=1) + self.attn_drop = nn.Dropout(attn_drop_rate) + + def forward(self, feat, coord, reference_index): + query, key, value = ( + self.linear_q(feat), + self.linear_k(feat), + self.linear_v(feat), + ) + key = pointops.grouping(reference_index, key, coord, with_xyz=True) + value = pointops.grouping(reference_index, value, coord, with_xyz=False) + pos, key = key[:, :, 0:3], key[:, :, 3:] + relation_qk = key - query.unsqueeze(1) + if self.pe_multiplier: + pem = self.linear_p_multiplier(pos) + relation_qk = relation_qk * pem + if self.pe_bias: + peb = self.linear_p_bias(pos) + relation_qk = relation_qk + peb + value = value + peb + + weight = self.weight_encoding(relation_qk) + weight = self.attn_drop(self.softmax(weight)) + + mask = torch.sign(reference_index + 1) + weight = torch.einsum("n s g, n s -> n s g", weight, mask) + value = einops.rearrange(value, "n ns (g i) -> n ns g i", g=self.groups) + feat = torch.einsum("n s g i, n s g -> n g i", value, weight) + feat = einops.rearrange(feat, "n g i -> n (g i)") + return feat + + +class Block(nn.Module): + def __init__( + self, + embed_channels, + groups, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(Block, self).__init__() + self.attn = GroupedVectorAttention( + embed_channels=embed_channels, + groups=groups, + qkv_bias=qkv_bias, + attn_drop_rate=attn_drop_rate, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + ) + self.fc1 = nn.Linear(embed_channels, embed_channels, bias=False) + self.fc3 = nn.Linear(embed_channels, embed_channels, bias=False) + self.norm1 = PointBatchNorm(embed_channels) + self.norm2 = PointBatchNorm(embed_channels) + self.norm3 = PointBatchNorm(embed_channels) + self.act = nn.ReLU(inplace=True) + self.enable_checkpoint = enable_checkpoint + self.drop_path = ( + DropPath(drop_path_rate) if drop_path_rate > 0.0 else nn.Identity() + ) + + def forward(self, points, reference_index): + coord, feat, offset = points + identity = feat + feat = self.act(self.norm1(self.fc1(feat))) + feat = ( + self.attn(feat, coord, reference_index) + if not self.enable_checkpoint + else checkpoint(self.attn, feat, coord, reference_index) + ) + feat = self.act(self.norm2(feat)) + feat = self.norm3(self.fc3(feat)) + feat = identity + self.drop_path(feat) + feat = self.act(feat) + return [coord, feat, offset] + + +class BlockSequence(nn.Module): + def __init__( + self, + depth, + embed_channels, + groups, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(BlockSequence, self).__init__() + + if isinstance(drop_path_rate, list): + drop_path_rates = drop_path_rate + assert len(drop_path_rates) == depth + elif isinstance(drop_path_rate, float): + drop_path_rates = [deepcopy(drop_path_rate) for _ in range(depth)] + else: + drop_path_rates = [0.0 for _ in range(depth)] + + self.neighbours = neighbours + self.blocks = nn.ModuleList() + for i in range(depth): + block = Block( + embed_channels=embed_channels, + groups=groups, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=drop_path_rates[i], + enable_checkpoint=enable_checkpoint, + ) + self.blocks.append(block) + + def forward(self, points): + coord, feat, offset = points + # reference index query of neighbourhood attention + # for windows attention, modify reference index query method + reference_index, _ = pointops.knn_query(self.neighbours, coord, offset) + for block in self.blocks: + points = block(points, reference_index) + return points + + +class GridPool(nn.Module): + """ + Partition-based Pooling (Grid Pooling) + """ + + def __init__(self, in_channels, out_channels, grid_size, bias=False): + super(GridPool, self).__init__() + self.in_channels = in_channels + self.out_channels = out_channels + self.grid_size = grid_size + + self.fc = nn.Linear(in_channels, out_channels, bias=bias) + self.norm = PointBatchNorm(out_channels) + self.act = nn.ReLU(inplace=True) + + def forward(self, points, start=None): + coord, feat, offset = points + batch = offset2batch(offset) + feat = self.act(self.norm(self.fc(feat))) + start = ( + segment_csr( + coord, + torch.cat([batch.new_zeros(1), torch.cumsum(batch.bincount(), dim=0)]), + reduce="min", + ) + if start is None + else start + ) + cluster = voxel_grid( + pos=coord - start[batch], size=self.grid_size, batch=batch, start=0 + ) + unique, cluster, counts = torch.unique( + cluster, sorted=True, return_inverse=True, return_counts=True + ) + _, sorted_cluster_indices = torch.sort(cluster) + idx_ptr = torch.cat([counts.new_zeros(1), torch.cumsum(counts, dim=0)]) + coord = segment_csr(coord[sorted_cluster_indices], idx_ptr, reduce="mean") + feat = segment_csr(feat[sorted_cluster_indices], idx_ptr, reduce="max") + batch = batch[idx_ptr[:-1]] + offset = batch2offset(batch) + return [coord, feat, offset], cluster + + +class UnpoolWithSkip(nn.Module): + """ + Map Unpooling with skip connection + """ + + def __init__( + self, + in_channels, + skip_channels, + out_channels, + bias=True, + skip=True, + backend="map", + ): + super(UnpoolWithSkip, self).__init__() + self.in_channels = in_channels + self.skip_channels = skip_channels + self.out_channels = out_channels + self.skip = skip + self.backend = backend + assert self.backend in ["map", "interp"] + + self.proj = nn.Sequential( + nn.Linear(in_channels, out_channels, bias=bias), + PointBatchNorm(out_channels), + nn.ReLU(inplace=True), + ) + self.proj_skip = nn.Sequential( + nn.Linear(skip_channels, out_channels, bias=bias), + PointBatchNorm(out_channels), + nn.ReLU(inplace=True), + ) + + def forward(self, points, skip_points, cluster=None): + coord, feat, offset = points + skip_coord, skip_feat, skip_offset = skip_points + if self.backend == "map" and cluster is not None: + feat = self.proj(feat)[cluster] + else: + feat = pointops.interpolation( + coord, skip_coord, self.proj(feat), offset, skip_offset + ) + if self.skip: + feat = feat + self.proj_skip(skip_feat) + return [skip_coord, feat, skip_offset] + + +class Encoder(nn.Module): + def __init__( + self, + depth, + in_channels, + embed_channels, + groups, + grid_size=None, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=None, + drop_path_rate=None, + enable_checkpoint=False, + ): + super(Encoder, self).__init__() + + self.down = GridPool( + in_channels=in_channels, + out_channels=embed_channels, + grid_size=grid_size, + ) + + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate if attn_drop_rate is not None else 0.0, + drop_path_rate=drop_path_rate if drop_path_rate is not None else 0.0, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points): + points, cluster = self.down(points) + return self.blocks(points), cluster + + +class Decoder(nn.Module): + def __init__( + self, + in_channels, + skip_channels, + embed_channels, + groups, + depth, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=None, + drop_path_rate=None, + enable_checkpoint=False, + unpool_backend="map", + ): + super(Decoder, self).__init__() + + self.up = UnpoolWithSkip( + in_channels=in_channels, + out_channels=embed_channels, + skip_channels=skip_channels, + backend=unpool_backend, + ) + + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate if attn_drop_rate is not None else 0.0, + drop_path_rate=drop_path_rate if drop_path_rate is not None else 0.0, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points, skip_points, cluster): + points = self.up(points, skip_points, cluster) + return self.blocks(points) + + +class GVAPatchEmbed(nn.Module): + def __init__( + self, + depth, + in_channels, + embed_channels, + groups, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(GVAPatchEmbed, self).__init__() + self.in_channels = in_channels + self.embed_channels = embed_channels + self.proj = nn.Sequential( + nn.Linear(in_channels, embed_channels, bias=False), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + ) + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=drop_path_rate, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points): + coord, feat, offset = points + feat = self.proj(feat) + return self.blocks([coord, feat, offset]) + + +@MODELS.register_module("PT-v2m2") +class PointTransformerV2(nn.Module): + def __init__( + self, + in_channels, + num_classes, + patch_embed_depth=1, + patch_embed_channels=48, + patch_embed_groups=6, + patch_embed_neighbours=8, + enc_depths=(2, 2, 6, 2), + enc_channels=(96, 192, 384, 512), + enc_groups=(12, 24, 48, 64), + enc_neighbours=(16, 16, 16, 16), + dec_depths=(1, 1, 1, 1), + dec_channels=(48, 96, 192, 384), + dec_groups=(6, 12, 24, 48), + dec_neighbours=(16, 16, 16, 16), + grid_sizes=(0.06, 0.12, 0.24, 0.48), + attn_qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0, + enable_checkpoint=False, + unpool_backend="map", + ): + super(PointTransformerV2, self).__init__() + self.in_channels = in_channels + self.num_classes = num_classes + self.num_stages = len(enc_depths) + assert self.num_stages == len(dec_depths) + assert self.num_stages == len(enc_channels) + assert self.num_stages == len(dec_channels) + assert self.num_stages == len(enc_groups) + assert self.num_stages == len(dec_groups) + assert self.num_stages == len(enc_neighbours) + assert self.num_stages == len(dec_neighbours) + assert self.num_stages == len(grid_sizes) + self.patch_embed = GVAPatchEmbed( + in_channels=in_channels, + embed_channels=patch_embed_channels, + groups=patch_embed_groups, + depth=patch_embed_depth, + neighbours=patch_embed_neighbours, + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + enable_checkpoint=enable_checkpoint, + ) + + enc_dp_rates = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(enc_depths)) + ] + dec_dp_rates = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(dec_depths)) + ] + enc_channels = [patch_embed_channels] + list(enc_channels) + dec_channels = list(dec_channels) + [enc_channels[-1]] + self.enc_stages = nn.ModuleList() + self.dec_stages = nn.ModuleList() + for i in range(self.num_stages): + enc = Encoder( + depth=enc_depths[i], + in_channels=enc_channels[i], + embed_channels=enc_channels[i + 1], + groups=enc_groups[i], + grid_size=grid_sizes[i], + neighbours=enc_neighbours[i], + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=enc_dp_rates[ + sum(enc_depths[:i]) : sum(enc_depths[: i + 1]) + ], + enable_checkpoint=enable_checkpoint, + ) + dec = Decoder( + depth=dec_depths[i], + in_channels=dec_channels[i + 1], + skip_channels=enc_channels[i], + embed_channels=dec_channels[i], + groups=dec_groups[i], + neighbours=dec_neighbours[i], + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=dec_dp_rates[ + sum(dec_depths[:i]) : sum(dec_depths[: i + 1]) + ], + enable_checkpoint=enable_checkpoint, + unpool_backend=unpool_backend, + ) + self.enc_stages.append(enc) + self.dec_stages.append(dec) + self.seg_head = ( + nn.Sequential( + nn.Linear(dec_channels[0], dec_channels[0]), + PointBatchNorm(dec_channels[0]), + nn.ReLU(inplace=True), + nn.Linear(dec_channels[0], num_classes), + ) + if num_classes > 0 + else nn.Identity() + ) + + def forward(self, data_dict): + coord = data_dict["coord"] + feat = data_dict["feat"] + offset = data_dict["offset"].int() + + # a batch of point cloud is a list of coord, feat and offset + points = [coord, feat, offset] + points = self.patch_embed(points) + skips = [[points]] + for i in range(self.num_stages): + points, cluster = self.enc_stages[i](points) + skips[-1].append(cluster) # record grid cluster of pooling + skips.append([points]) # record points info of current stage + + points = skips.pop(-1)[0] # unpooling points info in the last enc stage + for i in reversed(range(self.num_stages)): + skip_points, cluster = skips.pop(-1) + points = self.dec_stages[i](points, skip_points, cluster) + coord, feat, offset = points + seg_logits = self.seg_head(feat) + return seg_logits diff --git a/pointcept/models/point_transformer_v2/point_transformer_v2m3_pdnorm.py b/pointcept/models/point_transformer_v2/point_transformer_v2m3_pdnorm.py new file mode 100644 index 0000000000000000000000000000000000000000..c7b50881f861abf1bb2cb889c1827d9db5b09a89 --- /dev/null +++ b/pointcept/models/point_transformer_v2/point_transformer_v2m3_pdnorm.py @@ -0,0 +1,659 @@ +""" +Point Transformer V2M3 + +Enable Prompt-Driven Normalization for Point Prompt Training + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from copy import deepcopy +import math +import torch +import torch.nn as nn +from torch.utils.checkpoint import checkpoint +from torch_geometric.nn.pool import voxel_grid +from torch_scatter import segment_csr + +import einops +from timm.layers import DropPath +import pointops + +from pointcept.models.builder import MODELS +from pointcept.models.utils import offset2batch, batch2offset + + +class PDBatchNorm(torch.nn.Module): + def __init__( + self, + num_features, + context_channels=256, + eps=1e-3, + momentum=0.01, + conditions=("ScanNet", "S3DIS", "Structured3D"), + decouple=True, + adaptive=False, + affine=True, + ): + super().__init__() + self.conditions = conditions + self.decouple = decouple + self.adaptive = adaptive + self.affine = affine + if self.decouple: + self.bns = nn.ModuleList( + [ + nn.BatchNorm1d( + num_features=num_features, + eps=eps, + momentum=momentum, + affine=affine, + ) + for _ in conditions + ] + ) + else: + self.bn = nn.BatchNorm1d( + num_features=num_features, eps=eps, momentum=momentum, affine=affine + ) + if self.adaptive: + self.modulation = nn.Sequential( + nn.SiLU(), nn.Linear(context_channels, 2 * num_features, bias=True) + ) + + def forward(self, feat, condition=None, context=None): + if self.decouple: + assert condition in self.conditions + bn = self.bns[self.conditions.index(condition)] + else: + bn = self.bn + feat = bn(feat) + if self.adaptive: + assert context is not None + shift, scale = self.modulation(context).chunk(2, dim=1) + feat = feat * (1.0 + scale) + shift + return feat + + +class PointBatchNorm(nn.Module): + """ + Batch Normalization for Point Clouds data in shape of [B*N, C], [B*N, L, C] + """ + + def __init__(self, embed_channels): + super().__init__() + self.norm = nn.BatchNorm1d(embed_channels) + + def forward(self, input: torch.Tensor) -> torch.Tensor: + if input.dim() == 3: + return ( + self.norm(input.transpose(1, 2).contiguous()) + .transpose(1, 2) + .contiguous() + ) + elif input.dim() == 2: + return self.norm(input) + else: + raise NotImplementedError + + +class GroupedVectorAttention(nn.Module): + def __init__( + self, + embed_channels, + groups, + attn_drop_rate=0.0, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + ): + super(GroupedVectorAttention, self).__init__() + self.embed_channels = embed_channels + self.groups = groups + assert embed_channels % groups == 0 + self.attn_drop_rate = attn_drop_rate + self.qkv_bias = qkv_bias + self.pe_multiplier = pe_multiplier + self.pe_bias = pe_bias + + self.linear_q = nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=qkv_bias), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + ) + self.linear_k = nn.Sequential( + nn.Linear(embed_channels, embed_channels, bias=qkv_bias), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + ) + + self.linear_v = nn.Linear(embed_channels, embed_channels, bias=qkv_bias) + + if self.pe_multiplier: + self.linear_p_multiplier = nn.Sequential( + nn.Linear(3, embed_channels), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + nn.Linear(embed_channels, embed_channels), + ) + if self.pe_bias: + self.linear_p_bias = nn.Sequential( + nn.Linear(3, embed_channels), + PointBatchNorm(embed_channels), + nn.ReLU(inplace=True), + nn.Linear(embed_channels, embed_channels), + ) + self.weight_encoding = nn.Sequential( + nn.Linear(embed_channels, groups), + PointBatchNorm(groups), + nn.ReLU(inplace=True), + nn.Linear(groups, groups), + ) + self.softmax = nn.Softmax(dim=1) + self.attn_drop = nn.Dropout(attn_drop_rate) + + def forward(self, feat, coord, reference_index): + query, key, value = ( + self.linear_q(feat), + self.linear_k(feat), + self.linear_v(feat), + ) + key = pointops.grouping(reference_index, key, coord, with_xyz=True) + value = pointops.grouping(reference_index, value, coord, with_xyz=False) + pos, key = key[:, :, 0:3], key[:, :, 3:] + relation_qk = key - query.unsqueeze(1) + if self.pe_multiplier: + pem = self.linear_p_multiplier(pos) + relation_qk = relation_qk * pem + if self.pe_bias: + peb = self.linear_p_bias(pos) + relation_qk = relation_qk + peb + value = value + peb + + weight = self.weight_encoding(relation_qk) + weight = self.attn_drop(self.softmax(weight)) + + mask = torch.sign(reference_index + 1) + weight = torch.einsum("n s g, n s -> n s g", weight, mask) + value = einops.rearrange(value, "n ns (g i) -> n ns g i", g=self.groups) + feat = torch.einsum("n s g i, n s g -> n g i", value, weight) + feat = einops.rearrange(feat, "n g i -> n (g i)") + return feat + + +class Block(nn.Module): + def __init__( + self, + embed_channels, + groups, + norm_fn=None, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(Block, self).__init__() + self.attn = GroupedVectorAttention( + embed_channels=embed_channels, + groups=groups, + qkv_bias=qkv_bias, + attn_drop_rate=attn_drop_rate, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + ) + + assert norm_fn is not None + + self.fc1 = nn.Linear(embed_channels, embed_channels, bias=False) + self.fc3 = nn.Linear(embed_channels, embed_channels, bias=False) + self.norm1 = norm_fn(embed_channels) + self.norm2 = norm_fn(embed_channels) + self.norm3 = norm_fn(embed_channels) + self.act = nn.ReLU(inplace=True) + self.enable_checkpoint = enable_checkpoint + self.drop_path = ( + DropPath(drop_path_rate) if drop_path_rate > 0.0 else nn.Identity() + ) + + def forward(self, points, reference_index): + coord, feat, offset, condition, context = points + identity = feat + feat = self.act(self.norm1(self.fc1(feat), condition, context)) + feat = ( + self.attn(feat, coord, reference_index) + if not self.enable_checkpoint + else checkpoint(self.attn, feat, coord, reference_index) + ) + feat = self.act(self.norm2(feat, condition, context)) + feat = self.norm3(self.fc3(feat), condition, context) + feat = identity + self.drop_path(feat) + feat = self.act(feat) + return [coord, feat, offset, condition, context] + + +class BlockSequence(nn.Module): + def __init__( + self, + depth, + embed_channels, + groups, + neighbours=16, + norm_fn=None, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(BlockSequence, self).__init__() + + if isinstance(drop_path_rate, list): + drop_path_rates = drop_path_rate + assert len(drop_path_rates) == depth + elif isinstance(drop_path_rate, float): + drop_path_rates = [deepcopy(drop_path_rate) for _ in range(depth)] + else: + drop_path_rates = [0.0 for _ in range(depth)] + + self.neighbours = neighbours + self.blocks = nn.ModuleList() + for i in range(depth): + block = Block( + embed_channels=embed_channels, + groups=groups, + norm_fn=norm_fn, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=drop_path_rates[i], + enable_checkpoint=enable_checkpoint, + ) + self.blocks.append(block) + + def forward(self, points): + coord, feat, offset, condition, context = points + # reference index query of neighbourhood attention + # for windows attention, modify reference index query method + reference_index, _ = pointops.knn_query(self.neighbours, coord, offset) + for block in self.blocks: + points = block(points, reference_index) + return points + + +class GridPool(nn.Module): + """ + Partition-based Pooling (Grid Pooling) + """ + + def __init__(self, in_channels, out_channels, grid_size, norm_fn, bias=False): + super(GridPool, self).__init__() + self.in_channels = in_channels + self.out_channels = out_channels + self.grid_size = grid_size + + self.fc = nn.Linear(in_channels, out_channels, bias=bias) + self.norm = norm_fn(out_channels) + self.act = nn.ReLU(inplace=True) + + def forward(self, points, start=None): + coord, feat, offset, condition, context = points + batch = offset2batch(offset) + feat = self.act(self.norm(self.fc(feat), condition, context)) + start = ( + segment_csr( + coord, + torch.cat([batch.new_zeros(1), torch.cumsum(batch.bincount(), dim=0)]), + reduce="min", + ) + if start is None + else start + ) + cluster = voxel_grid( + pos=coord - start[batch], size=self.grid_size, batch=batch, start=0 + ) + unique, cluster, counts = torch.unique( + cluster, sorted=True, return_inverse=True, return_counts=True + ) + _, sorted_cluster_indices = torch.sort(cluster) + idx_ptr = torch.cat([counts.new_zeros(1), torch.cumsum(counts, dim=0)]) + coord = segment_csr(coord[sorted_cluster_indices], idx_ptr, reduce="mean") + feat = segment_csr(feat[sorted_cluster_indices], idx_ptr, reduce="max") + batch = batch[idx_ptr[:-1]] + offset = batch2offset(batch) + return [coord, feat, offset, condition, context], cluster + + +class UnpoolWithSkip(nn.Module): + """ + Map Unpooling with skip connection + """ + + def __init__( + self, + in_channels, + skip_channels, + out_channels, + norm_fn, + bias=True, + skip=True, + backend="map", + ): + super(UnpoolWithSkip, self).__init__() + self.in_channels = in_channels + self.skip_channels = skip_channels + self.out_channels = out_channels + self.skip = skip + self.backend = backend + assert self.backend in ["map", "interp"] + + self.proj_linear = nn.Linear(in_channels, out_channels, bias=bias) + self.proj_norm = norm_fn(out_channels) + self.proj_act = nn.ReLU(inplace=True) + + self.proj_skip_linear = nn.Linear(skip_channels, out_channels, bias=bias) + self.proj_skip_norm = norm_fn(out_channels) + self.proj_skip_act = nn.ReLU(inplace=True) + + def forward(self, points, skip_points, cluster=None): + coord, feat, offset, condition, context = points + skip_coord, skip_feat, skip_offset, _, _ = skip_points + feat = self.proj_act(self.proj_norm(self.proj_linear(feat), condition, context)) + if self.backend == "map" and cluster is not None: + feat = feat[cluster] + else: + feat = pointops.interpolation(coord, skip_coord, feat, offset, skip_offset) + if self.skip: + feat = feat + self.proj_skip_act( + self.proj_skip_norm( + self.proj_skip_linear(skip_feat), condition, context + ) + ) + return [skip_coord, feat, skip_offset, condition, context] + + +class Encoder(nn.Module): + def __init__( + self, + depth, + in_channels, + embed_channels, + groups, + norm_fn, + grid_size=None, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=None, + drop_path_rate=None, + enable_checkpoint=False, + ): + super(Encoder, self).__init__() + + self.down = GridPool( + in_channels=in_channels, + out_channels=embed_channels, + grid_size=grid_size, + norm_fn=norm_fn, + ) + + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + norm_fn=norm_fn, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate if attn_drop_rate is not None else 0.0, + drop_path_rate=drop_path_rate if drop_path_rate is not None else 0.0, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points): + points, cluster = self.down(points) + return self.blocks(points), cluster + + +class Decoder(nn.Module): + def __init__( + self, + in_channels, + skip_channels, + embed_channels, + groups, + depth, + norm_fn, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=None, + drop_path_rate=None, + enable_checkpoint=False, + unpool_backend="map", + ): + super(Decoder, self).__init__() + + self.up = UnpoolWithSkip( + in_channels=in_channels, + out_channels=embed_channels, + skip_channels=skip_channels, + backend=unpool_backend, + norm_fn=norm_fn, + ) + + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + norm_fn=norm_fn, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate if attn_drop_rate is not None else 0.0, + drop_path_rate=drop_path_rate if drop_path_rate is not None else 0.0, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points, skip_points, cluster): + points = self.up(points, skip_points, cluster) + return self.blocks(points) + + +class GVAPatchEmbed(nn.Module): + def __init__( + self, + depth, + in_channels, + embed_channels, + groups, + norm_fn, + neighbours=16, + qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0.0, + enable_checkpoint=False, + ): + super(GVAPatchEmbed, self).__init__() + self.in_channels = in_channels + self.embed_channels = embed_channels + self.proj_linear = nn.Linear(in_channels, embed_channels, bias=False) + self.proj_norm = norm_fn(embed_channels) + self.proj_act = nn.ReLU(inplace=True) + self.blocks = BlockSequence( + depth=depth, + embed_channels=embed_channels, + groups=groups, + neighbours=neighbours, + norm_fn=norm_fn, + qkv_bias=qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=drop_path_rate, + enable_checkpoint=enable_checkpoint, + ) + + def forward(self, points): + coord, feat, offset, condition, context = points + feat = self.proj_act(self.proj_norm(self.proj_linear(feat), condition, context)) + return self.blocks([coord, feat, offset, condition, context]) + + +@MODELS.register_module("PT-v2m3") +class PointTransformerV2(nn.Module): + def __init__( + self, + in_channels, + num_classes, + patch_embed_depth=1, + patch_embed_channels=48, + patch_embed_groups=6, + patch_embed_neighbours=8, + enc_depths=(2, 2, 6, 2), + enc_channels=(96, 192, 384, 512), + enc_groups=(12, 24, 48, 64), + enc_neighbours=(16, 16, 16, 16), + dec_depths=(1, 1, 1, 1), + dec_channels=(48, 96, 192, 384), + dec_groups=(6, 12, 24, 48), + dec_neighbours=(16, 16, 16, 16), + grid_sizes=(0.06, 0.12, 0.24, 0.48), + attn_qkv_bias=True, + pe_multiplier=False, + pe_bias=True, + attn_drop_rate=0.0, + drop_path_rate=0, + enable_checkpoint=False, + unpool_backend="map", + context_channels=256, + conditions=("ScanNet", "S3DIS", "Structured3D"), + norm_decouple=True, + norm_adaptive=True, + norm_affine=False, + ): + super(PointTransformerV2, self).__init__() + self.in_channels = in_channels + self.num_classes = num_classes + self.num_stages = len(enc_depths) + assert self.num_stages == len(dec_depths) + assert self.num_stages == len(enc_channels) + assert self.num_stages == len(dec_channels) + assert self.num_stages == len(enc_groups) + assert self.num_stages == len(dec_groups) + assert self.num_stages == len(enc_neighbours) + assert self.num_stages == len(dec_neighbours) + assert self.num_stages == len(grid_sizes) + + norm_fn = partial( + PDBatchNorm, + eps=1e-3, + momentum=0.01, + conditions=conditions, + context_channels=context_channels, + decouple=norm_decouple, + adaptive=norm_adaptive, + affine=norm_affine, + ) + + self.patch_embed = GVAPatchEmbed( + in_channels=in_channels, + embed_channels=patch_embed_channels, + groups=patch_embed_groups, + depth=patch_embed_depth, + neighbours=patch_embed_neighbours, + norm_fn=norm_fn, + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + enable_checkpoint=enable_checkpoint, + ) + + enc_dp_rates = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(enc_depths)) + ] + dec_dp_rates = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(dec_depths)) + ] + enc_channels = [patch_embed_channels] + list(enc_channels) + dec_channels = list(dec_channels) + [enc_channels[-1]] + self.enc_stages = nn.ModuleList() + self.dec_stages = nn.ModuleList() + for i in range(self.num_stages): + enc = Encoder( + depth=enc_depths[i], + in_channels=enc_channels[i], + embed_channels=enc_channels[i + 1], + groups=enc_groups[i], + grid_size=grid_sizes[i], + neighbours=enc_neighbours[i], + norm_fn=norm_fn, + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=enc_dp_rates[ + sum(enc_depths[:i]) : sum(enc_depths[: i + 1]) + ], + enable_checkpoint=enable_checkpoint, + ) + dec = Decoder( + depth=dec_depths[i], + in_channels=dec_channels[i + 1], + skip_channels=enc_channels[i], + embed_channels=dec_channels[i], + groups=dec_groups[i], + neighbours=dec_neighbours[i], + norm_fn=norm_fn, + qkv_bias=attn_qkv_bias, + pe_multiplier=pe_multiplier, + pe_bias=pe_bias, + attn_drop_rate=attn_drop_rate, + drop_path_rate=dec_dp_rates[ + sum(dec_depths[:i]) : sum(dec_depths[: i + 1]) + ], + enable_checkpoint=enable_checkpoint, + unpool_backend=unpool_backend, + ) + self.enc_stages.append(enc) + self.dec_stages.append(dec) + self.seg_head = ( + nn.Sequential(nn.Linear(dec_channels[0], num_classes)) + if num_classes > 0 + else nn.Identity() + ) + + def forward(self, data_dict): + coord = data_dict["coord"] + feat = data_dict["feat"] + offset = data_dict["offset"].int() + condition = data_dict["condition"][0] + context = data_dict["context"] if "context" in data_dict.keys() else None + + # a batch of point cloud is a list of coord, feat and offset + points = [coord, feat, offset, condition, context] + points = self.patch_embed(points) + skips = [[points]] + for i in range(self.num_stages): + points, cluster = self.enc_stages[i](points) + skips[-1].append(cluster) # record grid cluster of pooling + skips.append([points]) # record points info of current stage + + points = skips.pop(-1)[0] # unpooling points info in the last enc stage + for i in reversed(range(self.num_stages)): + skip_points, cluster = skips.pop(-1) + points = self.dec_stages[i](points, skip_points, cluster) + coord, feat, offset, _, _ = points + seg_logits = self.seg_head(feat) + return seg_logits diff --git a/pointcept/models/point_transformer_v3/__init__.py b/pointcept/models/point_transformer_v3/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..b0c3d05e866ee74d942a3f4526e3bf036b0a1401 --- /dev/null +++ b/pointcept/models/point_transformer_v3/__init__.py @@ -0,0 +1,3 @@ +from .point_transformer_v3m1_base import * +from .point_transformer_v3m2_sonata import * +from .point_transformer_v3m3_utonia import * diff --git a/pointcept/models/point_transformer_v3/point_transformer_v3m1_base.py b/pointcept/models/point_transformer_v3/point_transformer_v3m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..c412dff23ddf0f5bc7a31de46f1168d2fb34e856 --- /dev/null +++ b/pointcept/models/point_transformer_v3/point_transformer_v3m1_base.py @@ -0,0 +1,721 @@ +""" +Point Transformer - V3 Mode1 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from addict import Dict +import math +import torch +import torch.nn as nn +import spconv.pytorch as spconv +import torch_scatter +from timm.layers import DropPath + +try: + import flash_attn +except ImportError: + flash_attn = None + +from pointcept.models.point_prompt_training import PDNorm +from pointcept.models.builder import MODELS +from pointcept.models.utils.misc import offset2bincount +from pointcept.models.utils.structure import Point +from pointcept.models.modules import PointModule, PointSequential + + +class RPE(torch.nn.Module): + def __init__(self, patch_size, num_heads): + super().__init__() + self.patch_size = patch_size + self.num_heads = num_heads + self.pos_bnd = int((4 * patch_size) ** (1 / 3) * 2) + self.rpe_num = 2 * self.pos_bnd + 1 + self.rpe_table = torch.nn.Parameter(torch.zeros(3 * self.rpe_num, num_heads)) + torch.nn.init.trunc_normal_(self.rpe_table, std=0.02) + + def forward(self, coord): + idx = ( + coord.clamp(-self.pos_bnd, self.pos_bnd) # clamp into bnd + + self.pos_bnd # relative position to positive index + + torch.arange(3, device=coord.device) * self.rpe_num # x, y, z stride + ) + out = self.rpe_table.index_select(0, idx.reshape(-1)) + out = out.view(idx.shape + (-1,)).sum(3) + out = out.permute(0, 3, 1, 2) # (N, K, K, H) -> (N, H, K, K) + return out + + +class SerializedAttention(PointModule): + def __init__( + self, + channels, + num_heads, + patch_size, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + order_index=0, + enable_rpe=False, + enable_flash=True, + upcast_attention=True, + upcast_softmax=True, + ): + super().__init__() + assert channels % num_heads == 0 + self.channels = channels + self.num_heads = num_heads + self.scale = qk_scale or (channels // num_heads) ** -0.5 + self.order_index = order_index + self.upcast_attention = upcast_attention + self.upcast_softmax = upcast_softmax + self.enable_rpe = enable_rpe + self.enable_flash = enable_flash + if enable_flash: + assert ( + enable_rpe is False + ), "Set enable_rpe to False when enable Flash Attention" + assert ( + upcast_attention is False + ), "Set upcast_attention to False when enable Flash Attention" + assert ( + upcast_softmax is False + ), "Set upcast_softmax to False when enable Flash Attention" + assert flash_attn is not None, "Make sure flash_attn is installed." + self.patch_size = patch_size + self.attn_drop = attn_drop + self.flash_dtype = None + else: + # when disable flash attention, we still don't want to use mask + # consequently, patch size will auto set to the + # min number of patch_size_max and number of points + self.patch_size_max = patch_size + self.patch_size = 0 + self.attn_drop = torch.nn.Dropout(attn_drop) + + self.qkv = torch.nn.Linear(channels, channels * 3, bias=qkv_bias) + self.proj = torch.nn.Linear(channels, channels) + self.proj_drop = torch.nn.Dropout(proj_drop) + self.softmax = torch.nn.Softmax(dim=-1) + self.rpe = RPE(patch_size, num_heads) if self.enable_rpe else None + + @torch.no_grad() + def get_rel_pos(self, point, order): + K = self.patch_size + rel_pos_key = f"rel_pos_{self.order_index}" + if rel_pos_key not in point.keys(): + grid_coord = point.grid_coord[order] + grid_coord = grid_coord.reshape(-1, K, 3) + point[rel_pos_key] = grid_coord.unsqueeze(2) - grid_coord.unsqueeze(1) + return point[rel_pos_key] + + @torch.no_grad() + def get_padding_and_inverse(self, point): + pad_key = "pad" + unpad_key = "unpad" + cu_seqlens_key = "cu_seqlens_key" + if ( + pad_key not in point.keys() + or unpad_key not in point.keys() + or cu_seqlens_key not in point.keys() + ): + offset = point.offset + bincount = offset2bincount(offset) + bincount_pad = ( + torch.div( + bincount + self.patch_size - 1, + self.patch_size, + rounding_mode="trunc", + ) + * self.patch_size + ) + # only pad point when num of points larger than patch_size + mask_pad = bincount > self.patch_size + bincount_pad = ~mask_pad * bincount + mask_pad * bincount_pad + _offset = nn.functional.pad(offset, (1, 0)) + _offset_pad = nn.functional.pad(torch.cumsum(bincount_pad, dim=0), (1, 0)) + pad = torch.arange(_offset_pad[-1], device=offset.device) + unpad = torch.arange(_offset[-1], device=offset.device) + cu_seqlens = [] + for i in range(len(offset)): + unpad[_offset[i] : _offset[i + 1]] += _offset_pad[i] - _offset[i] + if bincount[i] != bincount_pad[i]: + pad[ + _offset_pad[i + 1] + - self.patch_size + + (bincount[i] % self.patch_size) : _offset_pad[i + 1] + ] = pad[ + _offset_pad[i + 1] + - 2 * self.patch_size + + (bincount[i] % self.patch_size) : _offset_pad[i + 1] + - self.patch_size + ] + pad[_offset_pad[i] : _offset_pad[i + 1]] -= _offset_pad[i] - _offset[i] + cu_seqlens.append( + torch.arange( + _offset_pad[i], + _offset_pad[i + 1], + step=self.patch_size, + dtype=torch.int32, + device=offset.device, + ) + ) + point[pad_key] = pad + point[unpad_key] = unpad + point[cu_seqlens_key] = nn.functional.pad( + torch.concat(cu_seqlens), (0, 1), value=_offset_pad[-1] + ) + return point[pad_key], point[unpad_key], point[cu_seqlens_key] + + def forward(self, point): + if not self.enable_flash: + self.patch_size = min( + offset2bincount(point.offset).min().tolist(), self.patch_size_max + ) + + H = self.num_heads + K = self.patch_size + C = self.channels + + pad, unpad, cu_seqlens = self.get_padding_and_inverse(point) + + order = point.serialized_order[self.order_index][pad] + inverse = unpad[point.serialized_inverse[self.order_index]] + + # padding and reshape feat and batch for serialized point patch + qkv = self.qkv(point.feat)[order] + + if not self.enable_flash: + # encode and reshape qkv: (N', K, 3, H, C') => (3, N', H, K, C') + q, k, v = ( + qkv.reshape(-1, K, 3, H, C // H).permute(2, 0, 3, 1, 4).unbind(dim=0) + ) + # attn + if self.upcast_attention: + q = q.float() + k = k.float() + attn = (q * self.scale) @ k.transpose(-2, -1) # (N', H, K, K) + if self.enable_rpe: + attn = attn + self.rpe(self.get_rel_pos(point, order)) + if self.upcast_softmax: + attn = attn.float() + attn = self.softmax(attn) + attn = self.attn_drop(attn).to(qkv.dtype) + feat = (attn @ v).transpose(1, 2).reshape(-1, C) + else: + if self.flash_dtype is None: + self.flash_dtype = ( + torch.bfloat16 + if torch.cuda.is_available() and torch.cuda.is_bf16_supported() + else torch.float16 + ) + feat = flash_attn.flash_attn_varlen_qkvpacked_func( + qkv.to(self.flash_dtype).reshape(-1, 3, H, C // H), + cu_seqlens, + max_seqlen=self.patch_size, + dropout_p=self.attn_drop if self.training else 0, + softmax_scale=self.scale, + ).reshape(-1, C) + feat = feat.to(qkv.dtype) + feat = feat[inverse] + + # ffn + feat = self.proj(feat) + feat = self.proj_drop(feat) + point.feat = feat + return point + + +class MLP(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=None, + out_channels=None, + act_layer=nn.GELU, + drop=0.0, + ): + super().__init__() + out_channels = out_channels or in_channels + hidden_channels = hidden_channels or in_channels + self.fc1 = nn.Linear(in_channels, hidden_channels) + self.act = act_layer() + self.fc2 = nn.Linear(hidden_channels, out_channels) + self.drop = nn.Dropout(drop) + + def forward(self, x): + x = self.fc1(x) + x = self.act(x) + x = self.drop(x) + x = self.fc2(x) + x = self.drop(x) + return x + + +class Block(PointModule): + def __init__( + self, + channels, + num_heads, + patch_size=48, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.0, + norm_layer=nn.LayerNorm, + act_layer=nn.GELU, + pre_norm=True, + order_index=0, + cpe_indice_key=None, + enable_rpe=False, + enable_flash=True, + upcast_attention=True, + upcast_softmax=True, + ): + super().__init__() + self.channels = channels + self.pre_norm = pre_norm + + self.cpe = PointSequential( + spconv.SubMConv3d( + channels, + channels, + kernel_size=3, + bias=True, + indice_key=cpe_indice_key, + ), + nn.Linear(channels, channels), + norm_layer(channels), + ) + + self.norm1 = PointSequential(norm_layer(channels)) + self.attn = SerializedAttention( + channels=channels, + patch_size=patch_size, + num_heads=num_heads, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + order_index=order_index, + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + ) + self.norm2 = PointSequential(norm_layer(channels)) + self.mlp = PointSequential( + MLP( + in_channels=channels, + hidden_channels=int(channels * mlp_ratio), + out_channels=channels, + act_layer=act_layer, + drop=proj_drop, + ) + ) + self.drop_path = PointSequential( + DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + ) + + def forward(self, point: Point): + shortcut = point.feat + point = self.cpe(point) + point.feat = shortcut + point.feat + shortcut = point.feat + if self.pre_norm: + point = self.norm1(point) + point = self.drop_path(self.attn(point)) + point.feat = shortcut + point.feat + if not self.pre_norm: + point = self.norm1(point) + + shortcut = point.feat + if self.pre_norm: + point = self.norm2(point) + point = self.drop_path(self.mlp(point)) + point.feat = shortcut + point.feat + if not self.pre_norm: + point = self.norm2(point) + point.sparse_conv_feat = point.sparse_conv_feat.replace_feature(point.feat) + return point + + +class SerializedPooling(PointModule): + def __init__( + self, + in_channels, + out_channels, + stride=2, + norm_layer=None, + act_layer=None, + reduce="max", + shuffle_orders=True, + traceable=True, # record parent and cluster + ): + super().__init__() + self.in_channels = in_channels + self.out_channels = out_channels + + assert stride == 2 ** (math.ceil(stride) - 1).bit_length() # 2, 4, 8 + # TODO: add support to grid pool (any stride) + self.stride = stride + assert reduce in ["sum", "mean", "min", "max"] + self.reduce = reduce + self.shuffle_orders = shuffle_orders + self.traceable = traceable + + self.proj = nn.Linear(in_channels, out_channels) + if norm_layer is not None: + self.norm = PointSequential(norm_layer(out_channels)) + if act_layer is not None: + self.act = PointSequential(act_layer()) + + def forward(self, point: Point): + pooling_depth = (math.ceil(self.stride) - 1).bit_length() + if pooling_depth > point.serialized_depth: + pooling_depth = 0 + assert { + "serialized_code", + "serialized_order", + "serialized_inverse", + "serialized_depth", + }.issubset( + point.keys() + ), "Run point.serialization() point cloud before SerializedPooling" + + code = point.serialized_code >> pooling_depth * 3 + code_, cluster, counts = torch.unique( + code[0], + sorted=True, + return_inverse=True, + return_counts=True, + ) + # indices of point sorted by cluster, for torch_scatter.segment_csr + _, indices = torch.sort(cluster) + # index pointer for sorted point, for torch_scatter.segment_csr + idx_ptr = torch.cat([counts.new_zeros(1), torch.cumsum(counts, dim=0)]) + # head_indices of each cluster, for reduce attr e.g. code, batch + head_indices = indices[idx_ptr[:-1]] + # generate down code, order, inverse + code = code[:, head_indices] + order = torch.argsort(code) + inverse = torch.zeros_like(order).scatter_( + dim=1, + index=order, + src=torch.arange(0, code.shape[1], device=order.device).repeat( + code.shape[0], 1 + ), + ) + + if self.shuffle_orders: + perm = torch.randperm(code.shape[0]) + code = code[perm] + order = order[perm] + inverse = inverse[perm] + + # collect information + point_dict = Dict( + feat=torch_scatter.segment_csr( + self.proj(point.feat)[indices], idx_ptr, reduce=self.reduce + ), + coord=torch_scatter.segment_csr( + point.coord[indices], idx_ptr, reduce="mean" + ), + grid_coord=point.grid_coord[head_indices] >> pooling_depth, + serialized_code=code, + serialized_order=order, + serialized_inverse=inverse, + serialized_depth=point.serialized_depth - pooling_depth, + batch=point.batch[head_indices], + ) + + if "condition" in point.keys(): + point_dict["condition"] = point.condition + if "context" in point.keys(): + point_dict["context"] = point.context + + if self.traceable: + point_dict["pooling_inverse"] = cluster + point_dict["pooling_parent"] = point + point = Point(point_dict) + if self.norm is not None: + point = self.norm(point) + if self.act is not None: + point = self.act(point) + point.sparsify() + return point + + +class SerializedUnpooling(PointModule): + def __init__( + self, + in_channels, + skip_channels, + out_channels, + norm_layer=None, + act_layer=None, + traceable=False, # record parent and cluster + ): + super().__init__() + self.proj = PointSequential(nn.Linear(in_channels, out_channels)) + self.proj_skip = PointSequential(nn.Linear(skip_channels, out_channels)) + + if norm_layer is not None: + self.proj.add(norm_layer(out_channels)) + self.proj_skip.add(norm_layer(out_channels)) + + if act_layer is not None: + self.proj.add(act_layer()) + self.proj_skip.add(act_layer()) + + self.traceable = traceable + + def forward(self, point): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + point = self.proj(point) + parent = self.proj_skip(parent) + parent.feat = parent.feat + point.feat[inverse] + + if self.traceable: + parent["unpooling_parent"] = point + return parent + + +class Embedding(PointModule): + def __init__( + self, + in_channels, + embed_channels, + norm_layer=None, + act_layer=None, + ): + super().__init__() + self.in_channels = in_channels + self.embed_channels = embed_channels + + # TODO: check remove spconv + self.stem = PointSequential( + conv=spconv.SubMConv3d( + in_channels, + embed_channels, + kernel_size=5, + padding=1, + bias=False, + indice_key="stem", + ) + ) + if norm_layer is not None: + self.stem.add(norm_layer(embed_channels), name="norm") + if act_layer is not None: + self.stem.add(act_layer(), name="act") + + def forward(self, point: Point): + point = self.stem(point) + return point + + +@MODELS.register_module("PT-v3m1") +class PointTransformerV3(PointModule): + def __init__( + self, + in_channels=6, + order=("z", "z-trans"), + stride=(2, 2, 2, 2), + enc_depths=(2, 2, 2, 6, 2), + enc_channels=(32, 64, 128, 256, 512), + enc_num_head=(2, 4, 8, 16, 32), + enc_patch_size=(48, 48, 48, 48, 48), + dec_depths=(2, 2, 2, 2), + dec_channels=(64, 64, 128, 256), + dec_num_head=(4, 4, 8, 16), + dec_patch_size=(48, 48, 48, 48), + mlp_ratio=4, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.3, + pre_norm=True, + shuffle_orders=True, + enable_rpe=False, + enable_flash=True, + upcast_attention=False, + upcast_softmax=False, + enc_mode=False, + pdnorm_bn=False, + pdnorm_ln=False, + pdnorm_decouple=True, + pdnorm_adaptive=False, + pdnorm_affine=True, + pdnorm_conditions=("ScanNet", "S3DIS", "Structured3D"), + ): + super().__init__() + self.num_stages = len(enc_depths) + self.order = [order] if isinstance(order, str) else order + self.enc_mode = enc_mode + self.shuffle_orders = shuffle_orders + + assert self.num_stages == len(stride) + 1 + assert self.num_stages == len(enc_depths) + assert self.num_stages == len(enc_channels) + assert self.num_stages == len(enc_num_head) + assert self.num_stages == len(enc_patch_size) + assert self.enc_mode or self.num_stages == len(dec_depths) + 1 + assert self.enc_mode or self.num_stages == len(dec_channels) + 1 + assert self.enc_mode or self.num_stages == len(dec_num_head) + 1 + assert self.enc_mode or self.num_stages == len(dec_patch_size) + 1 + + # norm layers + if pdnorm_bn: + bn_layer = partial( + PDNorm, + norm_layer=partial( + nn.BatchNorm1d, eps=1e-3, momentum=0.01, affine=pdnorm_affine + ), + conditions=pdnorm_conditions, + decouple=pdnorm_decouple, + adaptive=pdnorm_adaptive, + ) + else: + bn_layer = partial(nn.BatchNorm1d, eps=1e-3, momentum=0.01) + if pdnorm_ln: + ln_layer = partial( + PDNorm, + norm_layer=partial(nn.LayerNorm, elementwise_affine=pdnorm_affine), + conditions=pdnorm_conditions, + decouple=pdnorm_decouple, + adaptive=pdnorm_adaptive, + ) + else: + ln_layer = nn.LayerNorm + # activation layers + act_layer = nn.GELU + + self.embedding = Embedding( + in_channels=in_channels, + embed_channels=enc_channels[0], + norm_layer=bn_layer, + act_layer=act_layer, + ) + + # encoder + enc_drop_path = [ + x.item() for x in torch.linspace(0, drop_path, sum(enc_depths)) + ] + self.enc = PointSequential() + for s in range(self.num_stages): + enc_drop_path_ = enc_drop_path[ + sum(enc_depths[:s]) : sum(enc_depths[: s + 1]) + ] + enc = PointSequential() + if s > 0: + enc.add( + SerializedPooling( + in_channels=enc_channels[s - 1], + out_channels=enc_channels[s], + stride=stride[s - 1], + norm_layer=bn_layer, + act_layer=act_layer, + ), + name="down", + ) + for i in range(enc_depths[s]): + enc.add( + Block( + channels=enc_channels[s], + num_heads=enc_num_head[s], + patch_size=enc_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=enc_drop_path_[i], + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + ), + name=f"block{i}", + ) + if len(enc) != 0: + self.enc.add(module=enc, name=f"enc{s}") + + # decoder + if not self.enc_mode: + dec_drop_path = [ + x.item() for x in torch.linspace(0, drop_path, sum(dec_depths)) + ] + self.dec = PointSequential() + dec_channels = list(dec_channels) + [enc_channels[-1]] + for s in reversed(range(self.num_stages - 1)): + dec_drop_path_ = dec_drop_path[ + sum(dec_depths[:s]) : sum(dec_depths[: s + 1]) + ] + dec_drop_path_.reverse() + dec = PointSequential() + dec.add( + SerializedUnpooling( + in_channels=dec_channels[s + 1], + skip_channels=enc_channels[s], + out_channels=dec_channels[s], + norm_layer=bn_layer, + act_layer=act_layer, + ), + name="up", + ) + for i in range(dec_depths[s]): + dec.add( + Block( + channels=dec_channels[s], + num_heads=dec_num_head[s], + patch_size=dec_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=dec_drop_path_[i], + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + ), + name=f"block{i}", + ) + self.dec.add(module=dec, name=f"dec{s}") + + def forward(self, data_dict): + point = Point(data_dict) + point.serialization(order=self.order, shuffle_orders=self.shuffle_orders) + point.sparsify() + + point = self.embedding(point) + point = self.enc(point) + if not self.enc_mode: + point = self.dec(point) + # else: + # point.feat = torch_scatter.segment_csr( + # src=point.feat, + # indptr=nn.functional.pad(point.offset, (1, 0)), + # reduce="mean", + # ) + return point diff --git a/pointcept/models/point_transformer_v3/point_transformer_v3m2_sonata.py b/pointcept/models/point_transformer_v3/point_transformer_v3m2_sonata.py new file mode 100644 index 0000000000000000000000000000000000000000..604443f30bec67c912a9e671c26dd36940bd1d2a --- /dev/null +++ b/pointcept/models/point_transformer_v3/point_transformer_v3m2_sonata.py @@ -0,0 +1,739 @@ +""" +Point Transformer - V3 Mode2 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from addict import Dict +import torch +import torch.nn as nn +from torch.nn.init import trunc_normal_ +import spconv.pytorch as spconv +import torch_scatter +from timm.layers import DropPath + +try: + import flash_attn +except ImportError: + flash_attn = None + +from pointcept.models.builder import MODELS +from pointcept.models.utils.misc import offset2bincount +from pointcept.models.utils.structure import Point +from pointcept.models.modules import PointModule, PointSequential + + +class LayerScale(nn.Module): + def __init__( + self, + dim: int, + init_values: float = 1e-5, + inplace: bool = False, + ) -> None: + super().__init__() + self.inplace = inplace + self.gamma = nn.Parameter(init_values * torch.ones(dim)) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return x.mul_(self.gamma) if self.inplace else x * self.gamma + + +class RPE(torch.nn.Module): + def __init__(self, patch_size, num_heads): + super().__init__() + self.patch_size = patch_size + self.num_heads = num_heads + self.pos_bnd = int((4 * patch_size) ** (1 / 3) * 2) + self.rpe_num = 2 * self.pos_bnd + 1 + self.rpe_table = torch.nn.Parameter(torch.zeros(3 * self.rpe_num, num_heads)) + torch.nn.init.trunc_normal_(self.rpe_table, std=0.02) + + def forward(self, coord): + idx = ( + coord.clamp(-self.pos_bnd, self.pos_bnd) # clamp into bnd + + self.pos_bnd # relative position to positive index + + torch.arange(3, device=coord.device) * self.rpe_num # x, y, z stride + ) + out = self.rpe_table.index_select(0, idx.reshape(-1)) + out = out.view(idx.shape + (-1,)).sum(3) + out = out.permute(0, 3, 1, 2) # (N, K, K, H) -> (N, H, K, K) + return out + + +class SerializedAttention(PointModule): + def __init__( + self, + channels, + num_heads, + patch_size, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + order_index=0, + enable_rpe=False, + enable_flash=True, + upcast_attention=True, + upcast_softmax=True, + ): + super().__init__() + assert channels % num_heads == 0 + self.channels = channels + self.num_heads = num_heads + self.scale = qk_scale or (channels // num_heads) ** -0.5 + self.order_index = order_index + self.upcast_attention = upcast_attention + self.upcast_softmax = upcast_softmax + self.enable_rpe = enable_rpe + self.enable_flash = enable_flash + if enable_flash: + assert ( + enable_rpe is False + ), "Set enable_rpe to False when enable Flash Attention" + assert ( + upcast_attention is False + ), "Set upcast_attention to False when enable Flash Attention" + assert ( + upcast_softmax is False + ), "Set upcast_softmax to False when enable Flash Attention" + assert flash_attn is not None, "Make sure flash_attn is installed." + self.patch_size = patch_size + self.attn_drop = attn_drop + self.flash_dtype = None + else: + # when disable flash attention, we still don't want to use mask + # consequently, patch size will auto set to the + # min number of patch_size_max and number of points + self.patch_size_max = patch_size + self.patch_size = 0 + self.attn_drop = torch.nn.Dropout(attn_drop) + + self.qkv = torch.nn.Linear(channels, channels * 3, bias=qkv_bias) + self.proj = torch.nn.Linear(channels, channels) + self.proj_drop = torch.nn.Dropout(proj_drop) + self.softmax = torch.nn.Softmax(dim=-1) + self.rpe = RPE(patch_size, num_heads) if self.enable_rpe else None + + @torch.no_grad() + def get_rel_pos(self, point, order): + K = self.patch_size + rel_pos_key = f"rel_pos_{self.order_index}" + if rel_pos_key not in point.keys(): + grid_coord = point.grid_coord[order] + grid_coord = grid_coord.reshape(-1, K, 3) + point[rel_pos_key] = grid_coord.unsqueeze(2) - grid_coord.unsqueeze(1) + return point[rel_pos_key] + + @torch.no_grad() + def get_padding_and_inverse(self, point): + pad_key = "pad" + unpad_key = "unpad" + cu_seqlens_key = "cu_seqlens_key" + if ( + pad_key not in point.keys() + or unpad_key not in point.keys() + or cu_seqlens_key not in point.keys() + ): + offset = point.offset + bincount = offset2bincount(offset) + bincount_pad = ( + torch.div( + bincount + self.patch_size - 1, + self.patch_size, + rounding_mode="trunc", + ) + * self.patch_size + ) + # only pad point when num of points larger than patch_size + mask_pad = bincount > self.patch_size + bincount_pad = ~mask_pad * bincount + mask_pad * bincount_pad + _offset = nn.functional.pad(offset, (1, 0)) + _offset_pad = nn.functional.pad(torch.cumsum(bincount_pad, dim=0), (1, 0)) + pad = torch.arange(_offset_pad[-1], device=offset.device) + unpad = torch.arange(_offset[-1], device=offset.device) + cu_seqlens = [] + for i in range(len(offset)): + unpad[_offset[i] : _offset[i + 1]] += _offset_pad[i] - _offset[i] + if bincount[i] != bincount_pad[i]: + pad[ + _offset_pad[i + 1] + - self.patch_size + + (bincount[i] % self.patch_size) : _offset_pad[i + 1] + ] = pad[ + _offset_pad[i + 1] + - 2 * self.patch_size + + (bincount[i] % self.patch_size) : _offset_pad[i + 1] + - self.patch_size + ] + pad[_offset_pad[i] : _offset_pad[i + 1]] -= _offset_pad[i] - _offset[i] + cu_seqlens.append( + torch.arange( + _offset_pad[i], + _offset_pad[i + 1], + step=self.patch_size, + dtype=torch.int32, + device=offset.device, + ) + ) + point[pad_key] = pad + point[unpad_key] = unpad + point[cu_seqlens_key] = nn.functional.pad( + torch.concat(cu_seqlens), (0, 1), value=_offset_pad[-1] + ) + return point[pad_key], point[unpad_key], point[cu_seqlens_key] + + def forward(self, point): + if not self.enable_flash: + self.patch_size = min( + offset2bincount(point.offset).min().tolist(), self.patch_size_max + ) + + H = self.num_heads + K = self.patch_size + C = self.channels + + pad, unpad, cu_seqlens = self.get_padding_and_inverse(point) + + order = point.serialized_order[self.order_index][pad] + inverse = unpad[point.serialized_inverse[self.order_index]] + + # padding and reshape feat and batch for serialized point patch + qkv = self.qkv(point.feat)[order] + + if not self.enable_flash: + # encode and reshape qkv: (N', K, 3, H, C') => (3, N', H, K, C') + q, k, v = ( + qkv.reshape(-1, K, 3, H, C // H).permute(2, 0, 3, 1, 4).unbind(dim=0) + ) + # attn + if self.upcast_attention: + q = q.float() + k = k.float() + attn = (q * self.scale) @ k.transpose(-2, -1) # (N', H, K, K) + if self.enable_rpe: + attn = attn + self.rpe(self.get_rel_pos(point, order)) + if self.upcast_softmax: + attn = attn.float() + attn = self.softmax(attn) + attn = self.attn_drop(attn).to(qkv.dtype) + feat = (attn @ v).transpose(1, 2).reshape(-1, C) + else: + if self.flash_dtype is None: + self.flash_dtype = ( + torch.bfloat16 + if torch.cuda.is_available() and torch.cuda.is_bf16_supported() + else torch.float16 + ) + feat = flash_attn.flash_attn_varlen_qkvpacked_func( + qkv.to(self.flash_dtype).reshape(-1, 3, H, C // H), + cu_seqlens, + max_seqlen=self.patch_size, + dropout_p=self.attn_drop if self.training else 0, + softmax_scale=self.scale, + ).reshape(-1, C) + feat = feat.to(qkv.dtype) + feat = feat[inverse] + + # ffn + feat = self.proj(feat) + feat = self.proj_drop(feat) + point.feat = feat + return point + + +class MLP(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=None, + out_channels=None, + act_layer=nn.GELU, + drop=0.0, + ): + super().__init__() + out_channels = out_channels or in_channels + hidden_channels = hidden_channels or in_channels + self.fc1 = nn.Linear(in_channels, hidden_channels) + self.act = act_layer() + self.fc2 = nn.Linear(hidden_channels, out_channels) + self.drop = nn.Dropout(drop) + + def forward(self, x): + x = self.fc1(x) + x = self.act(x) + x = self.drop(x) + x = self.fc2(x) + x = self.drop(x) + return x + + +class Block(PointModule): + def __init__( + self, + channels, + num_heads, + patch_size=48, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.0, + layer_scale=None, + norm_layer=nn.LayerNorm, + act_layer=nn.GELU, + pre_norm=True, + order_index=0, + cpe_indice_key=None, + enable_rpe=False, + enable_flash=True, + upcast_attention=True, + upcast_softmax=True, + ): + super().__init__() + self.channels = channels + self.pre_norm = pre_norm + + self.cpe = PointSequential( + spconv.SubMConv3d( + channels, + channels, + kernel_size=3, + bias=True, + indice_key=cpe_indice_key, + ), + nn.Linear(channels, channels), + norm_layer(channels), + ) + + self.norm1 = PointSequential(norm_layer(channels)) + self.ls1 = PointSequential( + LayerScale(channels, init_values=layer_scale) + if layer_scale is not None + else nn.Identity() + ) + self.attn = SerializedAttention( + channels=channels, + patch_size=patch_size, + num_heads=num_heads, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + order_index=order_index, + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + ) + self.norm2 = PointSequential(norm_layer(channels)) + self.ls2 = PointSequential( + LayerScale(channels, init_values=layer_scale) + if layer_scale is not None + else nn.Identity() + ) + self.mlp = PointSequential( + MLP( + in_channels=channels, + hidden_channels=int(channels * mlp_ratio), + out_channels=channels, + act_layer=act_layer, + drop=proj_drop, + ) + ) + self.drop_path = PointSequential( + DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + ) + + def forward(self, point: Point): + shortcut = point.feat + point = self.cpe(point) + point.feat = shortcut + point.feat + shortcut = point.feat + if self.pre_norm: + point = self.norm1(point) + point = self.drop_path(self.ls1(self.attn(point))) + point.feat = shortcut + point.feat + if not self.pre_norm: + point = self.norm1(point) + + shortcut = point.feat + if self.pre_norm: + point = self.norm2(point) + point = self.drop_path(self.ls2(self.mlp(point))) + point.feat = shortcut + point.feat + if not self.pre_norm: + point = self.norm2(point) + point.sparse_conv_feat = point.sparse_conv_feat.replace_feature(point.feat) + return point + + +class GridPooling(PointModule): + def __init__( + self, + in_channels, + out_channels, + stride=2, + norm_layer=None, + act_layer=None, + reduce="max", + shuffle_orders=True, + traceable=True, # record parent and cluster + ): + super().__init__() + self.in_channels = in_channels + self.out_channels = out_channels + + self.stride = stride + assert reduce in ["sum", "mean", "min", "max"] + self.reduce = reduce + self.shuffle_orders = shuffle_orders + self.traceable = traceable + + self.proj = nn.Linear(in_channels, out_channels) + if norm_layer is not None: + self.norm = PointSequential(norm_layer(out_channels)) + if act_layer is not None: + self.act = PointSequential(act_layer()) + + def forward(self, point: Point): + if "grid_coord" in point.keys(): + grid_coord = point.grid_coord + elif {"coord", "grid_size"}.issubset(point.keys()): + grid_coord = torch.div( + point.coord - point.coord.min(0)[0], + point.grid_size, + rounding_mode="trunc", + ).int() + else: + raise AssertionError( + "[gird_coord] or [coord, grid_size] should be include in the Point" + ) + grid_coord = torch.div(grid_coord, self.stride, rounding_mode="trunc") + grid_coord = grid_coord | point.batch.view(-1, 1) << 48 + grid_coord, cluster, counts = torch.unique( + grid_coord, + sorted=True, + return_inverse=True, + return_counts=True, + dim=0, + ) + grid_coord = grid_coord & ((1 << 48) - 1) + # indices of point sorted by cluster, for torch_scatter.segment_csr + _, indices = torch.sort(cluster) + # index pointer for sorted point, for torch_scatter.segment_csr + idx_ptr = torch.cat([counts.new_zeros(1), torch.cumsum(counts, dim=0)]) + # head_indices of each cluster, for reduce attr e.g. code, batch + head_indices = indices[idx_ptr[:-1]] + point_dict = Dict( + feat=torch_scatter.segment_csr( + self.proj(point.feat)[indices], idx_ptr, reduce=self.reduce + ), + coord=torch_scatter.segment_csr( + point.coord[indices], idx_ptr, reduce="mean" + ), + grid_coord=grid_coord, + batch=point.batch[head_indices], + ) + if "origin_coord" in point.keys(): + point_dict["origin_coord"] = torch_scatter.segment_csr( + point.origin_coord[indices], idx_ptr, reduce="mean" + ) + if "condition" in point.keys(): + point_dict["condition"] = point.condition + if "context" in point.keys(): + point_dict["context"] = point.context + if "name" in point.keys(): + point_dict["name"] = point.name + if "split" in point.keys(): + point_dict["split"] = point.split + if "color" in point.keys(): + point_dict["color"] = torch_scatter.segment_csr( + point.color[indices], idx_ptr, reduce="mean" + ) + if "grid_size" in point.keys(): + point_dict["grid_size"] = point.grid_size * self.stride + + if self.traceable: + point_dict["pooling_inverse"] = cluster + point_dict["pooling_parent"] = point + point_dict["idx_ptr"] = idx_ptr + order = point.order + point = Point(point_dict) + if self.norm is not None: + point = self.norm(point) + if self.act is not None: + point = self.act(point) + point.serialization(order=order, shuffle_orders=self.shuffle_orders) + point.sparsify() + return point + + +class GridUnpooling(PointModule): + def __init__( + self, + in_channels, + skip_channels, + out_channels, + norm_layer=None, + act_layer=None, + traceable=False, # record parent and cluster + ): + super().__init__() + self.proj = PointSequential(nn.Linear(in_channels, out_channels)) + self.proj_skip = PointSequential(nn.Linear(skip_channels, out_channels)) + + if norm_layer is not None: + self.proj.add(norm_layer(out_channels)) + self.proj_skip.add(norm_layer(out_channels)) + + if act_layer is not None: + self.proj.add(act_layer()) + self.proj_skip.add(act_layer()) + + self.traceable = traceable + + def forward(self, point): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pooling_inverse + feat = point.feat + + parent = self.proj_skip(parent) + parent.feat = parent.feat + self.proj(point).feat[inverse] + parent.sparse_conv_feat = parent.sparse_conv_feat.replace_feature(parent.feat) + + if self.traceable: + point.feat = feat + parent["unpooling_parent"] = point + return parent + + +class Embedding(PointModule): + def __init__( + self, + in_channels, + embed_channels, + norm_layer=None, + act_layer=None, + mask_token=False, + ): + super().__init__() + self.in_channels = in_channels + self.embed_channels = embed_channels + + self.stem = PointSequential(linear=nn.Linear(in_channels, embed_channels)) + if norm_layer is not None: + self.stem.add(norm_layer(embed_channels), name="norm") + if act_layer is not None: + self.stem.add(act_layer(), name="act") + + if mask_token: + self.mask_token = nn.Parameter(torch.zeros(1, embed_channels)) + else: + self.mask_token = None + + def forward(self, point: Point): + point = self.stem(point) + if "mask" in point.keys(): + point.feat = torch.where( + point.mask.unsqueeze(-1), + self.mask_token.to(point.feat.dtype), + point.feat, + ) + return point + + +@MODELS.register_module("PT-v3m2") +class PointTransformerV3(PointModule): + def __init__( + self, + in_channels=6, + order=("z", "z-trans"), + stride=(2, 2, 2, 2), + enc_depths=(2, 2, 2, 6, 2), + enc_channels=(32, 64, 128, 256, 512), + enc_num_head=(2, 4, 8, 16, 32), + enc_patch_size=(48, 48, 48, 48, 48), + dec_depths=(2, 2, 2, 2), + dec_channels=(64, 64, 128, 256), + dec_num_head=(4, 4, 8, 16), + dec_patch_size=(48, 48, 48, 48), + mlp_ratio=4, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.3, + layer_scale=None, + pre_norm=True, + shuffle_orders=True, + enable_rpe=False, + enable_flash=True, + upcast_attention=False, + upcast_softmax=False, + traceable=False, + mask_token=False, + enc_mode=False, + freeze_encoder=False, + ): + super().__init__() + self.num_stages = len(enc_depths) + self.order = [order] if isinstance(order, str) else order + self.shuffle_orders = shuffle_orders + self.enc_mode = enc_mode + self.freeze_encoder = freeze_encoder + + assert self.num_stages == len(stride) + 1 + assert self.num_stages == len(enc_depths) + assert self.num_stages == len(enc_channels) + assert self.num_stages == len(enc_num_head) + assert self.num_stages == len(enc_patch_size) + assert self.enc_mode or self.num_stages == len(dec_depths) + 1 + assert self.enc_mode or self.num_stages == len(dec_channels) + 1 + assert self.enc_mode or self.num_stages == len(dec_num_head) + 1 + assert self.enc_mode or self.num_stages == len(dec_patch_size) + 1 + + # normalization layer + ln_layer = nn.LayerNorm + # activation layers + act_layer = nn.GELU + + self.embedding = Embedding( + in_channels=in_channels, + embed_channels=enc_channels[0], + norm_layer=ln_layer, + act_layer=act_layer, + mask_token=mask_token, + ) + + # encoder + enc_drop_path = [ + x.item() for x in torch.linspace(0, drop_path, sum(enc_depths)) + ] + self.enc = PointSequential() + for s in range(self.num_stages): + enc_drop_path_ = enc_drop_path[ + sum(enc_depths[:s]) : sum(enc_depths[: s + 1]) + ] + enc = PointSequential() + if s > 0: + enc.add( + GridPooling( + in_channels=enc_channels[s - 1], + out_channels=enc_channels[s], + stride=stride[s - 1], + norm_layer=ln_layer, + act_layer=act_layer, + ), + name="down", + ) + for i in range(enc_depths[s]): + enc.add( + Block( + channels=enc_channels[s], + num_heads=enc_num_head[s], + patch_size=enc_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=enc_drop_path_[i], + layer_scale=layer_scale, + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + ), + name=f"block{i}", + ) + if len(enc) != 0: + self.enc.add(module=enc, name=f"enc{s}") + + # decoder + if not self.enc_mode: + dec_drop_path = [ + x.item() for x in torch.linspace(0, drop_path, sum(dec_depths)) + ] + self.dec = PointSequential() + dec_channels = list(dec_channels) + [enc_channels[-1]] + for s in reversed(range(self.num_stages - 1)): + dec_drop_path_ = dec_drop_path[ + sum(dec_depths[:s]) : sum(dec_depths[: s + 1]) + ] + dec_drop_path_.reverse() + dec = PointSequential() + dec.add( + GridUnpooling( + in_channels=dec_channels[s + 1], + skip_channels=enc_channels[s], + out_channels=dec_channels[s], + norm_layer=ln_layer, + act_layer=act_layer, + traceable=traceable, + ), + name="up", + ) + for i in range(dec_depths[s]): + dec.add( + Block( + channels=dec_channels[s], + num_heads=dec_num_head[s], + patch_size=dec_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=dec_drop_path_[i], + layer_scale=layer_scale, + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + ), + name=f"block{i}", + ) + self.dec.add(module=dec, name=f"dec{s}") + if self.freeze_encoder: + for p in self.embedding.parameters(): + p.requires_grad = False + for p in self.enc.parameters(): + p.requires_grad = False + self.apply(self._init_weights) + + @staticmethod + def _init_weights(module): + if isinstance(module, nn.Linear): + trunc_normal_(module.weight, std=0.02) + if module.bias is not None: + nn.init.zeros_(module.bias) + elif isinstance(module, spconv.SubMConv3d): + trunc_normal_(module.weight, std=0.02) + if module.bias is not None: + nn.init.zeros_(module.bias) + + def forward(self, data_dict): + point = Point(data_dict) + point = self.embedding(point) + + point.serialization(order=self.order, shuffle_orders=self.shuffle_orders) + point.sparsify() + + point = self.enc(point) + if not self.enc_mode: + point = self.dec(point) + return point diff --git a/pointcept/models/point_transformer_v3/point_transformer_v3m3_utonia.py b/pointcept/models/point_transformer_v3/point_transformer_v3m3_utonia.py new file mode 100644 index 0000000000000000000000000000000000000000..1d7b3ae6d32db81bbf45d28c63b9c6988ab427c3 --- /dev/null +++ b/pointcept/models/point_transformer_v3/point_transformer_v3m3_utonia.py @@ -0,0 +1,930 @@ +""" +Point Transformer - V3 Mode3 + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from addict import Dict +import torch +import torch.nn as nn +from torch.nn.init import trunc_normal_ +import spconv.pytorch as spconv +import torch_scatter +from timm.layers import DropPath +import math + +try: + import flash_attn +except ImportError: + flash_attn = None + +from pointcept.models.builder import MODELS +from pointcept.models.utils.misc import offset2bincount +from pointcept.models.utils.structure import Point +from pointcept.models.modules import PointModule, PointSequential + + +class LayerScale(nn.Module): + def __init__( + self, + dim: int, + init_values: float = 1e-5, + inplace: bool = False, + ) -> None: + super().__init__() + self.inplace = inplace + self.gamma = nn.Parameter(init_values * torch.ones(dim)) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return x.mul_(self.gamma) if self.inplace else x * self.gamma + + +class Point3DRoPE(nn.Module): + def __init__(self, head_dim, base=10000): + super().__init__() + assert ( + head_dim % 3 == 0 + ), f"Head dimension must be divisible by 3 for 3D RoPE, {head_dim}" + + self.head_dim = head_dim + self.chunk_dim = head_dim // 3 + self.base = base + inv_freq = 1.0 / ( + self.base ** (torch.arange(0, self.chunk_dim, 2).float() / self.chunk_dim) + ) + self.register_buffer("inv_freq", inv_freq) + + def get_cos_sin(self, xyz): + x = xyz[:, 0:1] + y = xyz[:, 1:2] + z = xyz[:, 2:3] + freqs = self.inv_freq.unsqueeze(0) + + # (Total_N, chunk_dim/2) + emb_x = x * freqs + emb_y = y * freqs + emb_z = z * freqs + + emb_x = torch.cat((emb_x, emb_x), dim=-1) + emb_y = torch.cat((emb_y, emb_y), dim=-1) + emb_z = torch.cat((emb_z, emb_z), dim=-1) + emb_3d = torch.cat((emb_x, emb_y, emb_z), dim=-1) + return emb_3d.cos().unsqueeze(1), emb_3d.sin().unsqueeze(1) + + def rotate_half(self, x): + x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :] + return torch.cat((-x2, x1), dim=-1) + + def forward(self, q, k, xyz): + cos, sin = self.get_cos_sin(xyz) + qs = torch.split(q, self.chunk_dim, dim=-1) + ks = torch.split(k, self.chunk_dim, dim=-1) + + coss = torch.split(cos, self.chunk_dim, dim=-1) + sins = torch.split(sin, self.chunk_dim, dim=-1) + + q_outs = [] + k_outs = [] + + for i in range(3): + q_part = (qs[i] * coss[i]) + (self.rotate_half(qs[i]) * sins[i]) + k_part = (ks[i] * coss[i]) + (self.rotate_half(ks[i]) * sins[i]) + + q_outs.append(q_part) + k_outs.append(k_part) + + q_rot = torch.cat(q_outs, dim=-1) + k_rot = torch.cat(k_outs, dim=-1) + + return q_rot, k_rot + + +class RPE(torch.nn.Module): + def __init__(self, patch_size, num_heads): + super().__init__() + self.patch_size = patch_size + self.num_heads = num_heads + self.pos_bnd = int((4 * patch_size) ** (1 / 3) * 2) + self.rpe_num = 2 * self.pos_bnd + 1 + self.rpe_table = torch.nn.Parameter(torch.zeros(3 * self.rpe_num, num_heads)) + torch.nn.init.trunc_normal_(self.rpe_table, std=0.02) + + def forward(self, coord): + idx = ( + coord.clamp(-self.pos_bnd, self.pos_bnd) # clamp into bnd + + self.pos_bnd # relative position to positive index + + torch.arange(3, device=coord.device) * self.rpe_num # x, y, z stride + ) + out = self.rpe_table.index_select(0, idx.reshape(-1)) + out = out.view(idx.shape + (-1,)).sum(3) + out = out.permute(0, 3, 1, 2) # (N, K, K, H) -> (N, H, K, K) + return out + + +class SerializedAttention(PointModule): + def __init__( + self, + channels, + num_heads, + patch_size, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + order_index=0, + enable_rpe=False, + enable_flash=True, + upcast_attention=True, + upcast_softmax=True, + rope_base=10000, + shift_coords=None, + jitter_coords=None, + rescale_coords=None, + ): + super().__init__() + assert channels % num_heads == 0 + self.channels = channels + self.num_heads = num_heads + self.scale = qk_scale or (channels // num_heads) ** -0.5 + self.order_index = order_index + self.upcast_attention = upcast_attention + self.upcast_softmax = upcast_softmax + self.enable_rpe = enable_rpe + self.enable_flash = enable_flash + if enable_flash: + assert ( + enable_rpe is False + ), "Set enable_rpe to False when enable Flash Attention" + assert ( + upcast_attention is False + ), "Set upcast_attention to False when enable Flash Attention" + assert ( + upcast_softmax is False + ), "Set upcast_softmax to False when enable Flash Attention" + assert flash_attn is not None, "Make sure flash_attn is installed." + self.patch_size = patch_size + self.attn_drop = attn_drop + self.flash_dtype = None + else: + # when disable flash attention, we still don't want to use mask + # consequently, patch size will auto set to the + # min number of patch_size_max and number of points + self.patch_size_max = patch_size + self.patch_size = 0 + self.attn_drop = torch.nn.Dropout(attn_drop) + + self.qkv = torch.nn.Linear(channels, channels * 3, bias=qkv_bias) + self.proj = torch.nn.Linear(channels, channels) + self.proj_drop = torch.nn.Dropout(proj_drop) + self.softmax = torch.nn.Softmax(dim=-1) + self.rpe = RPE(patch_size, num_heads) if self.enable_rpe else None + self.rope_base = rope_base + if rope_base: + self.rope = Point3DRoPE(head_dim=channels // num_heads, base=rope_base) + self.shift_coords = shift_coords + self.jitter_coords = jitter_coords + self.rescale_coords = rescale_coords + + @torch.no_grad() + def get_rel_pos(self, point, order): + K = self.patch_size + rel_pos_key = f"rel_pos_{self.order_index}" + if rel_pos_key not in point.keys(): + grid_coord = point.grid_coord[order] + grid_coord = grid_coord.reshape(-1, K, 3) + point[rel_pos_key] = grid_coord.unsqueeze(2) - grid_coord.unsqueeze(1) + return point[rel_pos_key] + + @torch.no_grad() + def get_padding_and_inverse(self, point): + pad_key = "pad" + unpad_key = "unpad" + cu_seqlens_key = "cu_seqlens_key" + if ( + pad_key not in point.keys() + or unpad_key not in point.keys() + or cu_seqlens_key not in point.keys() + ): + offset = point.offset + bincount = offset2bincount(offset) + bincount_pad = ( + torch.div( + bincount + self.patch_size - 1, + self.patch_size, + rounding_mode="trunc", + ) + * self.patch_size + ) + # only pad point when num of points larger than patch_size + mask_pad = bincount > self.patch_size + bincount_pad = ~mask_pad * bincount + mask_pad * bincount_pad + _offset = nn.functional.pad(offset, (1, 0)) + _offset_pad = nn.functional.pad(torch.cumsum(bincount_pad, dim=0), (1, 0)) + pad = torch.arange(_offset_pad[-1], device=offset.device) + unpad = torch.arange(_offset[-1], device=offset.device) + cu_seqlens = [] + for i in range(len(offset)): + unpad[_offset[i] : _offset[i + 1]] += _offset_pad[i] - _offset[i] + if bincount[i] != bincount_pad[i]: + pad[ + _offset_pad[i + 1] + - self.patch_size + + (bincount[i] % self.patch_size) : _offset_pad[i + 1] + ] = pad[ + _offset_pad[i + 1] + - 2 * self.patch_size + + (bincount[i] % self.patch_size) : _offset_pad[i + 1] + - self.patch_size + ] + pad[_offset_pad[i] : _offset_pad[i + 1]] -= _offset_pad[i] - _offset[i] + cu_seqlens.append( + torch.arange( + _offset_pad[i], + _offset_pad[i + 1], + step=self.patch_size, + dtype=torch.int32, + device=offset.device, + ) + ) + point[pad_key] = pad + point[unpad_key] = unpad + point[cu_seqlens_key] = nn.functional.pad( + torch.concat(cu_seqlens), (0, 1), value=_offset_pad[-1] + ) + return point[pad_key], point[unpad_key], point[cu_seqlens_key] + + def forward(self, point): + if not self.enable_flash: + self.patch_size = min( + offset2bincount(point.offset).min().tolist(), self.patch_size_max + ) + + H = self.num_heads + K = self.patch_size + C = self.channels + + pad, unpad, cu_seqlens = self.get_padding_and_inverse(point) + + order = point.serialized_order[self.order_index][pad] + inverse = unpad[point.serialized_inverse[self.order_index]] + + # padding and reshape feat and batch for serialized point patch + qkv = self.qkv(point.feat)[order] + + # RoPE aug + if self.rope_base: + rope_coord = point.coord[order].clone() + if self.training: + dd = {"device": rope_coord.device, "dtype": rope_coord.dtype} + if self.shift_coords is not None and self.shift_coords > 0: + shift_xyz = torch.empty(3, **dd).uniform_( + -self.shift_coords, self.shift_coords + ) + rope_coord += shift_xyz.unsqueeze(0) + + if self.jitter_coords is not None and self.jitter_coords > 1.0: + jitter_max = math.log(self.jitter_coords) + jitter_min = -jitter_max + jitter_xyz = ( + torch.empty(3, **dd).uniform_(jitter_min, jitter_max).exp() + ) + rope_coord *= jitter_xyz.unsqueeze(0) + + if self.rescale_coords is not None and self.rescale_coords > 1.0: + rescale_max = math.log(self.rescale_coords) + rescale_min = -rescale_max + rescale_val = ( + torch.empty(1, **dd).uniform_(rescale_min, rescale_max).exp() + ) + rope_coord *= rescale_val + + qkv = qkv.reshape(-1, 3, H, C // H) + q, k, v = qkv.unbind(dim=1) + q, k = self.rope(q, k, rope_coord) + + if not self.enable_flash: + q = q.reshape(-1, K, H, C // H).permute(0, 2, 1, 3) + k = k.reshape(-1, K, H, C // H).permute(0, 2, 1, 3) + v = v.reshape(-1, K, H, C // H).permute(0, 2, 1, 3) + # attn + if self.upcast_attention: + q = q.float() + k = k.float() + attn = (q * self.scale) @ k.transpose(-2, -1) # (N', H, K, K) + if self.enable_rpe: + attn = attn + self.rpe(self.get_rel_pos(point, order)) + if self.upcast_softmax: + attn = attn.float() + attn = self.softmax(attn) + attn = self.attn_drop(attn).to(qkv.dtype) + feat = (attn @ v).transpose(1, 2).reshape(-1, C) + else: + qkv_roped = torch.stack([q, k, v], dim=1) + if self.flash_dtype is None: + self.flash_dtype = ( + torch.bfloat16 + if torch.cuda.is_available() and torch.cuda.is_bf16_supported() + else torch.float16 + ) + feat = flash_attn.flash_attn_varlen_qkvpacked_func( + qkv_roped.to(self.flash_dtype), + cu_seqlens, + max_seqlen=self.patch_size, + dropout_p=self.attn_drop if self.training else 0, + softmax_scale=self.scale, + ).reshape(-1, C) + feat = feat.to(qkv.dtype) + feat = feat[inverse] + else: + if not self.enable_flash: + # encode and reshape qkv: (N', K, 3, H, C') => (3, N', H, K, C') + q, k, v = ( + qkv.reshape(-1, K, 3, H, C // H) + .permute(2, 0, 3, 1, 4) + .unbind(dim=0) + ) + # attn + if self.upcast_attention: + q = q.float() + k = k.float() + attn = (q * self.scale) @ k.transpose(-2, -1) # (N', H, K, K) + if self.enable_rpe: + attn = attn + self.rpe(self.get_rel_pos(point, order)) + if self.upcast_softmax: + attn = attn.float() + attn = self.softmax(attn) + attn = self.attn_drop(attn).to(qkv.dtype) + feat = (attn @ v).transpose(1, 2).reshape(-1, C) + else: + if self.flash_dtype is None: + self.flash_dtype = ( + torch.bfloat16 + if torch.cuda.is_available() and torch.cuda.is_bf16_supported() + else torch.float16 + ) + feat = flash_attn.flash_attn_varlen_qkvpacked_func( + qkv.to(self.flash_dtype).reshape(-1, 3, H, C // H), + cu_seqlens, + max_seqlen=self.patch_size, + dropout_p=self.attn_drop if self.training else 0, + softmax_scale=self.scale, + ).reshape(-1, C) + feat = feat.to(qkv.dtype) + feat = feat[inverse] + + # ffn + feat = self.proj(feat) + feat = self.proj_drop(feat) + point.feat = feat + return point + + +class MLP(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=None, + out_channels=None, + act_layer=nn.GELU, + drop=0.0, + ): + super().__init__() + out_channels = out_channels or in_channels + hidden_channels = hidden_channels or in_channels + self.fc1 = nn.Linear(in_channels, hidden_channels) + self.act = act_layer() + self.fc2 = nn.Linear(hidden_channels, out_channels) + self.drop = nn.Dropout(drop) + + def forward(self, x): + x = self.fc1(x) + x = self.act(x) + x = self.drop(x) + x = self.fc2(x) + x = self.drop(x) + return x + + +class Block(PointModule): + def __init__( + self, + channels, + num_heads, + patch_size=48, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.0, + layer_scale=None, + norm_layer=nn.LayerNorm, + act_layer=nn.GELU, + pre_norm=True, + order_index=0, + cpe_indice_key=None, + enable_rpe=False, + enable_flash=True, + upcast_attention=True, + upcast_softmax=True, + rope_base=10000, + shift_coords=None, + jitter_coords=None, + rescale_coords=None, + ): + super().__init__() + self.channels = channels + self.pre_norm = pre_norm + + self.cpe = PointSequential( + spconv.SubMConv3d( + channels, + channels, + kernel_size=3, + bias=True, + indice_key=cpe_indice_key, + ), + nn.Linear(channels, channels), + norm_layer(channels), + ) + # self.cpe = PointSequential( + # nn.Linear(channels, channels), + # norm_layer(channels), + # ) + + self.norm1 = PointSequential(norm_layer(channels)) + self.ls1 = PointSequential( + LayerScale(channels, init_values=layer_scale) + if layer_scale is not None + else nn.Identity() + ) + self.attn = SerializedAttention( + channels=channels, + patch_size=patch_size, + num_heads=num_heads, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + order_index=order_index, + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + rope_base=rope_base, + shift_coords=shift_coords, + jitter_coords=jitter_coords, + rescale_coords=rescale_coords, + ) + self.norm2 = PointSequential(norm_layer(channels)) + self.ls2 = PointSequential( + LayerScale(channels, init_values=layer_scale) + if layer_scale is not None + else nn.Identity() + ) + self.mlp = PointSequential( + MLP( + in_channels=channels, + hidden_channels=int(channels * mlp_ratio), + out_channels=channels, + act_layer=act_layer, + drop=proj_drop, + ) + ) + self.drop_path = PointSequential( + DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + ) + + def forward(self, point: Point): + shortcut = point.feat + point = self.cpe(point) + point.feat = shortcut + point.feat + shortcut = point.feat + if self.pre_norm: + point = self.norm1(point) + point = self.drop_path(self.ls1(self.attn(point))) + point.feat = shortcut + point.feat + if not self.pre_norm: + point = self.norm1(point) + + shortcut = point.feat + if self.pre_norm: + point = self.norm2(point) + point = self.drop_path(self.ls2(self.mlp(point))) + point.feat = shortcut + point.feat + if not self.pre_norm: + point = self.norm2(point) + point.sparse_conv_feat = point.sparse_conv_feat.replace_feature(point.feat) + return point + + +class GridPooling(PointModule): + def __init__( + self, + in_channels, + out_channels, + stride=2, + norm_layer=None, + act_layer=None, + reduce="max", + shuffle_orders=True, + traceable=True, # record parent and cluster + ): + super().__init__() + self.in_channels = in_channels + self.out_channels = out_channels + + self.stride = stride + assert reduce in ["sum", "mean", "min", "max"] + self.reduce = reduce + self.shuffle_orders = shuffle_orders + self.traceable = traceable + + self.proj = nn.Linear(in_channels, out_channels) + if norm_layer is not None: + self.norm = PointSequential(norm_layer(out_channels)) + if act_layer is not None: + self.act = PointSequential(act_layer()) + + def forward(self, point: Point): + if "grid_coord" in point.keys(): + grid_coord = point.grid_coord + elif {"coord", "grid_size"}.issubset(point.keys()): + grid_coord = torch.div( + point.coord - point.coord.min(0)[0], + point.grid_size, + rounding_mode="trunc", + ).int() + else: + raise AssertionError( + "[gird_coord] or [coord, grid_size] should be include in the Point" + ) + grid_coord = torch.div(grid_coord, self.stride, rounding_mode="trunc") + grid_coord = grid_coord | point.batch.view(-1, 1) << 48 + grid_coord, cluster, counts = torch.unique( + grid_coord, + sorted=True, + return_inverse=True, + return_counts=True, + dim=0, + ) + grid_coord = grid_coord & ((1 << 48) - 1) + # indices of point sorted by cluster, for torch_scatter.segment_csr + _, indices = torch.sort(cluster) + # index pointer for sorted point, for torch_scatter.segment_csr + idx_ptr = torch.cat([counts.new_zeros(1), torch.cumsum(counts, dim=0)]) + # head_indices of each cluster, for reduce attr e.g. code, batch + head_indices = indices[idx_ptr[:-1]] + point_dict = Dict( + feat=torch_scatter.segment_csr( + self.proj(point.feat)[indices], idx_ptr, reduce=self.reduce + ), + coord=torch_scatter.segment_csr( + point.coord[indices], idx_ptr, reduce="mean" + ), + grid_coord=grid_coord, + batch=point.batch[head_indices], + ) + if "origin_coord" in point.keys(): + point_dict["origin_coord"] = torch_scatter.segment_csr( + point.origin_coord[indices], idx_ptr, reduce="mean" + ) + if "condition" in point.keys(): + point_dict["condition"] = point.condition + if "context" in point.keys(): + point_dict["context"] = point.context + if "name" in point.keys(): + point_dict["name"] = point.name + if "split" in point.keys(): + point_dict["split"] = point.split + if "color" in point.keys(): + point_dict["color"] = torch_scatter.segment_csr( + point.color[indices], idx_ptr, reduce="mean" + ) + if "grid_size" in point.keys(): + point_dict["grid_size"] = point.grid_size * self.stride + + if self.traceable: + point_dict["pooling_inverse"] = cluster + point_dict["pooling_parent"] = point + point_dict["idx_ptr"] = idx_ptr + order = point.order + point = Point(point_dict) + if self.norm is not None: + point = self.norm(point) + if self.act is not None: + point = self.act(point) + point.serialization(order=order, shuffle_orders=self.shuffle_orders) + point.sparsify() + return point + + +class GridUnpooling(PointModule): + def __init__( + self, + in_channels, + skip_channels, + out_channels, + norm_layer=None, + act_layer=None, + traceable=False, # record parent and cluster + ): + super().__init__() + self.proj = PointSequential(nn.Linear(in_channels, out_channels)) + self.proj_skip = PointSequential(nn.Linear(skip_channels, out_channels)) + + if norm_layer is not None: + self.proj.add(norm_layer(out_channels)) + self.proj_skip.add(norm_layer(out_channels)) + + if act_layer is not None: + self.proj.add(act_layer()) + self.proj_skip.add(act_layer()) + + self.traceable = traceable + + def forward(self, point): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pooling_inverse + feat = point.feat + + parent = self.proj_skip(parent) + parent.feat = parent.feat + self.proj(point).feat[inverse] + parent.sparse_conv_feat = parent.sparse_conv_feat.replace_feature(parent.feat) + + if self.traceable: + point.feat = feat + parent["unpooling_parent"] = point + return parent + + +class Embedding(PointModule): + def __init__( + self, + in_channels, + embed_channels, + norm_layer=None, + act_layer=None, + mask_token=False, + ): + super().__init__() + self.in_channels = in_channels + self.embed_channels = embed_channels + + self.stem = PointSequential(linear=nn.Linear(in_channels, embed_channels)) + if norm_layer is not None: + self.stem.add(norm_layer(embed_channels), name="norm") + if act_layer is not None: + self.stem.add(act_layer(), name="act") + + if mask_token: + self.mask_token = nn.Parameter(torch.zeros(1, embed_channels)) + else: + self.mask_token = None + + def forward(self, point: Point): + point = self.stem(point) + if "mask" in point.keys(): + point.feat = torch.where( + point.mask.unsqueeze(-1), + self.mask_token.to(point.feat.dtype), + point.feat, + ) + return point + + +@MODELS.register_module("PT-v3m3") +class PointTransformerV3(PointModule): + def __init__( + self, + in_channels=6, + order=("z", "z-trans"), + stride=(2, 2, 2, 2), + enc_depths=(2, 2, 2, 6, 2), + enc_channels=(32, 64, 128, 256, 512), + enc_num_head=(2, 4, 8, 16, 32), + enc_patch_size=(48, 48, 48, 48, 48), + dec_depths=(2, 2, 2, 2), + dec_channels=(64, 64, 128, 256), + dec_num_head=(4, 4, 8, 16), + dec_patch_size=(48, 48, 48, 48), + dec_rope_enable=True, + mlp_ratio=4, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + drop_path=0.3, + layer_scale=None, + pre_norm=True, + shuffle_orders=True, + enable_rpe=False, + enable_flash=True, + upcast_attention=False, + upcast_softmax=False, + traceable=False, + mask_token=False, + enc_mode=False, + freeze_encoder=False, + rope_base=10000, + shift_coords=None, + jitter_coords=None, + rescale_coords=None, + ): + super().__init__() + self.num_stages = len(enc_depths) + self.order = [order] if isinstance(order, str) else order + self.shuffle_orders = shuffle_orders + self.enc_mode = enc_mode + self.freeze_encoder = freeze_encoder + + assert self.num_stages == len(stride) + 1 + assert self.num_stages == len(enc_depths) + assert self.num_stages == len(enc_channels) + assert self.num_stages == len(enc_num_head) + assert self.num_stages == len(enc_patch_size) + assert self.enc_mode or self.num_stages == len(dec_depths) + 1 + assert self.enc_mode or self.num_stages == len(dec_channels) + 1 + assert self.enc_mode or self.num_stages == len(dec_num_head) + 1 + assert self.enc_mode or self.num_stages == len(dec_patch_size) + 1 + + # normalization layer + ln_layer = nn.LayerNorm + # activation layers + act_layer = nn.GELU + + self.embedding = Embedding( + in_channels=in_channels, + embed_channels=enc_channels[0], + norm_layer=ln_layer, + act_layer=act_layer, + mask_token=mask_token, + ) + + # encoder + enc_drop_path = [ + x.item() for x in torch.linspace(0, drop_path, sum(enc_depths)) + ] + self.enc = PointSequential() + for s in range(self.num_stages): + enc_drop_path_ = enc_drop_path[ + sum(enc_depths[:s]) : sum(enc_depths[: s + 1]) + ] + enc = PointSequential() + if s > 0: + enc.add( + GridPooling( + in_channels=enc_channels[s - 1], + out_channels=enc_channels[s], + stride=stride[s - 1], + norm_layer=ln_layer, + act_layer=act_layer, + ), + name="down", + ) + for i in range(enc_depths[s]): + enc.add( + Block( + channels=enc_channels[s], + num_heads=enc_num_head[s], + patch_size=enc_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=enc_drop_path_[i], + layer_scale=layer_scale, + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + rope_base=rope_base, + shift_coords=shift_coords, + jitter_coords=jitter_coords, + rescale_coords=rescale_coords, + ), + name=f"block{i}", + ) + if len(enc) != 0: + self.enc.add(module=enc, name=f"enc{s}") + + # decoder + if not self.enc_mode: + dec_drop_path = [ + x.item() for x in torch.linspace(0, drop_path, sum(dec_depths)) + ] + self.dec = PointSequential() + dec_channels = list(dec_channels) + [enc_channels[-1]] + for s in reversed(range(self.num_stages - 1)): + dec_drop_path_ = dec_drop_path[ + sum(dec_depths[:s]) : sum(dec_depths[: s + 1]) + ] + dec_drop_path_.reverse() + dec = PointSequential() + dec.add( + GridUnpooling( + in_channels=dec_channels[s + 1], + skip_channels=enc_channels[s], + out_channels=dec_channels[s], + norm_layer=ln_layer, + act_layer=act_layer, + traceable=traceable, + ), + name="up", + ) + if dec_rope_enable: + for i in range(dec_depths[s]): + dec.add( + Block( + channels=dec_channels[s], + num_heads=dec_num_head[s], + patch_size=dec_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=dec_drop_path_[i], + layer_scale=layer_scale, + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + rope_base=rope_base, + shift_coords=shift_coords, + jitter_coords=jitter_coords, + rescale_coords=rescale_coords, + ), + name=f"block{i}", + ) + else: + for i in range(dec_depths[s]): + dec.add( + Block( + channels=dec_channels[s], + num_heads=dec_num_head[s], + patch_size=dec_patch_size[s], + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=proj_drop, + drop_path=dec_drop_path_[i], + layer_scale=layer_scale, + norm_layer=ln_layer, + act_layer=act_layer, + pre_norm=pre_norm, + order_index=i % len(self.order), + cpe_indice_key=f"stage{s}", + enable_rpe=enable_rpe, + enable_flash=enable_flash, + upcast_attention=upcast_attention, + upcast_softmax=upcast_softmax, + rope_base=None, + ), + name=f"block{i}", + ) + self.dec.add(module=dec, name=f"dec{s}") + if self.freeze_encoder: + for p in self.embedding.parameters(): + p.requires_grad = False + for p in self.enc.parameters(): + p.requires_grad = False + self.apply(self._init_weights) + + @staticmethod + def _init_weights(module): + if isinstance(module, nn.Linear): + trunc_normal_(module.weight, std=0.02) + if module.bias is not None: + nn.init.zeros_(module.bias) + elif isinstance(module, spconv.SubMConv3d): + trunc_normal_(module.weight, std=0.02) + if module.bias is not None: + nn.init.zeros_(module.bias) + + def forward(self, data_dict): + point = Point(data_dict) + point = self.embedding(point) + + point.serialization(order=self.order, shuffle_orders=self.shuffle_orders) + point.sparsify() + + point = self.enc(point) + if not self.enc_mode: + point = self.dec(point) + return point diff --git a/pointcept/models/sgiformer/__init__.py b/pointcept/models/sgiformer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..c1ddffbc4eb3f565bd533aec1b7d45eee2564292 --- /dev/null +++ b/pointcept/models/sgiformer/__init__.py @@ -0,0 +1 @@ +from .sgiformer_v1m1_base import SGIFormer diff --git a/pointcept/models/sgiformer/loss.py b/pointcept/models/sgiformer/loss.py new file mode 100644 index 0000000000000000000000000000000000000000..3e3a6e5dc5e287211481fd4ecdaa5c906da4f95b --- /dev/null +++ b/pointcept/models/sgiformer/loss.py @@ -0,0 +1,434 @@ +import torch +import torch.nn as nn +import torch.nn.functional as F +from scipy.optimize import linear_sum_assignment +from torch.amp import autocast +from pointcept.utils.registry import Registry + +MATCHER = Registry("MATCHER") +COST = Registry("COST") + +LARGE_CONSTANT = 1e6 + + +@torch.jit.script +def batch_sigmoid_bce_loss(inputs, targets): + """Sigmoid BCE loss. + + Args: + inputs: of shape (n_queries, n_points). + targets: of shape (n_gts, n_points). + + Returns: + Tensor: Loss of shape (n_queries, n_gts). + """ + pos = F.binary_cross_entropy_with_logits( + inputs, torch.ones_like(inputs), reduction="none" + ) + neg = F.binary_cross_entropy_with_logits( + inputs, torch.zeros_like(inputs), reduction="none" + ) + + pos_loss = torch.einsum("nc,mc->nm", pos, targets) + neg_loss = torch.einsum("nc,mc->nm", neg, (1 - targets)) + return (pos_loss + neg_loss) / inputs.shape[1] + + +@torch.jit.script +def batch_dice_loss(inputs, targets): + """Dice loss. + + Args: + inputs: of shape (n_queries, n_points). + targets: of shape (n_gts, n_points). + + Returns: + Tensor: Loss of shape (n_queries, n_gts). + """ + inputs = inputs.sigmoid() + numerator = 2 * torch.einsum("nc,mc->nm", inputs, targets) + denominator = inputs.sum(-1)[:, None] + targets.sum(-1)[None, :] + loss = 1 - (numerator + 1) / (denominator + 1) + return loss + + +def get_iou(inputs, targets): + """IoU for to equal shape masks. + + Args: + inputs (Tensor): of shape (n_gts, n_points). + targets (Tensor): of shape (n_gts, n_points). + + Returns: + Tensor: IoU of shape (n_gts,). + """ + inputs = inputs.sigmoid() + binarized_inputs = (inputs >= 0.5).float() + targets = (targets > 0.5).float() + intersection = (binarized_inputs * targets).sum(-1) + union = targets.sum(-1) + binarized_inputs.sum(-1) - intersection + score = intersection / (union + 1e-6) + return score + + +def dice_loss(inputs, targets): + """Compute the DICE loss, similar to generalized IOU for masks. + + Args: + inputs (Tensor): A float tensor of arbitrary shape. + The predictions for each example. + targets (Tensor): A float tensor with the same shape as inputs. + Stores the binary classification label for each element in inputs + (0 for the negative class and 1 for the positive class). + + Returns: + Tensor: loss value. + """ + inputs = inputs.sigmoid() + numerator = 2 * (inputs * targets).sum(-1) + denominator = inputs.sum(-1) + targets.sum(-1) + loss = 1 - (numerator + 1) / (denominator + 1) + return loss.mean() + + +def sigmoid_focal_loss(inputs, targets, alpha: float = 0.25, gamma: float = 2): + """ + Loss used in RetinaNet for dense detection: https://arxiv.org/abs/1708.02002. + + Args: + inputs (torch.Tensor): A float tensor of arbitrary shape. + The predictions for each example. + targets (torch.Tensor): A float tensor with the same shape as inputs. Stores the binary + classification label for each element in inputs + (0 for the negative class and 1 for the positive class). + alpha (float, optional): Weighting factor in range (0, 1) to balance + positive vs negative examples. Default: 0.25. + gamma (float): Exponent of the modulating factor (1 - p_t) to + balance easy vs hard examples. Default: 2. + + Returns: + torch.Tensor: The computed sigmoid focal loss. + """ + prob = inputs.sigmoid() + ce_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction="none") + p_t = prob * targets + (1 - prob) * (1 - targets) + loss = ce_loss * ((1 - p_t) ** gamma) + + if alpha >= 0: + alpha_t = alpha * targets + (1 - alpha) * (1 - targets) + loss = alpha_t * loss + + return loss.mean() + + +class SGIFormerLoss(nn.Module): + def __init__( + self, + matcher, + loss_weight, + non_object_weight, + num_classes, + fix_dice_loss_weight, + iter_matcher, + fix_mean_loss=False, + semantic_ignore_index=-1, + loss_cls_type="ce_loss", + ): + super().__init__() + self.matcher = MATCHER.build(matcher) + class_weight = [1] * num_classes + [non_object_weight] + self.class_weight = class_weight + self.loss_weight = loss_weight + self.num_classes = num_classes + self.fix_dice_loss_weight = fix_dice_loss_weight + self.iter_matcher = iter_matcher + self.fix_mean_loss = fix_mean_loss + self.loss_cls_type = loss_cls_type + self.semantic_ignore_index = semantic_ignore_index + + def get_loss( + self, + pred_inst_info, + gt_inst_info, + matched_idx_list=None, + ): + if self.iter_matcher: + matched_idx_list = [] + pred_cls_list = pred_inst_info["cls_list"] + pred_mask_list = pred_inst_info["mask_list"] + pred_score_list = pred_inst_info["score_list"] + pred_inst_list = [] + for i in range(len(pred_cls_list)): + pred_inst_list.append(dict(cls=pred_cls_list[i], mask=pred_mask_list[i])) + + device = pred_mask_list[0].device + + cls_loss_list = [] + score_loss_list = [] + mask_bce_loss_list = [] + mask_dice_loss_list = [] + + for i in range(len(gt_inst_info)): + if len(gt_inst_info[i]["cls"]) == 0: + cls_loss_list.append( + torch.tensor(0.0, requires_grad=True, device=device) + ) + score_loss_list.append( + torch.tensor(0.0, requires_grad=True, device=device) + ) + mask_bce_loss_list.append( + torch.tensor(0.0, requires_grad=True, device=device) + ) + mask_dice_loss_list.append( + torch.tensor(0.0, requires_grad=True, device=device) + ) + continue + if self.iter_matcher: + pred_idx, gt_idx = self.matcher(pred_inst_list[i], gt_inst_info[i]) + matched_idx_list.append((pred_idx, gt_idx)) + else: + pred_idx, gt_idx = matched_idx_list[i] + + pred_cls = pred_cls_list[i] + pred_mask = pred_mask_list[i] + pred_score = pred_score_list[i] if pred_score_list is not None else None + gt_inst = gt_inst_info[i] + + # categorical loss + num_classes = pred_cls.shape[1] - 1 + # ignored index become the last class to predict + gt_cls = pred_cls.new_full((len(pred_cls),), num_classes, dtype=torch.long) + gt_cls[pred_idx] = gt_inst["cls"][gt_idx] + assert self.loss_cls_type == "ce_loss" + cls_loss_list.append( + F.cross_entropy( + pred_cls, gt_cls, pred_cls.new_tensor(self.class_weight) + ) + ) + + # 3 other losses + if gt_inst["mask"].shape[0] == 0: + continue + + pred_mask = pred_mask[pred_idx] + gt_mask = gt_inst["mask"][gt_idx] + mask_bce_loss_list.append( + F.binary_cross_entropy_with_logits(pred_mask, gt_mask.float()) + ) + mask_dice_loss_list.append(dice_loss(pred_mask, gt_mask.float())) + + # check if skip objectness loss + if pred_score is None: + continue + + with torch.no_grad(): + gt_score = get_iou(pred_mask, gt_mask).unsqueeze(1) + + filter_id, _ = torch.where(gt_score > 0.5) + if filter_id.numel(): + gt_score = gt_score[filter_id] + pred_score = pred_score[filter_id] + score_loss_list.append(F.mse_loss(pred_score, gt_score)) + + # process loss lists + cls_loss = torch.mean(torch.stack(cls_loss_list)) + + if len(score_loss_list): + score_loss = torch.stack(score_loss_list).sum() / len(pred_mask_list) + else: + score_loss = torch.tensor(0.0, requires_grad=True, device=device) + + if len(mask_bce_loss_list): + mask_bce_loss = torch.stack(mask_bce_loss_list).sum() / len(pred_mask_list) + mask_dice_loss = torch.stack(mask_dice_loss_list).sum() + + if self.fix_dice_loss_weight: + mask_dice_loss = mask_dice_loss / len(pred_mask_list) * 4 + + if self.fix_mean_loss: + mask_bce_loss = ( + mask_bce_loss * len(pred_mask_list) / len(mask_bce_loss_list) + ) + mask_dice_loss = ( + mask_dice_loss * len(pred_mask_list) / len(mask_dice_loss_list) + ) + else: + mask_bce_loss = torch.tensor(0.0, requires_grad=True, device=device) + mask_dice_loss = torch.tensor(0.0, requires_grad=True, device=device) + + loss = ( + self.loss_weight[0] * cls_loss + + self.loss_weight[1] * mask_bce_loss + + self.loss_weight[2] * mask_dice_loss + + self.loss_weight[3] * score_loss + ) + loss_dict = dict() + loss_dict["loss_cls"] = cls_loss + loss_dict["loss_mask"] = mask_bce_loss + loss_dict["loss_dice"] = mask_dice_loss + loss_dict["loss_score"] = score_loss + return loss, loss_dict, matched_idx_list + + @staticmethod + def loss_bias(pred_bias, gt_bias, gt_mask): + bias_dist = torch.sum(torch.abs(pred_bias - gt_bias), dim=-1) + bias_l1_loss = torch.sum(bias_dist * gt_mask) / (torch.sum(gt_mask) + 1e-8) + + loss_bias = bias_l1_loss + return loss_bias + + def forward(self, pred, target): + """Loss main function.""" + gt_inst_info = target["inst_info"] + gt_point_info = target["point_info"] + + pred_inst_info = dict( + cls_list=pred["cls_list"], + mask_list=pred["mask_list"], + score_list=pred["score_list"], + ) + loss, loss_dict, matched_idx_list = self.get_loss(pred_inst_info, gt_inst_info) + if "aux_pred_list" in pred: + if self.iter_matcher: + matched_idx_list = None + for pred_inst_info_ in pred["aux_pred_list"]: + aux_loss, _, _ = self.get_loss( + pred_inst_info_, gt_inst_info, matched_idx_list + ) + loss = loss + aux_loss + + if "seg_logits" in pred: + if pred["seg_logits"] is not None: + if gt_point_info["segment"].max() >= 0: + loss_seg = F.cross_entropy( + pred["seg_logits"], + gt_point_info["segment"], + pred["seg_logits"].new_tensor(self.class_weight), + ignore_index=self.semantic_ignore_index, + ) + else: + loss_seg = torch.tensor(0.0, requires_grad=True, device=loss.device) + + loss = loss + self.loss_weight[4] * loss_seg + else: + loss_seg = torch.tensor(0.0, requires_grad=True, device=loss.device) + loss_dict["loss_seg"] = loss_seg + if "bias" in pred: + if pred["bias"] is not None: + loss_bias = self.loss_bias( + pred["bias"], + gt_point_info["bias"], + gt_point_info["mask"], + ) + loss += self.loss_weight[5] * loss_bias + else: + loss_bias = torch.tensor(0.0, device=loss.device) + loss_dict["loss_bias"] = loss_bias + loss_dict["loss"] = loss + return loss_dict + + +@COST.register_module() +class QueryClassificationCost: + """Classification cost for queries. + + Args: + weigth (float): Weight of the cost. + """ + + def __init__(self, weight): + self.weight = weight + + def __call__(self, pred_inst, gt_inst, **kwargs): + """Compute match cost.""" + score = pred_inst["cls"].softmax(-1) + cost = -score[:, gt_inst["cls"]] + return cost * self.weight + + +@COST.register_module() +class MaskBCECost: + """Sigmoid BCE cost for mask. + + Args: + weigth (float): Weight of the cost. + """ + + def __init__(self, weight): + self.weight = weight + + def __call__(self, pred_inst, gt_inst, **kwargs): + """Compute match cost.""" + with autocast("cuda", enabled=False): + cost = batch_sigmoid_bce_loss( + pred_inst["mask"].float(), gt_inst["mask"].float() + ) + return cost * self.weight + + +@COST.register_module() +class MaskDiceCost: + """Dice cost for mask. + + Args: + weigth (float): Weight of the cost. + """ + + def __init__(self, weight): + self.weight = weight + + def __call__(self, pred_inst, gt_inst, **kwargs): + """Compute match cost.""" + with autocast("cuda", enabled=False): + cost = batch_dice_loss(pred_inst["mask"].float(), gt_inst["mask"].float()) + return cost * self.weight + + +@MATCHER.register_module() +class HungarianMatcher: + """Hungarian matcher. + + Args: + costs (List[ConfigDict]): Cost functions. + """ + + def __init__(self, costs): + self.costs = [] + for cost in costs: + self.costs.append(COST.build(cost)) + + @torch.no_grad() + def __call__(self, pred_inst, gt_inst, **kwargs): + """Compute match cost. + + Returns: + Tuple: + - Tensor: Query ids of shape (n_matched,), + - Tensor: Object ids of shape (n_matched,). + """ + inst_cls = gt_inst["cls"] + inst_num = len(inst_cls) + if inst_num == 0: + return inst_cls.new_empty((0,), dtype=torch.int64), inst_cls.new_empty( + (0,), dtype=torch.int64 + ) + + cost_values = [] + for cost in self.costs: + cost_values.append(cost(pred_inst, gt_inst)) + cost_value = torch.stack(cost_values).sum(dim=0) + + if torch.isnan(cost_value).any() or torch.isinf(cost_value).any(): + print( + "Warning: Detected NaN or Inf in cost_value. Replacing with large values." + ) + cost_value = torch.where( + torch.isnan(cost_value) | torch.isinf(cost_value), + torch.full_like(cost_value, LARGE_CONSTANT), + cost_value, + ) + query_ids, object_ids = linear_sum_assignment(cost_value.cpu().numpy()) + + return inst_cls.new_tensor(query_ids, dtype=torch.int64), inst_cls.new_tensor( + object_ids, dtype=torch.int64 + ) diff --git a/pointcept/models/sgiformer/nms.py b/pointcept/models/sgiformer/nms.py new file mode 100644 index 0000000000000000000000000000000000000000..c5f5e0401b50027ea71f273d534f12c1eb3d729f --- /dev/null +++ b/pointcept/models/sgiformer/nms.py @@ -0,0 +1,129 @@ +# Copyright (c) OpenMMLab. All rights reserved. +import torch + + +def mask_matrix_nms( + masks, + labels, + scores, + filter_thr=-1, + nms_pre=-1, + max_num=-1, + kernel="gaussian", + sigma=2.0, + mask_area=None, +): + """Matrix NMS for multi-class masks. + + Args: + masks (Tensor): Has shape (num_instances, m) + labels (Tensor): Labels of corresponding masks, + has shape (num_instances,). + scores (Tensor): Mask scores of corresponding masks, + has shape (num_instances). + filter_thr (float): Score threshold to filter the masks + after matrix nms. Default: -1, which means do not + use filter_thr. + nms_pre (int): The max number of instances to do the matrix nms. + Default: -1, which means do not use nms_pre. + max_num (int, optional): If there are more than max_num masks after + matrix, only top max_num will be kept. Default: -1, which means + do not use max_num. + kernel (str): 'linear' or 'gaussian'. + sigma (float): std in gaussian method. + mask_area (Tensor): The sum of seg_masks. + + Returns: + tuple(Tensor): Processed mask results. + + - scores (Tensor): Updated scores, has shape (n,). + - labels (Tensor): Remained labels, has shape (n,). + - masks (Tensor): Remained masks, has shape (n, m). + - keep_inds (Tensor): The indices number of + the remaining mask in the input mask, has shape (n,). + """ + assert len(labels) == len(masks) == len(scores) + if len(labels) == 0: + return ( + scores.new_zeros(0), + labels.new_zeros(0), + masks.new_zeros(0, *masks.shape[-1:]), + labels.new_zeros(0), + ) + if mask_area is None: + mask_area = masks.sum(1).float() + else: + assert len(masks) == len(mask_area) + + # sort and keep top nms_pre + scores, sort_inds = torch.sort(scores, descending=True) + + keep_inds = sort_inds + if nms_pre > 0 and len(sort_inds) > nms_pre: + sort_inds = sort_inds[:nms_pre] + keep_inds = keep_inds[:nms_pre] + scores = scores[:nms_pre] + masks = masks[sort_inds] + mask_area = mask_area[sort_inds] + labels = labels[sort_inds] + + num_masks = len(labels) + flatten_masks = masks.reshape(num_masks, -1).float() + # inter. + inter_matrix = torch.mm(flatten_masks, flatten_masks.transpose(1, 0)) + expanded_mask_area = mask_area.expand(num_masks, num_masks) + # Upper triangle iou matrix. + iou_matrix = ( + inter_matrix + / (expanded_mask_area + expanded_mask_area.transpose(1, 0) - inter_matrix) + ).triu(diagonal=1) + # label_specific matrix. + expanded_labels = labels.expand(num_masks, num_masks) + # Upper triangle label matrix. + label_matrix = (expanded_labels == expanded_labels.transpose(1, 0)).triu(diagonal=1) + + # IoU compensation + compensate_iou, _ = (iou_matrix * label_matrix).max(0) + compensate_iou = compensate_iou.expand(num_masks, num_masks).transpose(1, 0) + + # IoU decay + decay_iou = iou_matrix * label_matrix + + # Calculate the decay_coefficient + if kernel == "gaussian": + decay_matrix = torch.exp(-1 * sigma * (decay_iou**2)) + compensate_matrix = torch.exp(-1 * sigma * (compensate_iou**2)) + decay_coefficient, _ = (decay_matrix / compensate_matrix).min(0) + elif kernel == "linear": + decay_matrix = (1 - decay_iou) / (1 - compensate_iou) + decay_coefficient, _ = decay_matrix.min(0) + else: + raise NotImplementedError(f"{kernel} kernel is not supported in matrix nms!") + # update the score. + scores = scores * decay_coefficient + + if filter_thr > 0: + keep = scores >= filter_thr + keep_inds = keep_inds[keep] + if not keep.any(): + return ( + scores.new_zeros(0), + labels.new_zeros(0), + masks.new_zeros(0, *masks.shape[-1:]), + labels.new_zeros(0), + ) + masks = masks[keep] + scores = scores[keep] + labels = labels[keep] + + # sort and keep top max_num + scores, sort_inds = torch.sort(scores, descending=True) + keep_inds = keep_inds[sort_inds] + if max_num > 0 and len(sort_inds) > max_num: + sort_inds = sort_inds[:max_num] + keep_inds = keep_inds[:max_num] + scores = scores[:max_num] + masks = masks[sort_inds] + labels = labels[sort_inds] + + return scores, labels, masks, keep_inds diff --git a/pointcept/models/sgiformer/sgiformer_v1m1_base.py b/pointcept/models/sgiformer/sgiformer_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..14756b3fc21ea930f5fea46105d9d9535a90241f --- /dev/null +++ b/pointcept/models/sgiformer/sgiformer_v1m1_base.py @@ -0,0 +1,686 @@ +""" +SGIFormer +Xiaoyang slightly modified from: https://github.com/RayYoh/SGIFormer +1. Better batch-wise operation +2. Compatibility to Pointcept backbones +3. No concept of voxel (voxel fundamentally are grid sampled point) + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +import torch.nn as nn +import torch.nn.functional as F +from functools import partial +import torch_scatter +from pointcept.models.utils.structure import Point +from pointcept.models.utils.misc import offset2bincount, batch2offset +from pointcept.models.builder import MODELS, build_model + +from .loss import SGIFormerLoss +from .nms import mask_matrix_nms + + +### Decoder ### +class PositionEmbeddingCoordsSine(nn.Module): + def __init__( + self, + temperature=10000, + normalize=False, + scale=None, + pos_type="fourier", + d_pos=None, + d_in=3, + gauss_scale=1.0, + ): + super().__init__() + self.d_pos = d_pos + self.temperature = temperature + self.normalize = normalize + if scale is not None and normalize is False: + raise ValueError("normalize should be True if scale is passed") + if scale is None: + scale = 2 * torch.pi + assert pos_type in ["sine", "fourier"] + self.pos_type = pos_type + self.scale = scale + if pos_type == "fourier": + assert d_pos is not None + assert d_pos % 2 == 0 + # define a gaussian matrix input_ch -> output_ch + B = torch.empty((d_in, d_pos // 2)).normal_() + B *= gauss_scale + self.register_buffer("gauss_B", B) + self.d_pos = d_pos + + @staticmethod + def shift_scale_points(pred_xyz, src_range, dst_range=None): + """ + pred_xyz: B x N x 3 + src_range: [[B x 3], [B x 3]] - min and max XYZ coords + dst_range: [[B x 3], [B x 3]] - min and max XYZ coords + """ + if dst_range is None: + dst_range = [ + torch.zeros((src_range[0].shape[0], 3), device=src_range[0].device), + torch.ones((src_range[0].shape[0], 3), device=src_range[0].device), + ] + + if pred_xyz.ndim == 4: + src_range = [x[:, None] for x in src_range] + dst_range = [x[:, None] for x in dst_range] + + assert src_range[0].shape[0] == pred_xyz.shape[0] + assert dst_range[0].shape[0] == pred_xyz.shape[0] + assert src_range[0].shape[-1] == pred_xyz.shape[-1] + assert src_range[0].shape == src_range[1].shape + assert dst_range[0].shape == dst_range[1].shape + assert src_range[0].shape == dst_range[1].shape + + src_diff = src_range[1][:, None, :] - src_range[0][:, None, :] + dst_diff = dst_range[1][:, None, :] - dst_range[0][:, None, :] + prop_xyz = ( + ((pred_xyz - src_range[0][:, None, :]) * dst_diff) / src_diff + ) + dst_range[0][:, None, :] + return prop_xyz + + def get_sine_embeddings(self, xyz, num_channels, input_range): + num_channels = self.d_pos + # clone coords so that shift/scale operations do not affect original tensor + orig_xyz = xyz + xyz = orig_xyz.clone() + + ncoords = xyz.shape[1] + if self.normalize: + xyz = self.shift_scale_points(xyz, src_range=input_range) + + ndim = num_channels // xyz.shape[2] + if ndim % 2 != 0: + ndim -= 1 + # automatically handle remainder by assiging it to the first dim + rems = num_channels - (ndim * xyz.shape[2]) + + assert ( + ndim % 2 == 0 + ), f"Cannot handle odd sized ndim={ndim} where num_channels={num_channels} and xyz={xyz.shape}" + + final_embeds = [] + prev_dim = 0 + + for d in range(xyz.shape[2]): + cdim = ndim + if rems > 0: + # add remainder in increments of two to maintain even size + cdim += 2 + rems -= 2 + + if cdim != prev_dim: + dim_t = torch.arange(cdim, dtype=torch.float32, device=xyz.device) + dim_t = self.temperature ** (2 * (dim_t // 2) / cdim) + + # create batch x cdim x nccords embedding + raw_pos = xyz[:, :, d] + if self.scale: + raw_pos *= self.scale + pos = raw_pos[:, :, None] / dim_t + pos = torch.stack( + (pos[:, :, 0::2].sin(), pos[:, :, 1::2].cos()), dim=3 + ).flatten(2) + final_embeds.append(pos) + prev_dim = cdim + + final_embeds = torch.cat(final_embeds, dim=2) # .permute(0, 2, 1) + return final_embeds + + def get_fourier_embeddings(self, xyz, num_channels=None, input_range=None): + # Follows - https://people.eecs.berkeley.edu/~bmild/fourfeat/index.html + + if num_channels is None: + num_channels = self.gauss_B.shape[1] * 2 + + bsize, npoints = xyz.shape[0], xyz.shape[1] + assert num_channels > 0 and num_channels % 2 == 0 + d_in, max_d_out = self.gauss_B.shape[0], self.gauss_B.shape[1] + d_out = num_channels // 2 + assert d_out <= max_d_out + assert d_in == xyz.shape[-1] + + # clone coords so that shift/scale operations do not affect original tensor + orig_xyz = xyz + xyz = orig_xyz.clone() + + if self.normalize: + xyz = self.shift_scale_points(xyz, src_range=input_range) + + xyz *= 2 * torch.pi + xyz_proj = torch.mm(xyz.view(-1, d_in), self.gauss_B[:, :d_out]).view( + bsize, npoints, d_out + ) + final_embeds = [xyz_proj.sin(), xyz_proj.cos()] + + # return batch x d_pos x npoints embedding + final_embeds = torch.cat(final_embeds, dim=2) # .permute(0, 2, 1) + return final_embeds + + def forward(self, xyz, num_channels=None, input_range=None): + assert isinstance(xyz, torch.Tensor) + assert xyz.ndim == 3 + # xyz is batch x npoints x 3 + if self.pos_type == "sine": + with torch.no_grad(): + out = self.get_sine_embeddings(xyz, num_channels, input_range) + elif self.pos_type == "fourier": + with torch.no_grad(): + out = self.get_fourier_embeddings(xyz, num_channels, input_range) + else: + raise ValueError(f"Unknown {self.pos_type}") + + return out + + def extra_repr(self): + st = f"type={self.pos_type}, scale={self.scale}, normalize={self.normalize}" + if hasattr(self, "gauss_B"): + st += ( + f", gaussB={self.gauss_B.shape}, gaussBsum={self.gauss_B.sum().item()}" + ) + return st + + +class CrossAttentionLayer(nn.Module): + def __init__(self, d_model=256, nhead=8, dropout=0.0): + super().__init__() + self.attn = nn.MultiheadAttention( + d_model, nhead, dropout=dropout, batch_first=True + ) + self.norm = nn.LayerNorm(d_model) + self.dropout = nn.Dropout(dropout) + self._reset_parameters() + + def _reset_parameters(self): + for p in self.parameters(): + if p.dim() > 1: + nn.init.xavier_uniform_(p) + + def with_pos_embed(self, tensor, pos): + return tensor if pos is None else tensor + pos + + def forward(self, source, query, attn_masks=None, pe=None, query_pe=None): + """ + query Tensor (b, n_q, d_model) + """ + outputs = [] + for i in range(len(source)): + q_pos = query_pe[i] if query_pe is not None else None + pos = pe[i] if pe is not None else None + q = self.with_pos_embed(query[i], q_pos) + k = self.with_pos_embed(source[i], pos) + v = source[i] + attn_mask = attn_masks[i] if attn_masks else None + output, _ = self.attn(q, k, v, attn_mask=attn_mask) + output = self.dropout(output) + query[i] + output = self.norm(output) + outputs.append(output) + return outputs + + +class SelfAttentionLayer(nn.Module): + def __init__(self, d_model=256, nhead=8, dropout=0.0): + super().__init__() + self.attn = nn.MultiheadAttention( + d_model, nhead, dropout=dropout, batch_first=True + ) + self.norm = nn.LayerNorm(d_model) + self.dropout = nn.Dropout(dropout) + + def with_pos_embed(self, tensor, pos): + return tensor if pos is None else tensor + pos + + def forward(self, x, pe=None): + outputs = [] + for i in range(len(x)): + pos = pe[i] if pe is not None else None + q = k = self.with_pos_embed(x[i], pos) + output, _ = self.attn(q, k, x[i]) + output = self.dropout(output) + x[i] + output = self.norm(output) + outputs.append(output) + return outputs + + +class FFN(nn.Module): + def __init__(self, d_model, hidden_dim, dropout=0.0, activation_fn="relu"): + super().__init__() + self.net = nn.Sequential( + nn.Linear(d_model, hidden_dim), + nn.ReLU() if activation_fn == "relu" else nn.GELU(), + nn.Dropout(dropout), + nn.Linear(hidden_dim, d_model), + nn.Dropout(dropout), + ) + self.norm = nn.LayerNorm(d_model) + + def forward(self, x): + outputs = [] + for i in range(len(x)): + output = self.net(x[i]) + output = output + x[i] + output = self.norm(output) + outputs.append(output) + return outputs + + +class SGIFormerDecoder(nn.Module): + def __init__( + self, + dec_num_layer=3, + num_sample_query=200, + num_learn_query=200, + num_classes=18, + in_channel=32, + d_model=256, + nhead=8, + hidden_dim=1024, + dropout=0.0, + activation_fn="relu", + attn_mask=True, + use_score=False, + alpha=0.4, + ): + super().__init__() + norm_fn = partial(nn.BatchNorm1d, eps=1e-3, momentum=0.01) + self.use_score = use_score + self.dec_num_layer = dec_num_layer + self.num_classes = num_classes + self.d_model = d_model + self.attn_mask = attn_mask + self.alpha = alpha + + self.seg_head = nn.Sequential( + nn.Linear(in_channel, in_channel), + norm_fn(in_channel), + nn.ReLU(), + nn.Linear(in_channel, num_classes + 1), + ) + self.bias_head = nn.Sequential( + nn.Linear(in_channel, in_channel), + norm_fn(in_channel), + nn.ReLU(), + nn.Linear(in_channel, 3), + ) + + self.feat_proj = nn.Sequential( + nn.Linear(in_channel, d_model), nn.LayerNorm(d_model), nn.ReLU() + ) + self.rep_layer = nn.Sequential( + nn.Linear(d_model, num_sample_query), + nn.LayerNorm(num_sample_query), + nn.ReLU(), + ) + self.query_learn = nn.Embedding(num_learn_query, d_model) + + self.sp_feat_proj = nn.Sequential( + nn.Linear(in_channel, d_model), nn.LayerNorm(d_model), nn.ReLU() + ) + self.x_mask = nn.Sequential(nn.Linear(d_model, d_model), nn.ReLU()) + + self.sp_pos = PositionEmbeddingCoordsSine( + pos_type="fourier", + d_pos=d_model, + normalize=True, + ) + self.feat_query_attn_layers = nn.ModuleList([]) + self.feat_self_attn_layers = nn.ModuleList([]) + self.cross_attn_layers = nn.ModuleList([]) + self.self_attn_layers = nn.ModuleList([]) + self.ffn_layers = nn.ModuleList([]) + for i in range(self.dec_num_layer): + self.cross_attn_layers.append(CrossAttentionLayer(d_model, nhead, dropout)) + self.self_attn_layers.append(SelfAttentionLayer(d_model, nhead, dropout)) + self.ffn_layers.append(FFN(d_model, hidden_dim, dropout, activation_fn)) + + if i < self.dec_num_layer - 1: + self.feat_query_attn_layers.append( + CrossAttentionLayer(d_model, nhead, dropout) + ) + self.feat_self_attn_layers.append( + SelfAttentionLayer(d_model, nhead, dropout) + ) + + self.out_norm = nn.LayerNorm(d_model) + self.out_cls = nn.Sequential( + nn.Linear(d_model, d_model), nn.ReLU(), nn.Linear(d_model, num_classes + 1) + ) + if self.use_score: + self.out_score = nn.Sequential( + nn.Linear(d_model, d_model), nn.ReLU(), nn.Linear(d_model, 1) + ) + + def forward_head(self, query_list, sp_mask_feat_list): + pred_cls_list = [] + pred_mask_list = [] + attn_mask_list = [] + pred_score_list = [] if self.use_score else None + + for i in range(len(query_list)): + norm_query_ = self.out_norm(query_list[i]) + sp_mask_feat_ = sp_mask_feat_list[i] + pred_cls_list.append(self.out_cls(norm_query_)) + if self.use_score: + pred_score_list.append(self.out_score(norm_query_)) + pred_mask_ = torch.einsum("nd, md->nm", norm_query_, sp_mask_feat_) + if self.attn_mask: + attn_mask_ = (pred_mask_.sigmoid() < 0.5).bool() + attn_mask_[torch.where(attn_mask_.sum(-1) == attn_mask_.shape[-1])] = ( + False + ) + attn_mask_ = attn_mask_.detach() + attn_mask_list.append(attn_mask_) + pred_mask_list.append(pred_mask_) + attn_mask_list = attn_mask_list if self.attn_mask else None + return pred_cls_list, pred_score_list, pred_mask_list, attn_mask_list + + def forward(self, point): + seg_logits = self.seg_head(point.feat) + bias = self.bias_head(point.feat) + point.bincount = offset2bincount(point.offset) + point.sp_bincount = offset2bincount(point.sp_offset) + + # get query + score = seg_logits.softmax(dim=-1)[:, :-1] + sp_coord = torch_scatter.scatter( + point.coord + bias, + point.sp_inverse, + dim=0, + reduce="mean", + ) + + feat_proj = self.feat_proj(point.feat) + + query_list = [] + score_list = score.split(point.bincount.tolist()) + feat_proj_list = feat_proj.split(point.bincount.tolist()) + + for i in range(len(point.offset)): + score_ = score_list[i] + max_score_, _ = score_.max(dim=-1) + _, topk_idx = max_score_.topk( + int(self.alpha * score_.shape[0]), sorted=False + ) + top_proj_feat_ = feat_proj_list[i][topk_idx, :] + rep_ = self.rep_layer(top_proj_feat_) + act_ = torch.softmax(rep_, dim=0) + query_ = act_.T @ top_proj_feat_ + query_ = torch.cat((query_, self.query_learn.weight), dim=0) + query_list.append(query_) + + # get pos + sp_pos_list = [] + sp_coord_list = sp_coord.split(point.sp_bincount.tolist()) + for sp_coord_ in sp_coord_list: + p_min, p_max = sp_coord_.min(0)[0], sp_coord_.max(0)[0] + pos_emb = self.sp_pos( + sp_coord_.unsqueeze(0), + num_channels=self.d_model, + input_range=(p_min.unsqueeze(0), p_max.unsqueeze(0)), + )[0] + sp_pos_list.append(pos_emb) + sp_feat = self.sp_feat_proj(point.sp_feat) + sp_feat_list = sp_feat.split(point.sp_bincount.tolist()) + sp_mask_feat = self.x_mask(sp_feat) + sp_mask_feat_list = sp_mask_feat.split(point.sp_bincount.tolist()) + + # decoding + aux_pred_list = [self.forward_head(query_list, sp_mask_feat_list)] + attn_mask_list = aux_pred_list[-1][-1] + + for i in range(self.dec_num_layer): + source_list = [ + sp_feat_ + sp_pos_ + for sp_feat_, sp_pos_ in zip(sp_feat_list, sp_pos_list) + ] + # Xiaoyang Note - TODO: Batch-wised Flash Attention with causal mask + query_list = self.cross_attn_layers[i]( + source_list, query_list, attn_mask_list + ) + query_list = self.self_attn_layers[i](query_list) + query_list = self.ffn_layers[i](query_list) + + if i < self.dec_num_layer - 1: + sp_feat_list = self.feat_query_attn_layers[i]( + query_list, + sp_feat_list, + query_pe=sp_pos_list, + ) + sp_feat_list = self.feat_self_attn_layers[i]( + sp_feat_list, + sp_pos_list, + ) + + aux_pred_list.append(self.forward_head(query_list, sp_mask_feat_list)) + attn_mask_list = aux_pred_list[-1][-1] + pred_cls_list, pred_score_list, pred_mask_list, _ = aux_pred_list.pop(-1) + + return { + "cls_list": pred_cls_list, + "score_list": pred_score_list, + "mask_list": pred_mask_list, + "aux_pred_list": [ + { + "cls_list": pred_cls_list_, + "score_list": pred_score_list_, + "mask_list": pred_mask_list_, + } + for pred_cls_list_, pred_score_list_, pred_mask_list_, _ in aux_pred_list + ], + "seg_logits": seg_logits, + "bias": bias, + } + + +### SGIFormer ### +@MODELS.register_module("SGIFormer-v1m1") +class SGIFormer(nn.Module): + def __init__( + self, + backbone, + decoder=None, + criteria=None, + topk_insts=200, + score_thr=0.0, + npoint_thr=100, + sp_score_thr=0.55, + nms=True, + semantic_num_classes=20, + semantic_ignore_index=-1, + segment_ignore_index=(-1, 0, 1), + instance_ignore_index=-1, + ): + super().__init__() + + self.backbone = build_model(backbone) + self.decoder = SGIFormerDecoder(**decoder) + self.criteria = SGIFormerLoss(**criteria) + + self.topk_insts = topk_insts + self.score_thr = score_thr + self.npoint_thr = npoint_thr + self.sp_score_thr = sp_score_thr + self.nms = nms + + self.semantic_num_classes = semantic_num_classes + self.semantic_ignore_index = semantic_ignore_index + self.segment_ignore_index = segment_ignore_index + self.instance_ignore_index = instance_ignore_index + + @torch.no_grad() + def prepare_target(self, point): + target = dict() + # only predict instance classes + segment = point.segment.clone() + segment_ignore_index = torch.tensor( + self.segment_ignore_index, device=point.segment.device + ) + segment[torch.isin(point.segment, segment_ignore_index)] = ( + self.semantic_ignore_index + ) + for cls in sorted(self.segment_ignore_index, reverse=True): + if cls == self.semantic_ignore_index: + continue + segment[segment >= cls] -= 1 + target["point_info"] = dict( + segment=segment, + coord=point.coord, + bias=point.instance_centroid - point.coord, + mask=point.instance != self.instance_ignore_index, + ) + + target["inst_info"] = [] + bincount = offset2bincount(point.offset) + instance_list = point.instance.split(bincount.tolist()) + segment_list = segment.split(bincount.tolist()) + sp_inverse_list = point.sp_inverse.split(bincount.tolist()) + sp_inverse_list = [ + sp_inverse - sp_inverse.min() for sp_inverse in sp_inverse_list + ] + for instance_, segment_, sp_inverse_ in zip( + instance_list, segment_list, sp_inverse_list + ): + if instance_.max() >= 0: + inst_onehot_ = torch.zeros( + (instance_.shape[0], instance_.max() + 1), + device=instance_.device, + dtype=instance_.dtype, + ) + inst_mask_ = instance_ != self.instance_ignore_index + inst_onehot_[inst_mask_] = torch.nn.functional.one_hot( + instance_[inst_mask_], + ) + inst_sp_mask_ = torch_scatter.scatter( + inst_onehot_, + sp_inverse_, + reduce="mean", + dim=0, + ).T + inst_sp_mask_ = inst_sp_mask_ > 0.5 + inst_cls_ = torch_scatter.scatter( + segment_, + instance_, + reduce="max", + dim=0, + ) + else: + inst_sp_mask_ = torch.zeros( + (0, sp_inverse_.max()), + device=instance_.device, + dtype=torch.bool, + ) + inst_cls_ = torch.tensor([], device=instance_.device, dtype=torch.int) + target["inst_info"].append( + dict( + cls=inst_cls_, + mask=inst_sp_mask_, + ) + ) + return target + + def forward(self, data_dict, return_point=False): + if return_point: + return dict(point=self.backbone(data_dict)) + + point = self.backbone(data_dict) + assert isinstance(point, Point) + while "pooling_parent" in point.keys(): + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + # Xiaoyang Note: here we use same technology as PTv3 serialization + # to fused batched superpoint feature + sp, cluster = torch.unique( + point.batch << 48 | point.superpoint, + return_inverse=True, + ) + point["sp_feat"] = torch_scatter.scatter( + point.feat, cluster, dim=0, reduce="mean" + ) + point["sp_batch"] = torch_scatter.scatter( + point.batch, cluster, dim=0, reduce="max" + ) + point["sp_offset"] = batch2offset(point.sp_batch) + point["sp_inverse"] = cluster + pred = self.decoder(point) + if "segment" in data_dict.keys() and "instance" in data_dict.keys(): + target = self.prepare_target(point) + return_dict = self.criteria(pred, target) + else: + return_dict = dict() + + if not self.training: + # assume bs=1 for inference + assert len(pred["cls_list"]) == 1 + pred_cls = pred["cls_list"][0] + pred_mask = pred["mask_list"][0] + + pred_score = F.softmax(pred_cls, dim=-1)[:, :-1] + + if pred["score_list"] is not None: + pred_score *= pred["score_list"][0] + pred_classes = ( + torch.arange(self.semantic_num_classes, device=pred_score.device) + .unsqueeze(0) + .repeat(len(pred_cls), 1) + .flatten(0, 1) + ) + pred_score, topk_idx = pred_score.flatten(0, 1).topk( + self.topk_insts, sorted=False + ) + pred_classes = pred_classes[topk_idx] + + topk_idx = torch.div( + topk_idx, self.semantic_num_classes, rounding_mode="floor" + ) + pred_mask = pred_mask[topk_idx] + pred_mask_sigmoid = pred_mask.sigmoid() + + mask_scores = (pred_mask_sigmoid * (pred_mask > 0)).sum(1) / ( + (pred_mask > 0).sum(1) + 1e-6 + ) + pred_score = pred_score * mask_scores + + if self.nms: + pred_score, pred_classes, pred_mask_sigmoid, _ = mask_matrix_nms( + pred_mask_sigmoid, pred_classes, pred_score, kernel="linear" + ) + + pred_mask_sigmoid = pred_mask_sigmoid[:, point.sp_inverse] + pred_mask = pred_mask_sigmoid > self.sp_score_thr + + # score_thr + score_mask = pred_score > self.score_thr + pred_score = pred_score[score_mask] + pred_classes = pred_classes[score_mask] + pred_mask = pred_mask[score_mask] + + # npoint thr + npoint_mask = pred_mask.sum(1) > self.npoint_thr + pred_score = pred_score[npoint_mask] + pred_classes = pred_classes[npoint_mask] + pred_mask = pred_mask[npoint_mask] + + pred_mask = pred_mask.cpu().detach().numpy() + pred_classes = pred_classes.cpu().detach().numpy() + + sort_score, sort_index = pred_score.sort(descending=True) + sort_index = sort_index.cpu().detach().numpy() + sort_score = sort_score.cpu().detach().numpy() + + sort_classes = pred_classes[sort_index] + sorted_mask = pred_mask[sort_index] + + return_dict["pred_scores"] = sort_score + return_dict["pred_masks"] = sorted_mask + return_dict["pred_classes"] = sort_classes + + return return_dict diff --git a/pointcept/models/sonata/README.md b/pointcept/models/sonata/README.md new file mode 100644 index 0000000000000000000000000000000000000000..6a8b9835578e78be3b5fd80d22c290472b7a884e --- /dev/null +++ b/pointcept/models/sonata/README.md @@ -0,0 +1,151 @@ +# Sonata +This repo is the official training codebase of the paper **_Sonata: Self-Supervised Learning of Reliable Point Representations_**. Sonata is a powerful self-supervised learning framework (as well as our pre-trained model) that shows extraordinary parameter efficiency, data efficiency, and fine-tuning accuracy. + +*We recommend beginning with our inference [demo](https://github.com/facebookresearch/sonata) and it is also a tiny library for people who want to integrate pre-trained Sonata into their project. This section is designed for users interested in reproduce our pre-training.* + +[ **Pretrain** ] [ **Sonata** ] - [ [Homepage](https://xywu.me/sonata/) ] [ [Paper](https://arxiv.org/abs/2503.16429) ] [ [Inference Demo](https://github.com/facebookresearch/sonata) ] [ [Bib](#citation) ] + +
+teaser +
+ +## Quick start + +#### 1. Installation +Follow the instruction [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#installation). + +#### 2. Dataset +Downloading and preprocessing the following datasets: +- ScanNet v2 - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#scannet-v2) ] +- ScanNet++ v2 - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#scannet) ] +- S3DIS - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#s3dis) ] +- ArkitScenes - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#arkitscenes) ] +- HM3D - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#habitat---matterport-3d-hm3d) ] +- Structured3D - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#structured3d) ] + +The whole list of our supported datasets can be found [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#data-preparation) and some of the preprocessed datasets can be found [here](https://huggingface.co/Pointcept). + + +#### 3. Pre-training +Enable pre-training with 32 GPUs (minimal 16 GPUs) by running the following script: +```bash +# default config +sh scripts/train.sh -m 4 -g 8 -d sonata -c pretrain-sonata-v1m1-0-base -n pretrain-sonata-v1m1-0-base + +# (or) a variant seem like have a better performance +sh scripts/train.sh -m 4 -g 8 -d sonata -c pretrain-sonata-v1m2-0-uni-teacher-head -n pretrain-sonata-v1m2-0-uni-teacher-head + +# (or) if only have 16 gpus +sh scripts/train.sh -m 2 -g 8 -d sonata -c pretrain-sonata-v1m1-0-base -n pretrain-sonata-v1m1-0-base + +# (or) if OOM +# half the batch size and half the base learning rate +``` + +#### 4. Probing and Tuning +After pre-training, you can probe or tune the model with the following script: +(Our pre-trained model weight can be downloaded [here](https://huggingface.co/facebook/sonata/blob/main/pretrain-sonata-v1m1-0-base.pth)) +```bash +# Assume the pre-trained experiment is record in: +# exp/sonata/pretrain-sonata-v1m1-0-base + +# ScanNet +# linear probing +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0a-scannet-lin -n semseg-sonata-v1m1-0-base-0a-scannet-lin -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +# decoder probing +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0b-scannet-dec -n semseg-sonata-v1m1-0-base-0b-scannet-dec -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +# full fine-tuning +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0c-scannet-ft -n semseg-sonata-v1m1-0-base-0c-scannet-ft -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +# PPT multi-dataset joint fine-tuning (heavy, not necessary) +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0d-scannet-ppt -n semseg-sonata-v1m1-0-base-0d-scannet-ppt -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth + +# ScanNet Data Efficiency full fine-tuning +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0e0-scannet-ft-la20 -n semseg-sonata-v1m1-0-base-0e0-scannet-ft-la20 -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0e1-scannet-ft-la50 -n semseg-sonata-v1m1-0-base-0e1-scannet-ft-la50 -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0e2-scannet-ft-la100 -n semseg-sonata-v1m1-0-base-0e2-scannet-ft-la100 -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0e3-scannet-ft-la200 -n semseg-sonata-v1m1-0-base-0e3-scannet-ft-la200 -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0e4-scannet-ft-lr1 -n semseg-sonata-v1m1-0-base-0e4-scannet-ft-lr1 -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0e5-scannet-ft-lr5 -n semseg-sonata-v1m1-0-base-0e5-scannet-ft-lr5 -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0e6-scannet-ft-lr10 -n semseg-sonata-v1m1-0-base-0e6-scannet-ft-lr10 -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-0e7-scannet-ft-lr20 -n semseg-sonata-v1m1-0-base-0e7-scannet-ft-lr20 -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth + +# ScanNet200 +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-1a-scannet200-lin -n semseg-sonata-v1m1-0-base-1a-scannet200-lin -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-1b-scannet200-dec -n semseg-sonata-v1m1-0-base-1b-scannet200-dec -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-1c-scannet200-ft -n semseg-sonata-v1m1-0-base-1c-scannet200-ft -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth + +# ScanNetpp +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-2a-scannetpp-lin -n semseg-sonata-v1m1-0-base-2a-scannetpp-lin -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-2b-scannetpp-dec -n semseg-sonata-v1m1-0-base-2b-scannetpp-dec -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-2c-scannetpp-ft -n semseg-sonata-v1m1-0-base-2c-scannetpp-ft -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth + +# S3DIS Area 5 +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-3a-s3dis-lin -n semseg-sonata-v1m1-0-base-3a-s3dis-lin -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-3b-s3dis-dec -n semseg-sonata-v1m1-0-base-3b-s3dis-dec -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d sonata -c semseg-sonata-v1m1-3c-s3dis-ft -n semseg-sonata-v1m1-0-base-3c-s3dis-ft -w exp/sonata/pretrain-sonata-v1m1-0-base/model/model_last.pth + +# S3DIS 6-fold +# It's easy to adapt config for each split of S3DIS. +# Gathering all results file into one folder (RECORD_ROOT) +# and run the following script to get the final result: +python tools/test_s3dis_6fold.py --record_root $RECORD_ROOT +``` +The above configs are not verified with one additional running. If you encounter any problems, please feel free to let me know. Also, comparing changes between each configuration would be helpful in handling the configuration system of Pointcept. + +## Citation +If you find _Sonata_ useful to your research, please consider citing our line of works as an acknowledgment. (੭ˊ꒳​ˋ)੭✧ +```bib +@inproceedings{wu2025sonata, + title={Sonata: Self-Supervised Learning of Reliable Point Representations}, + author={Wu, Xiaoyang and DeTone, Daniel and Frost, Duncan and Shen, Tianwei and Xie, Chris and Yang, Nan and Engel, Jakob and Newcombe, Richard and Zhao, Hengshuang and Straub, Julian}, + booktitle={CVPR}, + year={2025} +} +``` + +```bib +@inproceedings{wu2024ptv3, + title={Point Transformer V3: Simpler, Faster, Stronger}, + author={Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang}, + booktitle={CVPR}, + year={2024} +} +``` +```bib +@inproceedings{wu2024ppt, + title={Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training}, + author={Wu, Xiaoyang and Tian, Zhuotao and Wen, Xin and Peng, Bohao and Liu, Xihui and Yu, Kaicheng and Zhao, Hengshuang}, + booktitle={CVPR}, + year={2024} +} +``` +```bib +@inproceedings{wu2023masked, + title={Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning}, + author={Wu, Xiaoyang and Wen, Xin and Liu, Xihui and Zhao, Hengshuang}, + journal={CVPR}, + year={2023} +} +``` +```bib +@inproceedings{wu2022ptv2, + title={Point transformer V2: Grouped Vector Attention and Partition-based Pooling}, + author={Wu, Xiaoyang and Lao, Yixing and Jiang, Li and Liu, Xihui and Zhao, Hengshuang}, + booktitle={NeurIPS}, + year={2022} +} +``` +```bib +@misc{pointcept2023, + title={Pointcept: A Codebase for Point Cloud Perception Research}, + author={Pointcept Contributors}, + howpublished={\url{https://github.com/Pointcept/Pointcept}}, + year={2023} +} +``` + +## License +- Sonata code is released by Meta under the [Apache 2.0 license](https://github.com/facebookresearch/sonata/blob/main/LICENSE), +- Sonata weight is released under the [CC-BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en) +(restricted by NC of datasets like HM3D, ArkitScenes). +- For commercial usage, please removing datasets restrict by non-commercial license. diff --git a/pointcept/models/sonata/__init__.py b/pointcept/models/sonata/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..6453e0d645f2ae4166df4ef4bcd443fc40de7b7e --- /dev/null +++ b/pointcept/models/sonata/__init__.py @@ -0,0 +1,3 @@ +from .sonata_v1m1_base import * +from .sonata_v1m2_uni_teacher_head import * +from .sonata_v1m3_distill import * diff --git a/pointcept/models/sonata/sonata_v1m1_base.py b/pointcept/models/sonata/sonata_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..e545b96ac528fb44b4709822214ce297e5dd8a9e --- /dev/null +++ b/pointcept/models/sonata/sonata_v1m1_base.py @@ -0,0 +1,532 @@ +""" +Sonata v1m1 Base + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from itertools import chain +from packaging import version +from functools import partial +import torch +import torch.nn as nn +import torch.nn.functional as F +import torch.distributed as dist +import torch_scatter +from timm.layers import trunc_normal_ + +import pointops +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS, build_model +from pointcept.models.modules import PointModel +from pointcept.models.utils import offset2batch, offset2bincount, batch2offset +from pointcept.utils.comm import get_world_size, all_gather +from pointcept.utils.scheduler import CosineScheduler + + +class OnlineCluster(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=4096, + embed_channels=512, + num_prototypes=4096, + ): + super().__init__() + self.mlp = nn.Sequential( + nn.Linear(in_channels, hidden_channels), + nn.GELU(), + nn.Linear(hidden_channels, embed_channels), + ) + self.apply(self._init_weights) + if version.parse(torch.__version__) >= version.parse("2.1.0"): + self.prototype = torch.nn.utils.parametrizations.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.parametrizations.weight.original0.data.fill_(1) + self.prototype.parametrizations.weight.original0.requires_grad = False + + else: + self.prototype = torch.nn.utils.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.weight_g.data.fill_(1) + self.prototype.weight_g.requires_grad = False + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + + def forward(self, feat): + feat = self.mlp(feat) + eps = 1e-6 if feat.dtype == torch.float16 else 1e-12 + feat = nn.functional.normalize(feat, dim=-1, p=2, eps=eps) + similarity = self.prototype(feat) + return similarity + + +@MODELS.register_module("Sonata-v1m1") +class Sonata(PointModel): + def __init__( + self, + backbone, + head_in_channels, + head_hidden_channels=4096, + head_embed_channels=512, + head_num_prototypes=4096, + teacher_custom=None, + num_global_view=2, + num_local_view=4, + mask_size_start=0.1, + mask_size_base=0.4, + mask_size_warmup_ratio=0.05, + mask_ratio_start=0.3, + mask_ratio_base=0.7, + mask_ratio_warmup_ratio=0.05, + mask_jitter=None, + teacher_temp_start=0.04, + teacher_temp_base=0.07, + teacher_temp_warmup_ratio=0.05, + student_temp=0.1, + mask_loss_weight=2 / 8, + roll_mask_loss_weight=2 / 8, + unmask_loss_weight=4 / 8, + momentum_base=0.996, + momentum_final=1, + match_max_k=8, + match_max_r=0.08, + up_cast_level=2, + ): + super(Sonata, self).__init__() + self.mask_loss_weight = mask_loss_weight + self.roll_mask_loss_weight = roll_mask_loss_weight + self.unmask_loss_weight = unmask_loss_weight + + self.num_global_view = num_global_view + self.num_local_view = num_local_view + + # masking and scheduler + self.mask_size = mask_size_start + self.mask_size_start = mask_size_start + self.mask_size_base = mask_size_base + self.mask_size_warmup_ratio = mask_size_warmup_ratio + self.mask_size_scheduler = None + + self.mask_ratio = mask_ratio_start + self.mask_ratio_start = mask_ratio_start + self.mask_ratio_base = mask_ratio_base + self.mask_ratio_warmup_ratio = mask_ratio_warmup_ratio + self.mask_ratio_scheduler = None + + self.mask_jitter = mask_jitter + + # temperature and scheduler + self.teacher_temp = teacher_temp_start + self.teacher_temp_start = teacher_temp_start + self.teacher_temp_base = teacher_temp_base + self.teacher_temp_warmup_ratio = teacher_temp_warmup_ratio + self.teacher_temp_scheduler = None + self.student_temp = student_temp + + # momentum and scheduler + self.momentum = momentum_base + self.momentum_base = momentum_base + self.momentum_final = momentum_final + self.momentum_scheduler = None + + # dynamic matching + self.match_max_k = match_max_k + self.match_max_r = match_max_r + + # up cast level + self.up_cast_level = up_cast_level + + # one of unmask, mask, roll mask loss enable + assert unmask_loss_weight + mask_loss_weight + roll_mask_loss_weight > 0 + # roll mask loss need more than one global view + assert num_global_view > 1 or roll_mask_loss_weight == 0 + # current roll mask only support two global views + assert num_global_view == 1 or num_global_view == 2 + + student_model_dict = dict() + teacher_model_dict = dict() + if teacher_custom is None: + teacher_custom = {} + student_backbone = build_model(backbone) + # turn off parameters like drop path for teacher model + backbone.update(teacher_custom) + + teacher_backbone = build_model(backbone) + student_model_dict["backbone"] = student_backbone + teacher_model_dict["backbone"] = teacher_backbone + + head = partial( + OnlineCluster, + in_channels=head_in_channels, + hidden_channels=head_hidden_channels, + embed_channels=head_embed_channels, + num_prototypes=head_num_prototypes, + ) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + student_model_dict["mask_head"] = head() + teacher_model_dict["mask_head"] = head() + if self.unmask_loss_weight > 0: + student_model_dict["unmask_head"] = head() + teacher_model_dict["unmask_head"] = head() + + self.student = nn.ModuleDict(student_model_dict) + self.teacher = nn.ModuleDict(teacher_model_dict) + for k, v in self.student.items(): + self.teacher[k].load_state_dict(self.student[k].state_dict()) + for p in self.teacher.parameters(): + p.requires_grad = False + + def before_train(self): + # make ModelHook after CheckPointLoader + total_steps = self.trainer.cfg.scheduler.total_steps + curr_step = self.trainer.start_epoch * len(self.trainer.train_loader) + # mask size scheduler + self.mask_size_scheduler = CosineScheduler( + start_value=self.mask_size_start, + base_value=self.mask_size_base, + final_value=self.mask_size_base, + warmup_iters=int(total_steps * self.mask_size_warmup_ratio), + total_iters=total_steps, + ) + self.mask_size_scheduler.iter = curr_step + + # mask ratio scheduler + self.mask_ratio_scheduler = CosineScheduler( + start_value=self.mask_ratio_start, + base_value=self.mask_ratio_base, + final_value=self.mask_ratio_base, + warmup_iters=int(total_steps * self.mask_ratio_warmup_ratio), + total_iters=total_steps, + ) + self.mask_ratio_scheduler.iter = curr_step + + # teacher temperature scheduler + self.teacher_temp_scheduler = CosineScheduler( + start_value=self.teacher_temp_start, + base_value=self.teacher_temp_base, + final_value=self.teacher_temp_base, + warmup_iters=int(total_steps * self.teacher_temp_warmup_ratio), + total_iters=total_steps, + ) + self.teacher_temp_scheduler.iter = curr_step + + # momentum scheduler + self.momentum_scheduler = CosineScheduler( + base_value=self.momentum_base, + final_value=self.momentum_final, + total_iters=total_steps, + ) + self.momentum_scheduler.iter = curr_step + + def before_step(self): + # update parameters from schedulers + self.mask_size = self.mask_size_scheduler.step() + self.mask_ratio = self.mask_ratio_scheduler.step() + self.teacher_temp = self.teacher_temp_scheduler.step() + self.momentum = self.momentum_scheduler.step() + + if self.trainer.writer is not None: + self.trainer.writer.add_scalar( + "params/mask_size", + self.mask_size, + self.mask_size_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/mask_ratio", + self.mask_ratio, + self.mask_ratio_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/teacher_temp", + self.teacher_temp, + self.teacher_temp_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/momentum", + self.momentum, + self.momentum_scheduler.iter, + ) + + def after_step(self): + # EMA update teacher + with torch.no_grad(): + m = self.momentum + student_param_list = list(self.student.parameters()) + teacher_param_list = list(self.teacher.parameters()) + torch._foreach_mul_(teacher_param_list, m) + torch._foreach_add_(teacher_param_list, student_param_list, alpha=1 - m) + + @staticmethod + def sinkhorn_knopp(feat, temp, num_iter=3): + feat = feat.float() + q = torch.exp(feat / temp).t() + n = sum(all_gather(q.shape[1])) # number of samples to assign + k = q.shape[0] # number of prototypes + + # make the matrix sums to 1 + sum_q = q.sum() + if get_world_size() > 1: + dist.all_reduce(sum_q) + q = q / sum_q + + for i in range(num_iter): + # normalize each row: total weight per prototype must be 1/k + q_row_sum = q.sum(dim=1, keepdim=True) + if get_world_size() > 1: + dist.all_reduce(q_row_sum) + q = q / q_row_sum / k + + # normalize each column: total weight per sample must be 1/n + q = q / q.sum(dim=0, keepdim=True) / n + + q *= n # the columns must sum to 1 so that Q is an assignment + return q.t() + + def generate_mask(self, coord, offset): + batch = offset2batch(offset) + mask_size = self.mask_size + mask_ratio = self.mask_ratio + + # Grouping points with grid patch + min_coord = torch_scatter.segment_coo(coord, batch, reduce="min") + grid_coord = ((coord - min_coord[batch]) // mask_size).int() + grid_coord = torch.cat([batch.unsqueeze(-1), grid_coord], dim=-1) + unique, point_cluster, counts = torch.unique( + grid_coord, dim=0, sorted=True, return_inverse=True, return_counts=True + ) + patch_num = unique.shape[0] + mask_patch_num = int(patch_num * mask_ratio) + patch_index = torch.randperm(patch_num, device=coord.device) + mask_patch_index = patch_index[:mask_patch_num] + point_mask = torch.isin(point_cluster, mask_patch_index) + return point_mask, point_cluster + + @torch.no_grad() + def match_neighbour( + self, + view1_coord, + view1_offset, + view2_coord, + view2_offset, + ): + index2, distance = pointops.knn_query( + 1, + view2_coord.float(), + view2_offset.int(), + view1_coord.float(), + view1_offset.int(), + ) + index1 = torch.arange( + index2.shape[0], device=index2.device, dtype=torch.long + ).unsqueeze(-1) + index = torch.cat([index1, index2], dim=-1)[ + distance.squeeze(-1) < self.match_max_r + ] + return index + + @torch.no_grad() + def roll_point(self, point): + n = self.num_global_view + # [pc1, pc1', pc2, pc2'] -> [pc1', pc1, pc2', pc2], only support num_global_view == 2 + bs = len(point.offset) // self.num_global_view + data_dict = {} + for key in point.keys(): + if key in ["feat", "coord", "origin_coord", "batch"]: + value = point[key].split(offset2bincount(point.offset).tolist()) + value = chain(*[value[n * b : n * (b + 1)][::-1] for b in range(bs)]) + if key == "batch": + value = [torch.ones_like(v) * i for i, v in enumerate(value)] + data_dict[key] = torch.cat(list(value), dim=0) + return Point(data_dict) + + def up_cast(self, point): + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return point + + def forward(self, data_dict, return_point=False): + if return_point: + point = self.teacher.backbone(data_dict) + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return dict(point=point) + + # prepare global_point, mask_global_point, local_point + with torch.no_grad(): + # global_point & masking + global_point = Point( + feat=data_dict["global_feat"], + coord=data_dict["global_coord"], + origin_coord=data_dict["global_origin_coord"], + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + global_mask, global_cluster = self.generate_mask( + global_point.coord, global_point.offset + ) + mask_global_coord = global_point.coord.clone().detach() + if self.mask_jitter is not None: + mask_global_coord[global_mask] += torch.clip( + torch.randn_like(mask_global_coord[global_mask]).mul( + self.mask_jitter + ), + max=self.mask_jitter * 2, + ) + + mask_global_point = Point( + feat=data_dict["global_feat"], + coord=mask_global_coord, + origin_coord=data_dict["global_origin_coord"], + mask=global_mask, + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # local point & matching + local_point = Point( + feat=data_dict["local_feat"], + coord=data_dict["local_coord"], + origin_coord=data_dict["local_origin_coord"], + offset=data_dict["local_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # create result dictionary for return + result_dict = dict(loss=[]) + # teacher backbone forward (shared with mask and unmask) + global_point_ = self.teacher.backbone(global_point) + global_point_ = self.up_cast(global_point_) + global_feat = global_point_.feat + + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + # teacher head forward + with torch.no_grad(): + global_point_.feat = self.teacher.mask_head(global_feat) + # student forward + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_pred_sim = self.student.mask_head(mask_global_point_.feat) + + if self.mask_loss_weight > 0: + with torch.no_grad(): + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + global_point_.origin_coord, + global_point_.offset, + ) + # teacher forward + mask_target_sim = self.sinkhorn_knopp( + global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + # loss + mask_loss = -torch.sum( + mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + mask_loss = torch_scatter.segment_coo( + mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["mask_loss"] = mask_loss + result_dict["loss"].append(mask_loss * self.mask_loss_weight) + + if self.roll_mask_loss_weight > 0: + roll_global_point_ = self.roll_point(global_point_) + with torch.no_grad(): + # match index for pred and roll target + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + roll_global_point_.origin_coord, + roll_global_point_.offset, + ) + # teacher forward + roll_mask_target_sim = self.sinkhorn_knopp( + roll_global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + roll_mask_loss = -torch.sum( + roll_mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + roll_mask_loss = torch_scatter.segment_coo( + roll_mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["roll_mask_loss"] = roll_mask_loss + result_dict["loss"].append(roll_mask_loss * self.roll_mask_loss_weight) + if self.unmask_loss_weight > 0: + # teacher head forward + with torch.no_grad(): + global_point_.feat = self.teacher.unmask_head(global_feat) + # student forward + local_point_ = self.student.backbone(local_point) + local_point_ = self.up_cast(local_point_) + unmask_pred_sim = self.student.unmask_head(local_point_.feat) + with torch.no_grad(): + principal_view_mask = global_point_.batch % self.num_global_view == 0 + principal_view_batch = ( + global_point_.batch[principal_view_mask] // self.num_global_view + ) + match_index = self.match_neighbour( + local_point_.origin_coord, + local_point_.offset[self.num_local_view - 1 :: self.num_local_view], + global_point_.origin_coord[principal_view_mask], + batch2offset(principal_view_batch), + ) + # teacher forward + unmask_target_sim = self.sinkhorn_knopp( + global_point_.feat[principal_view_mask][match_index[:, 1]], + self.teacher_temp, + ) + # loss + unmask_loss = -torch.sum( + unmask_target_sim + * F.log_softmax( + unmask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + unmask_loss = torch_scatter.segment_coo( + unmask_loss, + index=local_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["unmask_loss"] = unmask_loss + result_dict["loss"].append(unmask_loss * self.unmask_loss_weight) + result_dict["loss"] = sum(result_dict["loss"]) + + if get_world_size() > 1: + for loss in result_dict.values(): + dist.all_reduce(loss, op=dist.ReduceOp.AVG) + return result_dict diff --git a/pointcept/models/sonata/sonata_v1m2_uni_teacher_head.py b/pointcept/models/sonata/sonata_v1m2_uni_teacher_head.py new file mode 100644 index 0000000000000000000000000000000000000000..73c55b26382654dd8c0fcf22ba3bf3cca9bb309d --- /dev/null +++ b/pointcept/models/sonata/sonata_v1m2_uni_teacher_head.py @@ -0,0 +1,535 @@ +""" +Sonata v1m1 Base + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from itertools import chain +from packaging import version +from functools import partial +import torch +import torch.nn as nn +import torch.nn.functional as F +import torch.distributed as dist +import torch_scatter +from timm.layers import trunc_normal_ + +import pointops +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS, build_model +from pointcept.models.modules import PointModel +from pointcept.models.utils import offset2batch, offset2bincount, batch2offset +from pointcept.utils.comm import get_world_size, all_gather +from pointcept.utils.scheduler import CosineScheduler + + +class OnlineCluster(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=4096, + embed_channels=512, + num_prototypes=4096, + ): + super().__init__() + self.mlp = nn.Sequential( + nn.Linear(in_channels, hidden_channels), + nn.GELU(), + nn.Linear(hidden_channels, embed_channels), + ) + self.apply(self._init_weights) + if version.parse(torch.__version__) >= version.parse("2.1.0"): + self.prototype = torch.nn.utils.parametrizations.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.parametrizations.weight.original0.data.fill_(1) + self.prototype.parametrizations.weight.original0.requires_grad = False + + else: + self.prototype = torch.nn.utils.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.weight_g.data.fill_(1) + self.prototype.weight_g.requires_grad = False + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + + def forward(self, feat): + feat = self.mlp(feat) + eps = 1e-6 if feat.dtype == torch.float16 else 1e-12 + feat = nn.functional.normalize(feat, dim=-1, p=2, eps=eps) + similarity = self.prototype(feat) + return similarity + + +@MODELS.register_module("Sonata-v1m2") +class Sonata(PointModel): + def __init__( + self, + backbone, + head_in_channels, + head_hidden_channels=4096, + head_embed_channels=512, + head_num_prototypes=4096, + teacher_custom=None, + num_global_view=2, + num_local_view=4, + mask_size_start=0.1, + mask_size_base=0.4, + mask_size_warmup_ratio=0.05, + mask_ratio_start=0.3, + mask_ratio_base=0.7, + mask_ratio_warmup_ratio=0.05, + mask_jitter=None, + teacher_temp_start=0.04, + teacher_temp_base=0.07, + teacher_temp_warmup_ratio=0.05, + student_temp=0.1, + mask_loss_weight=2 / 8, + roll_mask_loss_weight=2 / 8, + unmask_loss_weight=4 / 8, + momentum_base=0.996, + momentum_final=1, + match_max_k=8, + match_max_r=0.08, + up_cast_level=2, + ): + super(Sonata, self).__init__() + self.mask_loss_weight = mask_loss_weight + self.roll_mask_loss_weight = roll_mask_loss_weight + self.unmask_loss_weight = unmask_loss_weight + + self.num_global_view = num_global_view + self.num_local_view = num_local_view + + # masking and scheduler + self.mask_size = mask_size_start + self.mask_size_start = mask_size_start + self.mask_size_base = mask_size_base + self.mask_size_warmup_ratio = mask_size_warmup_ratio + self.mask_size_scheduler = None + + self.mask_ratio = mask_ratio_start + self.mask_ratio_start = mask_ratio_start + self.mask_ratio_base = mask_ratio_base + self.mask_ratio_warmup_ratio = mask_ratio_warmup_ratio + self.mask_ratio_scheduler = None + + self.mask_jitter = mask_jitter + + # temperature and scheduler + self.teacher_temp = teacher_temp_start + self.teacher_temp_start = teacher_temp_start + self.teacher_temp_base = teacher_temp_base + self.teacher_temp_warmup_ratio = teacher_temp_warmup_ratio + self.teacher_temp_scheduler = None + self.student_temp = student_temp + + # momentum and scheduler + self.momentum = momentum_base + self.momentum_base = momentum_base + self.momentum_final = momentum_final + self.momentum_scheduler = None + + # dynamic matching + self.match_max_k = match_max_k + self.match_max_r = match_max_r + + # up cast level + self.up_cast_level = up_cast_level + + # one of unmask, mask, roll mask loss enable + assert unmask_loss_weight + mask_loss_weight + roll_mask_loss_weight > 0 + # roll mask loss need more than one global view + assert num_global_view > 1 or roll_mask_loss_weight == 0 + # current roll mask only support two global views + assert num_global_view == 1 or num_global_view == 2 + + student_model_dict = dict() + teacher_model_dict = dict() + if teacher_custom is None: + teacher_custom = {} + student_backbone = build_model(backbone) + # turn off parameters like drop path for teacher model + backbone.update(teacher_custom) + + teacher_backbone = build_model(backbone) + student_model_dict["backbone"] = student_backbone + teacher_model_dict["backbone"] = teacher_backbone + + head = partial( + OnlineCluster, + in_channels=head_in_channels, + hidden_channels=head_hidden_channels, + embed_channels=head_embed_channels, + num_prototypes=head_num_prototypes, + ) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + student_model_dict["mask_head"] = head() + # teacher share only one head and EMA update by student.mask_head (global) + teacher_model_dict["mask_head"] = head() + + if self.unmask_loss_weight > 0: + student_model_dict["unmask_head"] = head() + # dummy head for EMA update implementation + teacher_model_dict["unmask_head"] = head() + + self.student = nn.ModuleDict(student_model_dict) + self.teacher = nn.ModuleDict(teacher_model_dict) + for k, v in self.student.items(): + self.teacher[k].load_state_dict(self.student[k].state_dict()) + for p in self.teacher.parameters(): + p.requires_grad = False + + def before_train(self): + # make ModelHook after CheckPointLoader + total_steps = self.trainer.cfg.scheduler.total_steps + curr_step = self.trainer.start_epoch * len(self.trainer.train_loader) + # mask size scheduler + self.mask_size_scheduler = CosineScheduler( + start_value=self.mask_size_start, + base_value=self.mask_size_base, + final_value=self.mask_size_base, + warmup_iters=int(total_steps * self.mask_size_warmup_ratio), + total_iters=total_steps, + ) + self.mask_size_scheduler.iter = curr_step + + # mask ratio scheduler + self.mask_ratio_scheduler = CosineScheduler( + start_value=self.mask_ratio_start, + base_value=self.mask_ratio_base, + final_value=self.mask_ratio_base, + warmup_iters=int(total_steps * self.mask_ratio_warmup_ratio), + total_iters=total_steps, + ) + self.mask_ratio_scheduler.iter = curr_step + + # teacher temperature scheduler + self.teacher_temp_scheduler = CosineScheduler( + start_value=self.teacher_temp_start, + base_value=self.teacher_temp_base, + final_value=self.teacher_temp_base, + warmup_iters=int(total_steps * self.teacher_temp_warmup_ratio), + total_iters=total_steps, + ) + self.teacher_temp_scheduler.iter = curr_step + + # momentum scheduler + self.momentum_scheduler = CosineScheduler( + base_value=self.momentum_base, + final_value=self.momentum_final, + total_iters=total_steps, + ) + self.momentum_scheduler.iter = curr_step + + def before_step(self): + # update parameters from schedulers + self.mask_size = self.mask_size_scheduler.step() + self.mask_ratio = self.mask_ratio_scheduler.step() + self.teacher_temp = self.teacher_temp_scheduler.step() + self.momentum = self.momentum_scheduler.step() + + if self.trainer.writer is not None: + self.trainer.writer.add_scalar( + "params/mask_size", + self.mask_size, + self.mask_size_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/mask_ratio", + self.mask_ratio, + self.mask_ratio_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/teacher_temp", + self.teacher_temp, + self.teacher_temp_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/momentum", + self.momentum, + self.momentum_scheduler.iter, + ) + + def after_step(self): + # EMA update teacher + with torch.no_grad(): + m = self.momentum + student_param_list = list(self.student.parameters()) + teacher_param_list = list(self.teacher.parameters()) + torch._foreach_mul_(teacher_param_list, m) + torch._foreach_add_(teacher_param_list, student_param_list, alpha=1 - m) + + @staticmethod + def sinkhorn_knopp(feat, temp, num_iter=3): + feat = feat.float() + q = torch.exp(feat / temp).t() + n = sum(all_gather(q.shape[1])) # number of samples to assign + k = q.shape[0] # number of prototypes + + # make the matrix sums to 1 + sum_q = q.sum() + if get_world_size() > 1: + dist.all_reduce(sum_q) + q = q / sum_q + + for i in range(num_iter): + # normalize each row: total weight per prototype must be 1/k + q_row_sum = q.sum(dim=1, keepdim=True) + if get_world_size() > 1: + dist.all_reduce(q_row_sum) + q = q / q_row_sum / k + + # normalize each column: total weight per sample must be 1/n + q = q / q.sum(dim=0, keepdim=True) / n + + q *= n # the columns must sum to 1 so that Q is an assignment + return q.t() + + def generate_mask(self, coord, offset): + batch = offset2batch(offset) + mask_size = self.mask_size + mask_ratio = self.mask_ratio + + # Grouping points with grid patch + min_coord = torch_scatter.segment_coo(coord, batch, reduce="min") + grid_coord = ((coord - min_coord[batch]) // mask_size).int() + grid_coord = torch.cat([batch.unsqueeze(-1), grid_coord], dim=-1) + unique, point_cluster, counts = torch.unique( + grid_coord, dim=0, sorted=True, return_inverse=True, return_counts=True + ) + patch_num = unique.shape[0] + mask_patch_num = int(patch_num * mask_ratio) + patch_index = torch.randperm(patch_num, device=coord.device) + mask_patch_index = patch_index[:mask_patch_num] + point_mask = torch.isin(point_cluster, mask_patch_index) + return point_mask, point_cluster + + @torch.no_grad() + def match_neighbour( + self, + view1_coord, + view1_offset, + view2_coord, + view2_offset, + ): + index2, distance = pointops.knn_query( + 1, + view2_coord.float(), + view2_offset.int(), + view1_coord.float(), + view1_offset.int(), + ) + index1 = torch.arange( + index2.shape[0], device=index2.device, dtype=torch.long + ).unsqueeze(-1) + index = torch.cat([index1, index2], dim=-1)[ + distance.squeeze(-1) < self.match_max_r + ] + return index + + @torch.no_grad() + def roll_point(self, point): + n = self.num_global_view + # [pc1, pc1', pc2, pc2'] -> [pc1', pc1, pc2', pc2], only support num_global_view == 2 + bs = len(point.offset) // self.num_global_view + data_dict = {} + for key in point.keys(): + if key in ["feat", "coord", "origin_coord", "batch"]: + value = point[key].split(offset2bincount(point.offset).tolist()) + value = chain(*[value[n * b : n * (b + 1)][::-1] for b in range(bs)]) + if key == "batch": + value = [torch.ones_like(v) * i for i, v in enumerate(value)] + data_dict[key] = torch.cat(list(value), dim=0) + return Point(data_dict) + + def up_cast(self, point): + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return point + + def forward(self, data_dict, return_point=False): + if return_point: + point = self.teacher.backbone(data_dict) + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return dict(point=point) + + # prepare global_point, mask_global_point, local_point + with torch.no_grad(): + # global_point & masking + global_point = Point( + feat=data_dict["global_feat"], + coord=data_dict["global_coord"], + origin_coord=data_dict["global_origin_coord"], + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + global_mask, global_cluster = self.generate_mask( + global_point.coord, global_point.offset + ) + mask_global_coord = global_point.coord.clone().detach() + if self.mask_jitter is not None: + mask_global_coord[global_mask] += torch.clip( + torch.randn_like(mask_global_coord[global_mask]).mul( + self.mask_jitter + ), + max=self.mask_jitter * 2, + ) + + mask_global_point = Point( + feat=data_dict["global_feat"], + coord=mask_global_coord, + origin_coord=data_dict["global_origin_coord"], + mask=global_mask, + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # local point & matching + local_point = Point( + feat=data_dict["local_feat"], + coord=data_dict["local_coord"], + origin_coord=data_dict["local_origin_coord"], + offset=data_dict["local_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # create result dictionary for return + result_dict = dict(loss=[]) + # teacher backbone forward (shared with mask and unmask) + global_point_ = self.teacher.backbone(global_point) + global_point_ = self.up_cast(global_point_) + # teacher head forward + # only use one shared head for both mask and unmask + # priority: mask (global) > unmask (local) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + global_point_.feat = self.teacher.mask_head(global_point_.feat) + else: + global_point_.feat = self.teacher.unmask_head(global_point_.feat) + + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + # student forward + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_pred_sim = self.student.mask_head(mask_global_point_.feat) + + if self.mask_loss_weight > 0: + with torch.no_grad(): + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + global_point_.origin_coord, + global_point_.offset, + ) + # teacher forward + mask_target_sim = self.sinkhorn_knopp( + global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + # loss + mask_loss = -torch.sum( + mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + mask_loss = torch_scatter.segment_coo( + mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["mask_loss"] = mask_loss + result_dict["loss"].append(mask_loss * self.mask_loss_weight) + + if self.roll_mask_loss_weight > 0: + roll_global_point_ = self.roll_point(global_point_) + with torch.no_grad(): + # match index for pred and roll target + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + roll_global_point_.origin_coord, + roll_global_point_.offset, + ) + # teacher forward + roll_mask_target_sim = self.sinkhorn_knopp( + roll_global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + roll_mask_loss = -torch.sum( + roll_mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + roll_mask_loss = torch_scatter.segment_coo( + roll_mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["roll_mask_loss"] = roll_mask_loss + result_dict["loss"].append(roll_mask_loss * self.roll_mask_loss_weight) + if self.unmask_loss_weight > 0: + # student forward + local_point_ = self.student.backbone(local_point) + local_point_ = self.up_cast(local_point_) + unmask_pred_sim = self.student.unmask_head(local_point_.feat) + with torch.no_grad(): + principal_view_mask = global_point_.batch % self.num_global_view == 0 + principal_view_batch = ( + global_point_.batch[principal_view_mask] // self.num_global_view + ) + match_index = self.match_neighbour( + local_point_.origin_coord, + local_point_.offset[self.num_local_view - 1 :: self.num_local_view], + global_point_.origin_coord[principal_view_mask], + batch2offset(principal_view_batch), + ) + # teacher forward + unmask_target_sim = self.sinkhorn_knopp( + global_point_.feat[principal_view_mask][match_index[:, 1]], + self.teacher_temp, + ) + # loss + unmask_loss = -torch.sum( + unmask_target_sim + * F.log_softmax( + unmask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + unmask_loss = torch_scatter.segment_coo( + unmask_loss, + index=local_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["unmask_loss"] = unmask_loss + result_dict["loss"].append(unmask_loss * self.unmask_loss_weight) + result_dict["loss"] = sum(result_dict["loss"]) + + if get_world_size() > 1: + for loss in result_dict.values(): + dist.all_reduce(loss, op=dist.ReduceOp.AVG) + return result_dict diff --git a/pointcept/models/sonata/sonata_v1m3_distill.py b/pointcept/models/sonata/sonata_v1m3_distill.py new file mode 100644 index 0000000000000000000000000000000000000000..5a61ef6c1c03a6865b5556ad7fd71becdcee8c46 --- /dev/null +++ b/pointcept/models/sonata/sonata_v1m3_distill.py @@ -0,0 +1,565 @@ +""" +Sonata V1m3 Distill + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from itertools import chain +from packaging import version +from functools import partial +import torch +import torch.nn as nn +import torch.nn.functional as F +import torch.distributed as dist +import torch_scatter +from timm.layers import trunc_normal_ + +import pointops +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS, build_model +from pointcept.models.modules import PointModel +from pointcept.models.utils import offset2batch, offset2bincount, batch2offset +from pointcept.utils.comm import get_world_size, all_gather +from pointcept.utils.scheduler import CosineScheduler + + +class OnlineCluster(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=4096, + embed_channels=512, + num_prototypes=4096, + ): + super().__init__() + self.mlp = nn.Sequential( + nn.Linear(in_channels, hidden_channels), + nn.GELU(), + nn.Linear(hidden_channels, embed_channels), + ) + self.apply(self._init_weights) + if version.parse(torch.__version__) >= version.parse("2.1.0"): + self.prototype = torch.nn.utils.parametrizations.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.parametrizations.weight.original0.data.fill_(1) + self.prototype.parametrizations.weight.original0.requires_grad = False + + else: + self.prototype = torch.nn.utils.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.weight_g.data.fill_(1) + self.prototype.weight_g.requires_grad = False + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + + def forward(self, feat): + feat = self.mlp(feat) + eps = 1e-6 if feat.dtype == torch.float16 else 1e-12 + feat = nn.functional.normalize(feat, dim=-1, p=2, eps=eps) + similarity = self.prototype(feat) + return similarity + + +@MODELS.register_module("Sonata-v1m3") +class Sonata(PointModel): + def __init__( + self, + backbone_s, + backbone_t, + head_in_channels_s, + head_in_channels_t, + head_hidden_channels=4096, + head_embed_channels=512, + head_num_prototypes=4096, + teacher_custom=None, + num_global_view=2, + num_local_view=4, + mask_size_start=0.1, + mask_size_base=0.4, + mask_size_warmup_ratio=0.05, + mask_ratio_start=0.3, + mask_ratio_base=0.7, + mask_ratio_warmup_ratio=0.05, + mask_jitter=None, + teacher_temp_start=0.04, + teacher_temp_base=0.07, + teacher_temp_warmup_ratio=0.05, + student_temp=0.1, + mask_loss_weight=2 / 8, + roll_mask_loss_weight=2 / 8, + unmask_loss_weight=4 / 8, + momentum_base=0.996, + momentum_final=1, + match_max_k=8, + match_max_r=0.08, + up_cast_level=2, + teacher_pretrained_path=None, + ): + super(Sonata, self).__init__() + self.mask_loss_weight = mask_loss_weight + self.roll_mask_loss_weight = roll_mask_loss_weight + self.unmask_loss_weight = unmask_loss_weight + + self.num_global_view = num_global_view + self.num_local_view = num_local_view + + # masking and scheduler + self.mask_size = mask_size_start + self.mask_size_start = mask_size_start + self.mask_size_base = mask_size_base + self.mask_size_warmup_ratio = mask_size_warmup_ratio + self.mask_size_scheduler = None + + self.mask_ratio = mask_ratio_start + self.mask_ratio_start = mask_ratio_start + self.mask_ratio_base = mask_ratio_base + self.mask_ratio_warmup_ratio = mask_ratio_warmup_ratio + self.mask_ratio_scheduler = None + + self.mask_jitter = mask_jitter + + # temperature and scheduler + self.teacher_temp = teacher_temp_start + self.teacher_temp_start = teacher_temp_start + self.teacher_temp_base = teacher_temp_base + self.teacher_temp_warmup_ratio = teacher_temp_warmup_ratio + self.teacher_temp_scheduler = None + self.student_temp = student_temp + + # momentum and scheduler + self.momentum = momentum_base + self.momentum_base = momentum_base + self.momentum_final = momentum_final + self.momentum_scheduler = None + + # dynamic matching + self.match_max_k = match_max_k + self.match_max_r = match_max_r + + # up cast level + self.up_cast_level = up_cast_level + + # one of unmask, mask, roll mask loss enable + assert unmask_loss_weight + mask_loss_weight + roll_mask_loss_weight > 0 + # roll mask loss need more than one global view + assert num_global_view > 1 or roll_mask_loss_weight == 0 + # current roll mask only support two global views + assert num_global_view == 1 or num_global_view == 2 + + student_model_dict = dict() + teacher_model_dict = dict() + if teacher_custom is None: + teacher_custom = {} + student_backbone = build_model(backbone_s) + # turn off parameters like drop path for teacher model + backbone_t.update(teacher_custom) + + teacher_backbone = build_model(backbone_t) + student_model_dict["backbone"] = student_backbone + teacher_model_dict["backbone"] = teacher_backbone + + head_t = partial( + OnlineCluster, + in_channels=head_in_channels_t, + hidden_channels=head_hidden_channels, + embed_channels=head_embed_channels, + num_prototypes=head_num_prototypes, + ) + head_s = partial( + OnlineCluster, + in_channels=head_in_channels_s, + hidden_channels=head_hidden_channels, + embed_channels=head_embed_channels, + num_prototypes=head_num_prototypes, + ) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + student_model_dict["mask_head"] = head_s() + teacher_model_dict["mask_head"] = head_t() + if self.unmask_loss_weight > 0: + student_model_dict["unmask_head"] = head_s() + teacher_model_dict["unmask_head"] = head_t() + + self.student = nn.ModuleDict(student_model_dict) + self.teacher = nn.ModuleDict(teacher_model_dict) + self.teacher = self.load_sonata(self.teacher, path=teacher_pretrained_path) + for p in self.teacher.parameters(): + p.requires_grad = False + + def load_sonata( + self, + model, + path="./ckpt/sonata.pth", + if_loadhead=True, + ): + checkpoint = torch.load(path, map_location=lambda storage, loc: storage.cuda()) + name_test = [n for n, p in model.named_parameters()] + weight = {} + whether_weight = False + if "state_dict" in checkpoint.keys(): + checkpoint = checkpoint["state_dict"] + for key, value in checkpoint.items(): + if if_loadhead: + if "module.student." in key: + whether_weight = True + key = key.replace("module.student.", "module.") + key = key[7:] # module.xxx.xxx -> xxx.xxx + weight[key] = value + else: + if "module.student.backbone." in key: + whether_weight = True + key = key.replace("module.student.backbone.", "module.") + key = key[7:] # module.xxx.xxx -> xxx.xxx + weight[key] = value + if whether_weight: + load_state_info = model.load_state_dict(weight, strict=False) + else: + load_state_info = model.load_state_dict(checkpoint) + print(f"Missing keys: {load_state_info[0]}") + print(f"Unexpected keys: {load_state_info[1]}") + return model + + def before_train(self): + # make ModelHook after CheckPointLoader + total_steps = self.trainer.cfg.scheduler.total_steps + curr_step = self.trainer.start_epoch * len(self.trainer.train_loader) + # mask size scheduler + self.mask_size_scheduler = CosineScheduler( + start_value=self.mask_size_start, + base_value=self.mask_size_base, + final_value=self.mask_size_base, + warmup_iters=int(total_steps * self.mask_size_warmup_ratio), + total_iters=total_steps, + ) + self.mask_size_scheduler.iter = curr_step + + # mask ratio scheduler + self.mask_ratio_scheduler = CosineScheduler( + start_value=self.mask_ratio_start, + base_value=self.mask_ratio_base, + final_value=self.mask_ratio_base, + warmup_iters=int(total_steps * self.mask_ratio_warmup_ratio), + total_iters=total_steps, + ) + self.mask_ratio_scheduler.iter = curr_step + + # teacher temperature scheduler + self.teacher_temp_scheduler = CosineScheduler( + start_value=self.teacher_temp_start, + base_value=self.teacher_temp_base, + final_value=self.teacher_temp_base, + warmup_iters=int(total_steps * self.teacher_temp_warmup_ratio), + total_iters=total_steps, + ) + self.teacher_temp_scheduler.iter = curr_step + + # momentum scheduler + self.momentum_scheduler = CosineScheduler( + base_value=self.momentum_base, + final_value=self.momentum_final, + total_iters=total_steps, + ) + self.momentum_scheduler.iter = curr_step + + def before_step(self): + # update parameters from schedulers + self.mask_size = self.mask_size_scheduler.step() + self.mask_ratio = self.mask_ratio_scheduler.step() + self.teacher_temp = self.teacher_temp_scheduler.step() + self.momentum = self.momentum_scheduler.step() + + if self.trainer.writer is not None: + self.trainer.writer.add_scalar( + "params/mask_size", + self.mask_size, + self.mask_size_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/mask_ratio", + self.mask_ratio, + self.mask_ratio_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/teacher_temp", + self.teacher_temp, + self.teacher_temp_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/momentum", + self.momentum, + self.momentum_scheduler.iter, + ) + + def after_step(self): + pass + + @staticmethod + def sinkhorn_knopp(feat, temp, num_iter=3): + feat = feat.float() + q = torch.exp(feat / temp).t() + n = sum(all_gather(q.shape[1])) # number of samples to assign + k = q.shape[0] # number of prototypes + + # make the matrix sums to 1 + sum_q = q.sum() + if get_world_size() > 1: + dist.all_reduce(sum_q) + q = q / sum_q + + for i in range(num_iter): + # normalize each row: total weight per prototype must be 1/k + q_row_sum = q.sum(dim=1, keepdim=True) + if get_world_size() > 1: + dist.all_reduce(q_row_sum) + q = q / q_row_sum / k + + # normalize each column: total weight per sample must be 1/n + q = q / q.sum(dim=0, keepdim=True) / n + + q *= n # the columns must sum to 1 so that Q is an assignment + return q.t() + + def generate_mask(self, coord, offset): + batch = offset2batch(offset) + mask_size = self.mask_size + mask_ratio = self.mask_ratio + + # Grouping points with grid patch + min_coord = torch_scatter.segment_coo(coord, batch, reduce="min") + grid_coord = ((coord - min_coord[batch]) // mask_size).int() + grid_coord = torch.cat([batch.unsqueeze(-1), grid_coord], dim=-1) + unique, point_cluster, counts = torch.unique( + grid_coord, dim=0, sorted=True, return_inverse=True, return_counts=True + ) + patch_num = unique.shape[0] + mask_patch_num = int(patch_num * mask_ratio) + patch_index = torch.randperm(patch_num, device=coord.device) + mask_patch_index = patch_index[:mask_patch_num] + point_mask = torch.isin(point_cluster, mask_patch_index) + return point_mask, point_cluster + + @torch.no_grad() + def match_neighbour( + self, + view1_coord, + view1_offset, + view2_coord, + view2_offset, + ): + index2, distance = pointops.knn_query( + 1, + view2_coord.float(), + view2_offset.int(), + view1_coord.float(), + view1_offset.int(), + ) + index1 = torch.arange( + index2.shape[0], device=index2.device, dtype=torch.long + ).unsqueeze(-1) + index = torch.cat([index1, index2], dim=-1)[ + distance.squeeze(-1) < self.match_max_r + ] + return index + + @torch.no_grad() + def roll_point(self, point): + n = self.num_global_view + # [pc1, pc1', pc2, pc2'] -> [pc1', pc1, pc2', pc2], only support num_global_view == 2 + bs = len(point.offset) // self.num_global_view + data_dict = {} + for key in point.keys(): + if key in ["feat", "coord", "origin_coord", "batch"]: + value = point[key].split(offset2bincount(point.offset).tolist()) + value = chain(*[value[n * b : n * (b + 1)][::-1] for b in range(bs)]) + if key == "batch": + value = [torch.ones_like(v) * i for i, v in enumerate(value)] + data_dict[key] = torch.cat(list(value), dim=0) + return Point(data_dict) + + def up_cast(self, point): + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return point + + def forward(self, data_dict, return_point=False): + if return_point: + point = self.teacher.backbone(data_dict) + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return dict(point=point) + + # prepare global_point, mask_global_point, local_point + with torch.no_grad(): + # global_point & masking + global_point = Point( + feat=data_dict["global_feat_full"], + coord=data_dict["global_coord"], + origin_coord=data_dict["global_origin_coord"], + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + global_mask, global_cluster = self.generate_mask( + global_point.coord, global_point.offset + ) + mask_global_coord = global_point.coord.clone().detach() + if self.mask_jitter is not None: + mask_global_coord[global_mask] += torch.clip( + torch.randn_like(mask_global_coord[global_mask]).mul( + self.mask_jitter + ), + max=self.mask_jitter * 2, + ) + + mask_global_point = Point( + feat=data_dict["global_feat"], + coord=mask_global_coord, + origin_coord=data_dict["global_origin_coord"], + mask=global_mask, + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # local point & matching + local_point = Point( + feat=data_dict["local_feat"], + coord=data_dict["local_coord"], + origin_coord=data_dict["local_origin_coord"], + offset=data_dict["local_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # create result dictionary for return + result_dict = dict(loss=[]) + # teacher backbone forward (shared with mask and unmask) + global_point_ = self.teacher.backbone(global_point) + global_point_ = self.up_cast(global_point_) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + global_point_.feat = self.teacher.mask_head(global_point_.feat) + else: + global_point_.feat = self.teacher.unmask_head(global_point_.feat) + + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + # student forward + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_pred_sim = self.student.mask_head(mask_global_point_.feat) + + if self.mask_loss_weight > 0: + with torch.no_grad(): + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + global_point_.origin_coord, + global_point_.offset, + ) + # teacher forward + mask_target_sim = self.sinkhorn_knopp( + global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + # loss + mask_loss = -torch.sum( + mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + mask_loss = torch_scatter.segment_coo( + mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["mask_loss"] = mask_loss + result_dict["loss"].append(mask_loss * self.mask_loss_weight) + + if self.roll_mask_loss_weight > 0: + roll_global_point_ = self.roll_point(global_point_) + with torch.no_grad(): + # match index for pred and roll target + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + roll_global_point_.origin_coord, + roll_global_point_.offset, + ) + # teacher forward + roll_mask_target_sim = self.sinkhorn_knopp( + roll_global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + roll_mask_loss = -torch.sum( + roll_mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + roll_mask_loss = torch_scatter.segment_coo( + roll_mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["roll_mask_loss"] = roll_mask_loss + result_dict["loss"].append(roll_mask_loss * self.roll_mask_loss_weight) + if self.unmask_loss_weight > 0: + # student forward + local_point_ = self.student.backbone(local_point) + local_point_ = self.up_cast(local_point_) + unmask_pred_sim = self.student.unmask_head(local_point_.feat) + with torch.no_grad(): + principal_view_mask = global_point_.batch % self.num_global_view == 0 + principal_view_batch = ( + global_point_.batch[principal_view_mask] // self.num_global_view + ) + match_index = self.match_neighbour( + local_point_.origin_coord, + local_point_.offset[self.num_local_view - 1 :: self.num_local_view], + global_point_.origin_coord[principal_view_mask], + batch2offset(principal_view_batch), + ) + # teacher forward + unmask_target_sim = self.sinkhorn_knopp( + global_point_.feat[principal_view_mask][match_index[:, 1]], + self.teacher_temp, + ) + # loss + unmask_loss = -torch.sum( + unmask_target_sim + * F.log_softmax( + unmask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + unmask_loss = torch_scatter.segment_coo( + unmask_loss, + index=local_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["unmask_loss"] = unmask_loss + result_dict["loss"].append(unmask_loss * self.unmask_loss_weight) + result_dict["loss"] = sum(result_dict["loss"]) + + if get_world_size() > 1: + for loss in result_dict.values(): + dist.all_reduce(loss, op=dist.ReduceOp.AVG) + return result_dict diff --git a/pointcept/models/sparse_unet/__init__.py b/pointcept/models/sparse_unet/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..aafc3859f43bd47da8cd966a57d4307cc771525f --- /dev/null +++ b/pointcept/models/sparse_unet/__init__.py @@ -0,0 +1,4 @@ +from .mink_unet import * +from .spconv_unet_v1m1_base import * +from .spconv_unet_v1m2_bn_momentum import * +from .spconv_unet_v1m3_pdnorm import * diff --git a/pointcept/models/sparse_unet/mink_unet.py b/pointcept/models/sparse_unet/mink_unet.py new file mode 100644 index 0000000000000000000000000000000000000000..1ff8a01d0500abc207b82a8252a8131b5e671469 --- /dev/null +++ b/pointcept/models/sparse_unet/mink_unet.py @@ -0,0 +1,442 @@ +""" +SparseUNet Driven by MinkowskiEngine + +Modified from chrischoy/SpatioTemporalSegmentation + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +import torch.nn as nn + +try: + import MinkowskiEngine as ME +except ImportError: + ME = None + +from pointcept.models.builder import MODELS + + +def offset2batch(offset): + return ( + torch.cat( + [ + ( + torch.tensor([i] * (o - offset[i - 1])) + if i > 0 + else torch.tensor([i] * o) + ) + for i, o in enumerate(offset) + ], + dim=0, + ) + .long() + .to(offset.device) + ) + + +class BasicBlock(nn.Module): + expansion = 1 + + def __init__( + self, + inplanes, + planes, + stride=1, + dilation=1, + downsample=None, + bn_momentum=0.1, + dimension=-1, + ): + super(BasicBlock, self).__init__() + assert dimension > 0 + + self.conv1 = ME.MinkowskiConvolution( + inplanes, + planes, + kernel_size=3, + stride=stride, + dilation=dilation, + dimension=dimension, + ) + self.norm1 = ME.MinkowskiBatchNorm(planes, momentum=bn_momentum) + self.conv2 = ME.MinkowskiConvolution( + planes, + planes, + kernel_size=3, + stride=1, + dilation=dilation, + dimension=dimension, + ) + self.norm2 = ME.MinkowskiBatchNorm(planes, momentum=bn_momentum) + self.relu = ME.MinkowskiReLU(inplace=True) + self.downsample = downsample + + def forward(self, x): + residual = x + + out = self.conv1(x) + out = self.norm1(out) + out = self.relu(out) + + out = self.conv2(out) + out = self.norm2(out) + + if self.downsample is not None: + residual = self.downsample(x) + + out += residual + out = self.relu(out) + + return out + + +class Bottleneck(nn.Module): + expansion = 4 + + def __init__( + self, + inplanes, + planes, + stride=1, + dilation=1, + downsample=None, + bn_momentum=0.1, + dimension=-1, + ): + super(Bottleneck, self).__init__() + assert dimension > 0 + + self.conv1 = ME.MinkowskiConvolution( + inplanes, planes, kernel_size=1, dimension=dimension + ) + self.norm1 = ME.MinkowskiBatchNorm(planes, momentum=bn_momentum) + + self.conv2 = ME.MinkowskiConvolution( + planes, + planes, + kernel_size=3, + stride=stride, + dilation=dilation, + dimension=dimension, + ) + self.norm2 = ME.MinkowskiBatchNorm(planes, momentum=bn_momentum) + + self.conv3 = ME.MinkowskiConvolution( + planes, planes * self.expansion, kernel_size=1, dimension=dimension + ) + self.norm3 = ME.MinkowskiBatchNorm( + planes * self.expansion, momentum=bn_momentum + ) + + self.relu = ME.MinkowskiReLU(inplace=True) + self.downsample = downsample + + def forward(self, x): + residual = x + + out = self.conv1(x) + out = self.norm1(out) + out = self.relu(out) + + out = self.conv2(out) + out = self.norm2(out) + out = self.relu(out) + + out = self.conv3(out) + out = self.norm3(out) + + if self.downsample is not None: + residual = self.downsample(x) + + out += residual + out = self.relu(out) + + return out + + +class MinkUNetBase(nn.Module): + BLOCK = None + PLANES = None + DILATIONS = (1, 1, 1, 1, 1, 1, 1, 1) + LAYERS = (2, 2, 2, 2, 2, 2, 2, 2) + PLANES = (32, 64, 128, 256, 256, 128, 96, 96) + INIT_DIM = 32 + OUT_TENSOR_STRIDE = 1 + + def __init__(self, in_channels, out_channels, dimension=3): + super().__init__() + assert ME is not None, "Please follow `README.md` to install MinkowskiEngine.`" + self.D = dimension + assert self.BLOCK is not None + # Output of the first conv concated to conv6 + self.inplanes = self.INIT_DIM + self.conv0p1s1 = ME.MinkowskiConvolution( + in_channels, self.inplanes, kernel_size=5, dimension=self.D + ) + + self.bn0 = ME.MinkowskiBatchNorm(self.inplanes) + + self.conv1p1s2 = ME.MinkowskiConvolution( + self.inplanes, self.inplanes, kernel_size=2, stride=2, dimension=self.D + ) + self.bn1 = ME.MinkowskiBatchNorm(self.inplanes) + + self.block1 = self._make_layer(self.BLOCK, self.PLANES[0], self.LAYERS[0]) + + self.conv2p2s2 = ME.MinkowskiConvolution( + self.inplanes, self.inplanes, kernel_size=2, stride=2, dimension=self.D + ) + self.bn2 = ME.MinkowskiBatchNorm(self.inplanes) + + self.block2 = self._make_layer(self.BLOCK, self.PLANES[1], self.LAYERS[1]) + + self.conv3p4s2 = ME.MinkowskiConvolution( + self.inplanes, self.inplanes, kernel_size=2, stride=2, dimension=self.D + ) + + self.bn3 = ME.MinkowskiBatchNorm(self.inplanes) + self.block3 = self._make_layer(self.BLOCK, self.PLANES[2], self.LAYERS[2]) + + self.conv4p8s2 = ME.MinkowskiConvolution( + self.inplanes, self.inplanes, kernel_size=2, stride=2, dimension=self.D + ) + self.bn4 = ME.MinkowskiBatchNorm(self.inplanes) + self.block4 = self._make_layer(self.BLOCK, self.PLANES[3], self.LAYERS[3]) + + self.convtr4p16s2 = ME.MinkowskiConvolutionTranspose( + self.inplanes, self.PLANES[4], kernel_size=2, stride=2, dimension=self.D + ) + self.bntr4 = ME.MinkowskiBatchNorm(self.PLANES[4]) + + self.inplanes = self.PLANES[4] + self.PLANES[2] * self.BLOCK.expansion + self.block5 = self._make_layer(self.BLOCK, self.PLANES[4], self.LAYERS[4]) + self.convtr5p8s2 = ME.MinkowskiConvolutionTranspose( + self.inplanes, self.PLANES[5], kernel_size=2, stride=2, dimension=self.D + ) + self.bntr5 = ME.MinkowskiBatchNorm(self.PLANES[5]) + + self.inplanes = self.PLANES[5] + self.PLANES[1] * self.BLOCK.expansion + self.block6 = self._make_layer(self.BLOCK, self.PLANES[5], self.LAYERS[5]) + self.convtr6p4s2 = ME.MinkowskiConvolutionTranspose( + self.inplanes, self.PLANES[6], kernel_size=2, stride=2, dimension=self.D + ) + self.bntr6 = ME.MinkowskiBatchNorm(self.PLANES[6]) + + self.inplanes = self.PLANES[6] + self.PLANES[0] * self.BLOCK.expansion + self.block7 = self._make_layer(self.BLOCK, self.PLANES[6], self.LAYERS[6]) + self.convtr7p2s2 = ME.MinkowskiConvolutionTranspose( + self.inplanes, self.PLANES[7], kernel_size=2, stride=2, dimension=self.D + ) + self.bntr7 = ME.MinkowskiBatchNorm(self.PLANES[7]) + + self.inplanes = self.PLANES[7] + self.INIT_DIM + self.block8 = self._make_layer(self.BLOCK, self.PLANES[7], self.LAYERS[7]) + + self.final = ME.MinkowskiConvolution( + self.PLANES[7] * self.BLOCK.expansion, + out_channels, + kernel_size=1, + bias=True, + dimension=self.D, + ) + self.relu = ME.MinkowskiReLU(inplace=True) + + self.weight_initialization() + + def weight_initialization(self): + for m in self.modules(): + if isinstance(m, ME.MinkowskiConvolution): + ME.utils.kaiming_normal_(m.kernel, mode="fan_out", nonlinearity="relu") + + if isinstance(m, ME.MinkowskiBatchNorm): + nn.init.constant_(m.bn.weight, 1) + nn.init.constant_(m.bn.bias, 0) + + def _make_layer(self, block, planes, blocks, stride=1, dilation=1, bn_momentum=0.1): + downsample = None + if stride != 1 or self.inplanes != planes * block.expansion: + downsample = nn.Sequential( + ME.MinkowskiConvolution( + self.inplanes, + planes * block.expansion, + kernel_size=1, + stride=stride, + dimension=self.D, + ), + ME.MinkowskiBatchNorm(planes * block.expansion), + ) + layers = [] + layers.append( + block( + self.inplanes, + planes, + stride=stride, + dilation=dilation, + downsample=downsample, + dimension=self.D, + ) + ) + self.inplanes = planes * block.expansion + for i in range(1, blocks): + layers.append( + block( + self.inplanes, planes, stride=1, dilation=dilation, dimension=self.D + ) + ) + + return nn.Sequential(*layers) + + def forward(self, data_dict): + grid_coord = data_dict["grid_coord"] + feat = data_dict["feat"] + offset = data_dict["offset"] + batch = offset2batch(offset) + in_field = ME.TensorField( + feat, + coordinates=torch.cat([batch.unsqueeze(-1).int(), grid_coord.int()], dim=1), + quantization_mode=ME.SparseTensorQuantizationMode.UNWEIGHTED_AVERAGE, + minkowski_algorithm=ME.MinkowskiAlgorithm.SPEED_OPTIMIZED, + device=feat.device, + ) + x = in_field.sparse() + + out = self.conv0p1s1(x) + out = self.bn0(out) + out_p1 = self.relu(out) + + out = self.conv1p1s2(out_p1) + out = self.bn1(out) + out = self.relu(out) + out_b1p2 = self.block1(out) + + out = self.conv2p2s2(out_b1p2) + out = self.bn2(out) + out = self.relu(out) + out_b2p4 = self.block2(out) + + out = self.conv3p4s2(out_b2p4) + out = self.bn3(out) + out = self.relu(out) + out_b3p8 = self.block3(out) + + # tensor_stride=16 + out = self.conv4p8s2(out_b3p8) + out = self.bn4(out) + out = self.relu(out) + out = self.block4(out) + + # tensor_stride=8 + out = self.convtr4p16s2(out) + out = self.bntr4(out) + out = self.relu(out) + + out = ME.cat(out, out_b3p8) + out = self.block5(out) + + # tensor_stride=4 + out = self.convtr5p8s2(out) + out = self.bntr5(out) + out = self.relu(out) + + out = ME.cat(out, out_b2p4) + out = self.block6(out) + + # tensor_stride=2 + out = self.convtr6p4s2(out) + out = self.bntr6(out) + out = self.relu(out) + + out = ME.cat(out, out_b1p2) + out = self.block7(out) + + # tensor_stride=1 + out = self.convtr7p2s2(out) + out = self.bntr7(out) + out = self.relu(out) + + out = ME.cat(out, out_p1) + out = self.block8(out) + + return self.final(out).slice(in_field).F + + +@MODELS.register_module() +class MinkUNet14(MinkUNetBase): + BLOCK = BasicBlock + LAYERS = (1, 1, 1, 1, 1, 1, 1, 1) + + +@MODELS.register_module() +class MinkUNet18(MinkUNetBase): + BLOCK = BasicBlock + LAYERS = (2, 2, 2, 2, 2, 2, 2, 2) + + +@MODELS.register_module() +class MinkUNet34(MinkUNetBase): + BLOCK = BasicBlock + LAYERS = (2, 3, 4, 6, 2, 2, 2, 2) + + +@MODELS.register_module() +class MinkUNet50(MinkUNetBase): + BLOCK = Bottleneck + LAYERS = (2, 3, 4, 6, 2, 2, 2, 2) + + +@MODELS.register_module() +class MinkUNet101(MinkUNetBase): + BLOCK = Bottleneck + LAYERS = (2, 3, 4, 23, 2, 2, 2, 2) + + +@MODELS.register_module() +class MinkUNet14A(MinkUNet14): + PLANES = (32, 64, 128, 256, 128, 128, 96, 96) + + +@MODELS.register_module() +class MinkUNet14B(MinkUNet14): + PLANES = (32, 64, 128, 256, 128, 128, 128, 128) + + +@MODELS.register_module() +class MinkUNet14C(MinkUNet14): + PLANES = (32, 64, 128, 256, 192, 192, 128, 128) + + +@MODELS.register_module() +class MinkUNet14D(MinkUNet14): + PLANES = (32, 64, 128, 256, 384, 384, 384, 384) + + +@MODELS.register_module() +class MinkUNet18A(MinkUNet18): + PLANES = (32, 64, 128, 256, 128, 128, 96, 96) + + +@MODELS.register_module() +class MinkUNet18B(MinkUNet18): + PLANES = (32, 64, 128, 256, 128, 128, 128, 128) + + +@MODELS.register_module() +class MinkUNet18D(MinkUNet18): + PLANES = (32, 64, 128, 256, 384, 384, 384, 384) + + +@MODELS.register_module() +class MinkUNet34A(MinkUNet34): + PLANES = (32, 64, 128, 256, 256, 128, 96, 96) + + +@MODELS.register_module() +class MinkUNet34B(MinkUNet34): + PLANES = (32, 64, 128, 256, 256, 128, 64, 32) + + +@MODELS.register_module() +class MinkUNet34C(MinkUNet34): + PLANES = (32, 64, 128, 256, 256, 128, 96, 96) diff --git a/pointcept/models/sparse_unet/spconv_unet_v1m1_base.py b/pointcept/models/sparse_unet/spconv_unet_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..25bcfe260ec17ed71aa62565739d343dd304f9c7 --- /dev/null +++ b/pointcept/models/sparse_unet/spconv_unet_v1m1_base.py @@ -0,0 +1,463 @@ +""" +SparseUNet Driven by SpConv (recommend) + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from collections import OrderedDict + +import torch +import torch.nn as nn + +import spconv.pytorch as spconv +from torch_geometric.utils import scatter + +from timm.layers import trunc_normal_ + +from pointcept.models.builder import MODELS +from pointcept.models.utils import offset2batch + + +class BasicBlock(spconv.SparseModule): + expansion = 1 + + def __init__( + self, + in_channels, + embed_channels, + stride=1, + norm_fn=None, + indice_key=None, + bias=False, + ): + super().__init__() + + assert norm_fn is not None + + if in_channels == embed_channels: + self.proj = spconv.SparseSequential(nn.Identity()) + else: + self.proj = spconv.SparseSequential( + spconv.SubMConv3d( + in_channels, embed_channels, kernel_size=1, bias=False + ), + norm_fn(embed_channels), + ) + + self.conv1 = spconv.SubMConv3d( + in_channels, + embed_channels, + kernel_size=3, + stride=stride, + padding=1, + bias=bias, + indice_key=indice_key, + ) + self.bn1 = norm_fn(embed_channels) + self.relu = nn.ReLU() + self.conv2 = spconv.SubMConv3d( + embed_channels, + embed_channels, + kernel_size=3, + stride=stride, + padding=1, + bias=bias, + indice_key=indice_key, + ) + self.bn2 = norm_fn(embed_channels) + self.stride = stride + + def forward(self, x): + residual = x + + out = self.conv1(x) + out = out.replace_feature(self.bn1(out.features)) + out = out.replace_feature(self.relu(out.features)) + + out = self.conv2(out) + out = out.replace_feature(self.bn2(out.features)) + + out = out.replace_feature(out.features + self.proj(residual).features) + out = out.replace_feature(self.relu(out.features)) + + return out + + +@MODELS.register_module("SpUNet-v1m1") +class SpUNetBase(nn.Module): + def __init__( + self, + in_channels, + num_classes, + base_channels=32, + channels=(32, 64, 128, 256, 256, 128, 96, 96), + layers=(2, 3, 4, 6, 2, 2, 2, 2), + enc_mode=False, + ): + super().__init__() + assert len(layers) % 2 == 0 + assert len(layers) == len(channels) + self.in_channels = in_channels + self.num_classes = num_classes + self.base_channels = base_channels + self.channels = channels + self.layers = layers + self.num_stages = len(layers) // 2 + self.enc_mode = enc_mode + + norm_fn = partial(nn.BatchNorm1d, eps=1e-3, momentum=0.01) + block = BasicBlock + + self.conv_input = spconv.SparseSequential( + spconv.SubMConv3d( + in_channels, + base_channels, + kernel_size=5, + padding=1, + bias=False, + indice_key="stem", + ), + norm_fn(base_channels), + nn.ReLU(), + ) + + enc_channels = base_channels + dec_channels = channels[-1] + self.down = nn.ModuleList() + self.up = nn.ModuleList() + self.enc = nn.ModuleList() + self.dec = nn.ModuleList() if not self.enc_mode else None + + for s in range(self.num_stages): + # encode num_stages + self.down.append( + spconv.SparseSequential( + spconv.SparseConv3d( + enc_channels, + channels[s], + kernel_size=2, + stride=2, + bias=False, + indice_key=f"spconv{s + 1}", + ), + norm_fn(channels[s]), + nn.ReLU(), + ) + ) + self.enc.append( + spconv.SparseSequential( + OrderedDict( + [ + # (f"block{i}", block(enc_channels, channels[s], norm_fn=norm_fn, indice_key=f"subm{s + 1}")) + # if i == 0 else + ( + f"block{i}", + block( + channels[s], + channels[s], + norm_fn=norm_fn, + indice_key=f"subm{s + 1}", + ), + ) + for i in range(layers[s]) + ] + ) + ) + ) + if not self.enc_mode: + # decode num_stages + self.up.append( + spconv.SparseSequential( + spconv.SparseInverseConv3d( + channels[len(channels) - s - 2], + dec_channels, + kernel_size=2, + bias=False, + indice_key=f"spconv{s + 1}", + ), + norm_fn(dec_channels), + nn.ReLU(), + ) + ) + self.dec.append( + spconv.SparseSequential( + OrderedDict( + [ + ( + ( + f"block{i}", + block( + dec_channels + enc_channels, + dec_channels, + norm_fn=norm_fn, + indice_key=f"subm{s}", + ), + ) + if i == 0 + else ( + f"block{i}", + block( + dec_channels, + dec_channels, + norm_fn=norm_fn, + indice_key=f"subm{s}", + ), + ) + ) + for i in range(layers[len(channels) - s - 1]) + ] + ) + ) + ) + + enc_channels = channels[s] + dec_channels = channels[len(channels) - s - 2] + + final_in_channels = ( + channels[-1] if not self.enc_mode else channels[self.num_stages - 1] + ) + self.final = ( + spconv.SubMConv3d( + final_in_channels, num_classes, kernel_size=1, padding=1, bias=True + ) + if num_classes > 0 + else spconv.Identity() + ) + self.apply(self._init_weights) + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, spconv.SubMConv3d): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.BatchNorm1d): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + def forward(self, input_dict): + grid_coord = input_dict["grid_coord"] + feat = input_dict["feat"] + offset = input_dict["offset"] + + batch = offset2batch(offset) + sparse_shape = torch.add(torch.max(grid_coord, dim=0).values, 96).tolist() + x = spconv.SparseConvTensor( + features=feat, + indices=torch.cat( + [batch.unsqueeze(-1).int(), grid_coord.int()], dim=1 + ).contiguous(), + spatial_shape=sparse_shape, + batch_size=batch[-1].tolist() + 1, + ) + x = self.conv_input(x) + skips = [x] + # enc forward + for s in range(self.num_stages): + x = self.down[s](x) + x = self.enc[s](x) + skips.append(x) + x = skips.pop(-1) + if not self.enc_mode: + # dec forward + for s in reversed(range(self.num_stages)): + x = self.up[s](x) + skip = skips.pop(-1) + x = x.replace_feature(torch.cat((x.features, skip.features), dim=1)) + x = self.dec[s](x) + + x = self.final(x) + if self.enc_mode: + x = x.replace_feature( + scatter(x.features, x.indices[:, 0].long(), reduce="mean", dim=0) + ) + return x.features + + +@MODELS.register_module() +class SpUNetNoSkipBase(nn.Module): + def __init__( + self, + in_channels, + out_channels, + base_channels=32, + channels=(32, 64, 128, 256, 256, 128, 96, 96), + layers=(2, 3, 4, 6, 2, 2, 2, 2), + ): + super().__init__() + assert len(layers) % 2 == 0 + assert len(layers) == len(channels) + self.in_channels = in_channels + self.out_channels = out_channels + self.base_channels = base_channels + self.channels = channels + self.layers = layers + self.num_stages = len(layers) // 2 + + norm_fn = partial(nn.BatchNorm1d, eps=1e-3, momentum=0.01) + block = BasicBlock + + self.conv_input = spconv.SparseSequential( + spconv.SubMConv3d( + in_channels, + base_channels, + kernel_size=5, + padding=1, + bias=False, + indice_key="stem", + ), + norm_fn(base_channels), + nn.ReLU(), + ) + + enc_channels = base_channels + dec_channels = channels[-1] + self.down = nn.ModuleList() + self.up = nn.ModuleList() + self.enc = nn.ModuleList() + self.dec = nn.ModuleList() + + for s in range(self.num_stages): + # encode num_stages + self.down.append( + spconv.SparseSequential( + spconv.SparseConv3d( + enc_channels, + channels[s], + kernel_size=2, + stride=2, + bias=False, + indice_key=f"spconv{s + 1}", + ), + norm_fn(channels[s]), + nn.ReLU(), + ) + ) + self.enc.append( + spconv.SparseSequential( + OrderedDict( + [ + # (f"block{i}", block(enc_channels, channels[s], norm_fn=norm_fn, indice_key=f"subm{s + 1}")) + # if i == 0 else + ( + f"block{i}", + block( + channels[s], + channels[s], + norm_fn=norm_fn, + indice_key=f"subm{s + 1}", + ), + ) + for i in range(layers[s]) + ] + ) + ) + ) + + # decode num_stages + self.up.append( + spconv.SparseSequential( + spconv.SparseInverseConv3d( + channels[len(channels) - s - 2], + dec_channels, + kernel_size=2, + bias=False, + indice_key=f"spconv{s + 1}", + ), + norm_fn(dec_channels), + nn.ReLU(), + ) + ) + self.dec.append( + spconv.SparseSequential( + OrderedDict( + [ + ( + ( + f"block{i}", + block( + dec_channels, + dec_channels, + norm_fn=norm_fn, + indice_key=f"subm{s}", + ), + ) + if i == 0 + else ( + f"block{i}", + block( + dec_channels, + dec_channels, + norm_fn=norm_fn, + indice_key=f"subm{s}", + ), + ) + ) + for i in range(layers[len(channels) - s - 1]) + ] + ) + ) + ) + enc_channels = channels[s] + dec_channels = channels[len(channels) - s - 2] + + self.final = ( + spconv.SubMConv3d( + channels[-1], out_channels, kernel_size=1, padding=1, bias=True + ) + if out_channels > 0 + else spconv.Identity() + ) + self.apply(self._init_weights) + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, spconv.SubMConv3d): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.BatchNorm1d): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + def forward(self, data_dict): + grid_coord = data_dict["grid_coord"] + feat = data_dict["feat"] + offset = data_dict["offset"] + batch = offset2batch(offset) + sparse_shape = torch.add(torch.max(grid_coord, dim=0).values, 1).tolist() + x = spconv.SparseConvTensor( + features=feat, + indices=torch.cat( + [batch.unsqueeze(-1).int(), grid_coord.int()], dim=1 + ).contiguous(), + spatial_shape=sparse_shape, + batch_size=batch[-1].tolist() + 1, + ) + x = self.conv_input(x) + skips = [x] + # enc forward + for s in range(self.num_stages): + x = self.down[s](x) + x = self.enc[s](x) + skips.append(x) + x = skips.pop(-1) + # dec forward + for s in reversed(range(self.num_stages)): + x = self.up[s](x) + # skip = skips.pop(-1) + # x = x.replace_feature(torch.cat((x.features, skip.features), dim=1)) + x = self.dec[s](x) + + x = self.final(x) + return x.features diff --git a/pointcept/models/sparse_unet/spconv_unet_v1m2_bn_momentum.py b/pointcept/models/sparse_unet/spconv_unet_v1m2_bn_momentum.py new file mode 100644 index 0000000000000000000000000000000000000000..375915a40537f2356c89dd711c4e026d4ccdebd7 --- /dev/null +++ b/pointcept/models/sparse_unet/spconv_unet_v1m2_bn_momentum.py @@ -0,0 +1,290 @@ +""" +SparseUNet Driven by SpConv (recommend) + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from collections import OrderedDict + +import torch +import torch.nn as nn + +try: + import spconv.pytorch as spconv +except ImportError: + import warnings + + warnings.warn("Please follow `README.md` to install spconv2.`") + +from timm.layers import trunc_normal_ +from pointcept.models.builder import MODELS + + +def offset2batch(offset): + return ( + torch.cat( + [ + ( + torch.tensor([i] * (o - offset[i - 1])) + if i > 0 + else torch.tensor([i] * o) + ) + for i, o in enumerate(offset) + ], + dim=0, + ) + .long() + .to(offset.device) + ) + + +class BasicBlock(spconv.SparseModule): + expansion = 1 + + def __init__( + self, + in_channels, + embed_channels, + stride=1, + norm_fn=None, + indice_key=None, + bias=False, + ): + super().__init__() + + assert norm_fn is not None + + if in_channels == embed_channels: + self.proj = spconv.SparseSequential(nn.Identity()) + else: + self.proj = spconv.SparseSequential( + spconv.SubMConv3d( + in_channels, embed_channels, kernel_size=1, bias=False + ), + norm_fn(embed_channels, momentum=0.02), + ) + + self.conv1 = spconv.SubMConv3d( + in_channels, + embed_channels, + kernel_size=3, + stride=stride, + padding=1, + bias=bias, + indice_key=indice_key, + ) + self.bn1 = norm_fn(embed_channels) + self.relu = nn.ReLU() + self.conv2 = spconv.SubMConv3d( + embed_channels, + embed_channels, + kernel_size=3, + stride=stride, + padding=1, + bias=bias, + indice_key=indice_key, + ) + self.bn2 = norm_fn(embed_channels) + self.stride = stride + + def forward(self, x): + residual = x + + out = self.conv1(x) + out = out.replace_feature(self.bn1(out.features)) + out = out.replace_feature(self.relu(out.features)) + + out = self.conv2(out) + out = out.replace_feature(self.bn2(out.features)) + + out = out.replace_feature(out.features + self.proj(residual).features) + out = out.replace_feature(self.relu(out.features)) + + return out + + +@MODELS.register_module("SpUNet-v1m2") +class SpUNetBase(nn.Module): + def __init__( + self, + in_channels, + num_classes, + base_channels=32, + channels=(32, 64, 128, 256, 256, 128, 96, 96), + layers=(2, 3, 4, 6, 2, 2, 2, 2), + bn_momentum=0.1, + ): + super().__init__() + assert len(layers) % 2 == 0 + assert len(layers) == len(channels) + self.in_channels = in_channels + self.num_classes = num_classes + self.base_channels = base_channels + self.channels = channels + self.layers = layers + self.num_stages = len(layers) // 2 + + norm_fn = partial(nn.BatchNorm1d, eps=1e-5, momentum=bn_momentum) + block = BasicBlock + + self.conv_input = spconv.SparseSequential( + spconv.SubMConv3d( + in_channels, + base_channels, + kernel_size=5, + padding=1, + bias=False, + indice_key="stem", + ), + norm_fn(base_channels, momentum=0.02), + nn.ReLU(), + ) + + enc_channels = base_channels + dec_channels = channels[-1] + self.down = nn.ModuleList() + self.up = nn.ModuleList() + self.enc = nn.ModuleList() + self.dec = nn.ModuleList() + + for s in range(self.num_stages): + # encode num_stages + self.down.append( + spconv.SparseSequential( + spconv.SparseConv3d( + enc_channels, + channels[s], + kernel_size=2, + stride=2, + bias=False, + indice_key=f"spconv{s + 1}", + ), + norm_fn(channels[s], momentum=0.02), + nn.ReLU(), + ) + ) + self.enc.append( + spconv.SparseSequential( + OrderedDict( + [ + # (f"block{i}", block(enc_channels, channels[s], norm_fn=norm_fn, indice_key=f"subm{s + 1}")) + # if i == 0 else + ( + f"block{i}", + block( + channels[s], + channels[s], + norm_fn=norm_fn, + indice_key=f"subm{s + 1}", + ), + ) + for i in range(layers[s]) + ] + ) + ) + ) + + # decode num_stages + self.up.append( + spconv.SparseSequential( + spconv.SparseInverseConv3d( + channels[len(channels) - s - 2], + dec_channels, + kernel_size=2, + bias=False, + indice_key=f"spconv{s + 1}", + ), + norm_fn(dec_channels, momentum=0.02), + nn.ReLU(), + ) + ) + self.dec.append( + spconv.SparseSequential( + OrderedDict( + [ + ( + ( + f"block{i}", + block( + dec_channels + enc_channels, + dec_channels, + norm_fn=norm_fn, + indice_key=f"subm{s}", + ), + ) + if i == 0 + else ( + f"block{i}", + block( + dec_channels, + dec_channels, + norm_fn=norm_fn, + indice_key=f"subm{s}", + ), + ) + ) + for i in range(layers[len(channels) - s - 1]) + ] + ) + ) + ) + enc_channels = channels[s] + dec_channels = channels[len(channels) - s - 2] + + self.final = ( + spconv.SubMConv3d( + channels[-1], num_classes, kernel_size=1, padding=1, bias=True + ) + if num_classes > 0 + else spconv.Identity() + ) + self.apply(self._init_weights) + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, spconv.SubMConv3d): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.BatchNorm1d): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + def forward(self, data_dict): + grid_coord = data_dict["grid_coord"] + feat = data_dict["feat"] + offset = data_dict["offset"] + + batch = offset2batch(offset) + sparse_shape = torch.add(torch.max(grid_coord, dim=0).values, 1).tolist() + x = spconv.SparseConvTensor( + features=feat, + indices=torch.cat( + [batch.unsqueeze(-1).int(), grid_coord.int()], dim=1 + ).contiguous(), + spatial_shape=sparse_shape, + batch_size=batch[-1].tolist() + 1, + ) + x = self.conv_input(x) + skips = [x] + # enc forward + for s in range(self.num_stages): + x = self.down[s](x) + x = self.enc[s](x) + skips.append(x) + x = skips.pop(-1) + # dec forward + for s in reversed(range(self.num_stages)): + x = self.up[s](x) + skip = skips.pop(-1) + x = x.replace_feature(torch.cat((x.features, skip.features), dim=1)) + x = self.dec[s](x) + + x = self.final(x) + return x.features diff --git a/pointcept/models/sparse_unet/spconv_unet_v1m3_pdnorm.py b/pointcept/models/sparse_unet/spconv_unet_v1m3_pdnorm.py new file mode 100644 index 0000000000000000000000000000000000000000..970a9eadce0196cacf99fc1fdaac66f2c0264d72 --- /dev/null +++ b/pointcept/models/sparse_unet/spconv_unet_v1m3_pdnorm.py @@ -0,0 +1,432 @@ +""" +SparseUNet V1M3 + +Enable Prompt-Driven Normalization for Point Prompt Training + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from functools import partial +from collections import OrderedDict + +import torch +import torch.nn as nn + +import spconv.pytorch as spconv +from torch_geometric.utils import scatter + +from timm.layers import trunc_normal_ + +from pointcept.models.builder import MODELS +from pointcept.models.utils import offset2batch + + +class PDBatchNorm(torch.nn.Module): + def __init__( + self, + num_features, + context_channels=256, + eps=1e-3, + momentum=0.01, + conditions=("ScanNet", "S3DIS", "Structured3D"), + decouple=True, + adaptive=False, + affine=True, + ): + super().__init__() + self.conditions = conditions + self.decouple = decouple + self.adaptive = adaptive + self.affine = affine + if self.decouple: + self.bns = nn.ModuleList( + [ + nn.BatchNorm1d( + num_features=num_features, + eps=eps, + momentum=momentum, + affine=affine, + ) + for _ in conditions + ] + ) + else: + self.bn = nn.BatchNorm1d( + num_features=num_features, eps=eps, momentum=momentum, affine=affine + ) + if self.adaptive: + self.modulation = nn.Sequential( + nn.SiLU(), nn.Linear(context_channels, 2 * num_features, bias=True) + ) + + def forward(self, feat, condition=None, context=None): + if self.decouple: + assert condition in self.conditions + bn = self.bns[self.conditions.index(condition)] + else: + bn = self.bn + feat = bn(feat) + if self.adaptive: + assert context is not None + shift, scale = self.modulation(context).chunk(2, dim=1) + feat = feat * (1.0 + scale) + shift + return feat + + +class BasicBlock(spconv.SparseModule): + expansion = 1 + + def __init__( + self, + in_channels, + embed_channels, + stride=1, + norm_fn=None, + indice_key=None, + bias=False, + ): + super().__init__() + + assert norm_fn is not None + + self.in_channels = in_channels + self.embed_channels = embed_channels + if in_channels == embed_channels: + self.proj = spconv.SparseSequential(nn.Identity()) + else: + # TODO remove norm after project + self.proj_conv = spconv.SubMConv3d( + in_channels, embed_channels, kernel_size=1, bias=False + ) + self.proj_norm = norm_fn(embed_channels) + + self.conv1 = spconv.SubMConv3d( + in_channels, + embed_channels, + kernel_size=3, + stride=stride, + padding=1, + bias=bias, + indice_key=indice_key, + ) + self.bn1 = norm_fn(embed_channels) + self.relu = nn.ReLU() + self.conv2 = spconv.SubMConv3d( + embed_channels, + embed_channels, + kernel_size=3, + stride=stride, + padding=1, + bias=bias, + indice_key=indice_key, + ) + self.bn2 = norm_fn(embed_channels) + self.stride = stride + + def forward(self, x): + x, condition, context = x + residual = x + + out = self.conv1(x) + out = out.replace_feature(self.bn1(out.features, condition, context)) + out = out.replace_feature(self.relu(out.features)) + + out = self.conv2(out) + out = out.replace_feature(self.bn2(out.features, condition, context)) + + if self.in_channels == self.embed_channels: + residual = self.proj(residual) + else: + residual = residual.replace_feature( + self.proj_norm(self.proj_conv(residual).features, condition, context) + ) + out = out.replace_feature(out.features + residual.features) + out = out.replace_feature(self.relu(out.features)) + return out, condition, context + + +class SPConvDown(nn.Module): + def __init__( + self, + in_channels, + out_channels, + indice_key, + kernel_size=2, + bias=False, + norm_fn=None, + ): + super().__init__() + self.conv = spconv.SparseConv3d( + in_channels, + out_channels, + kernel_size=kernel_size, + stride=kernel_size, + bias=bias, + indice_key=indice_key, + ) + self.bn = norm_fn(out_channels) + self.relu = nn.ReLU() + + def forward(self, x): + x, condition, context = x + out = self.conv(x) + out = out.replace_feature(self.bn(out.features, condition, context)) + out = out.replace_feature(self.relu(out.features)) + return out + + +class SPConvUp(nn.Module): + def __init__( + self, + in_channels, + out_channels, + indice_key, + kernel_size=2, + bias=False, + norm_fn=None, + ): + super().__init__() + self.conv = spconv.SparseInverseConv3d( + in_channels, + out_channels, + kernel_size=kernel_size, + bias=bias, + indice_key=indice_key, + ) + self.bn = norm_fn(out_channels) + self.relu = nn.ReLU() + + def forward(self, x): + x, condition, context = x + out = self.conv(x) + out = out.replace_feature(self.bn(out.features, condition, context)) + out = out.replace_feature(self.relu(out.features)) + return out + + +class SPConvPatchEmbedding(nn.Module): + def __init__(self, in_channels, out_channels, kernel_size=5, norm_fn=None): + super().__init__() + self.conv = spconv.SubMConv3d( + in_channels, + out_channels, + kernel_size=kernel_size, + padding=1, + bias=False, + indice_key="stem", + ) + self.bn = norm_fn(out_channels) + self.relu = nn.ReLU() + + def forward(self, x): + x, condition, context = x + out = self.conv(x) + out = out.replace_feature(self.bn(out.features, condition, context)) + out = out.replace_feature(self.relu(out.features)) + return out + + +@MODELS.register_module("SpUNet-v1m3") +class SpUNetBase(nn.Module): + def __init__( + self, + in_channels, + num_classes=0, + base_channels=32, + context_channels=256, + channels=(32, 64, 128, 256, 256, 128, 96, 96), + layers=(2, 3, 4, 6, 2, 2, 2, 2), + enc_mode=False, + conditions=("ScanNet", "S3DIS", "Structured3D"), + zero_init=True, + norm_decouple=True, + norm_adaptive=True, + norm_affine=False, + ): + super().__init__() + assert len(layers) % 2 == 0 + assert len(layers) == len(channels) + self.in_channels = in_channels + self.num_classes = num_classes + self.base_channels = base_channels + self.channels = channels + self.layers = layers + self.num_stages = len(layers) // 2 + self.enc_mode = enc_mode + self.conditions = conditions + self.zero_init = zero_init + + norm_fn = partial( + PDBatchNorm, + eps=1e-3, + momentum=0.01, + conditions=conditions, + context_channels=context_channels, + decouple=norm_decouple, + adaptive=norm_adaptive, + affine=norm_affine, + ) + block = BasicBlock + + self.conv_input = SPConvPatchEmbedding( + in_channels, base_channels, kernel_size=5, norm_fn=norm_fn + ) + + enc_channels = base_channels + dec_channels = channels[-1] + self.down = nn.ModuleList() + self.up = nn.ModuleList() + self.enc = nn.ModuleList() + self.dec = nn.ModuleList() if not self.enc_mode else None + + for s in range(self.num_stages): + # encode num_stages + self.down.append( + SPConvDown( + enc_channels, + channels[s], + kernel_size=2, + bias=False, + indice_key=f"spconv{s + 1}", + norm_fn=norm_fn, + ) + ) + self.enc.append( + spconv.SparseSequential( + OrderedDict( + [ + # (f"block{i}", block(enc_channels, channels[s], norm_fn=norm_fn, indice_key=f"subm{s + 1}")) + # if i == 0 else + ( + f"block{i}", + block( + channels[s], + channels[s], + norm_fn=norm_fn, + indice_key=f"subm{s + 1}", + ), + ) + for i in range(layers[s]) + ] + ) + ) + ) + if not self.enc_mode: + # decode num_stages + self.up.append( + SPConvUp( + channels[len(channels) - s - 2], + dec_channels, + kernel_size=2, + bias=False, + indice_key=f"spconv{s + 1}", + norm_fn=norm_fn, + ) + ) + self.dec.append( + spconv.SparseSequential( + OrderedDict( + [ + ( + ( + f"block{i}", + block( + dec_channels + enc_channels, + dec_channels, + norm_fn=norm_fn, + indice_key=f"subm{s}", + ), + ) + if i == 0 + else ( + f"block{i}", + block( + dec_channels, + dec_channels, + norm_fn=norm_fn, + indice_key=f"subm{s}", + ), + ) + ) + for i in range(layers[len(channels) - s - 1]) + ] + ) + ) + ) + + enc_channels = channels[s] + dec_channels = channels[len(channels) - s - 2] + + final_in_channels = ( + channels[-1] if not self.enc_mode else channels[self.num_stages - 1] + ) + self.final = ( + spconv.SubMConv3d( + final_in_channels, num_classes, kernel_size=1, padding=1, bias=True + ) + if num_classes > 0 + else spconv.Identity() + ) + self.apply(self._init_weights) + + def _init_weights(self, m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, spconv.SubMConv3d): + trunc_normal_(m.weight, std=0.02) + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.BatchNorm1d): + if m.affine: + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + elif isinstance(m, PDBatchNorm): + if self.zero_init: + nn.init.constant_(m.modulation[-1].weight, 0) + nn.init.constant_(m.modulation[-1].bias, 0) + + def forward(self, input_dict): + grid_coord = input_dict["grid_coord"] + feat = input_dict["feat"] + offset = input_dict["offset"] + if isinstance(input_dict["condition"], list): + condition = input_dict["condition"][0] + else: + condition = input_dict["condition"] + context = input_dict["context"] if "context" in input_dict.keys() else None + + batch = offset2batch(offset) + sparse_shape = torch.add(torch.max(grid_coord, dim=0).values, 96).tolist() + x = spconv.SparseConvTensor( + features=feat, + indices=torch.cat( + [batch.unsqueeze(-1).int(), grid_coord.int()], dim=1 + ).contiguous(), + spatial_shape=sparse_shape, + batch_size=batch[-1].tolist() + 1, + ) + x = self.conv_input([x, condition, context]) + skips = [x] + # enc forward + for s in range(self.num_stages): + x = self.down[s]([x, condition, context]) + x, _, _ = self.enc[s]([x, condition, context]) + skips.append(x) + x = skips.pop(-1) + if not self.enc_mode: + # dec forward + for s in reversed(range(self.num_stages)): + x = self.up[s]([x, condition, context]) + skip = skips.pop(-1) + x = x.replace_feature(torch.cat((x.features, skip.features), dim=1)) + x, _, _ = self.dec[s]([x, condition, context]) + + x = self.final(x) + if self.enc_mode: + x = x.replace_feature( + scatter(x.features, x.indices[:, 0].long(), reduce="mean", dim=0) + ) + return x.features diff --git a/pointcept/models/spvcnn/__init__.py b/pointcept/models/spvcnn/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..ecdc75a6878026437124c187323ca9676bf35c76 --- /dev/null +++ b/pointcept/models/spvcnn/__init__.py @@ -0,0 +1 @@ +from .ts_spvcnn import * diff --git a/pointcept/models/spvcnn/ts_spvcnn.py b/pointcept/models/spvcnn/ts_spvcnn.py new file mode 100644 index 0000000000000000000000000000000000000000..c26f1ea8d41a8edfbd1542b0f9e607e660441b38 --- /dev/null +++ b/pointcept/models/spvcnn/ts_spvcnn.py @@ -0,0 +1,438 @@ +""" +SPVCNN + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +import torch.nn as nn + +try: + import torchsparse + import torchsparse.nn as spnn + import torchsparse.nn.functional as F + from torchsparse.nn.utils import get_kernel_offsets + from torchsparse import PointTensor, SparseTensor +except ImportError: + torchsparse = None + + +from pointcept.models.utils import offset2batch +from pointcept.models.builder import MODELS + + +def initial_voxelize(z): + pc_hash = F.sphash(torch.floor(z.C).int()) + sparse_hash = torch.unique(pc_hash) + idx_query = F.sphashquery(pc_hash, sparse_hash) + counts = F.spcount(idx_query.int(), len(sparse_hash)) + + inserted_coords = F.spvoxelize(torch.floor(z.C), idx_query, counts) + inserted_coords = torch.round(inserted_coords).int() + inserted_feat = F.spvoxelize(z.F, idx_query, counts) + + new_tensor = SparseTensor(inserted_feat, inserted_coords, 1) + new_tensor.cmaps.setdefault(new_tensor.stride, new_tensor.coords) + z.additional_features["idx_query"][1] = idx_query + z.additional_features["counts"][1] = counts + return new_tensor + + +# x: SparseTensor, z: PointTensor +# return: SparseTensor +def point_to_voxel(x, z): + if ( + z.additional_features is None + or z.additional_features.get("idx_query") is None + or z.additional_features["idx_query"].get(x.s) is None + ): + pc_hash = F.sphash( + torch.cat( + [ + torch.floor(z.C[:, :3] / x.s[0]).int() * x.s[0], + z.C[:, -1].int().view(-1, 1), + ], + 1, + ) + ) + sparse_hash = F.sphash(x.C) + idx_query = F.sphashquery(pc_hash, sparse_hash) + counts = F.spcount(idx_query.int(), x.C.shape[0]) + z.additional_features["idx_query"][x.s] = idx_query + z.additional_features["counts"][x.s] = counts + else: + idx_query = z.additional_features["idx_query"][x.s] + counts = z.additional_features["counts"][x.s] + + inserted_feat = F.spvoxelize(z.F, idx_query, counts) + new_tensor = SparseTensor(inserted_feat, x.C, x.s) + new_tensor.cmaps = x.cmaps + new_tensor.kmaps = x.kmaps + + return new_tensor + + +# x: SparseTensor, z: PointTensor +# return: PointTensor +def voxel_to_point(x, z, nearest=False): + if ( + z.idx_query is None + or z.weights is None + or z.idx_query.get(x.s) is None + or z.weights.get(x.s) is None + ): + off = spnn.utils.get_kernel_offsets(2, x.s, 1, device=z.F.device) + old_hash = F.sphash( + torch.cat( + [ + torch.floor(z.C[:, :3] / x.s[0]).int() * x.s[0], + z.C[:, -1].int().view(-1, 1), + ], + 1, + ), + off, + ) + pc_hash = F.sphash(x.C.to(z.F.device)) + idx_query = F.sphashquery(old_hash, pc_hash) + weights = ( + F.calc_ti_weights(z.C, idx_query, scale=x.s[0]).transpose(0, 1).contiguous() + ) + idx_query = idx_query.transpose(0, 1).contiguous() + if nearest: + weights[:, 1:] = 0.0 + idx_query[:, 1:] = -1 + new_feat = F.spdevoxelize(x.F, idx_query, weights) + new_tensor = PointTensor( + new_feat, z.C, idx_query=z.idx_query, weights=z.weights + ) + new_tensor.additional_features = z.additional_features + new_tensor.idx_query[x.s] = idx_query + new_tensor.weights[x.s] = weights + z.idx_query[x.s] = idx_query + z.weights[x.s] = weights + + else: + new_feat = F.spdevoxelize(x.F, z.idx_query.get(x.s), z.weights.get(x.s)) + new_tensor = PointTensor( + new_feat, z.C, idx_query=z.idx_query, weights=z.weights + ) + new_tensor.additional_features = z.additional_features + + return new_tensor + + +class BasicConvolutionBlock(nn.Module): + def __init__(self, inc, outc, ks=3, stride=1, dilation=1): + super().__init__() + self.net = nn.Sequential( + spnn.Conv3d(inc, outc, kernel_size=ks, dilation=dilation, stride=stride), + spnn.BatchNorm(outc), + spnn.ReLU(True), + ) + + def forward(self, x): + out = self.net(x) + return out + + +class BasicDeconvolutionBlock(nn.Module): + def __init__(self, inc, outc, ks=3, stride=1): + super().__init__() + self.net = nn.Sequential( + spnn.Conv3d(inc, outc, kernel_size=ks, stride=stride, transposed=True), + spnn.BatchNorm(outc), + spnn.ReLU(True), + ) + + def forward(self, x): + return self.net(x) + + +class ResidualBlock(nn.Module): + def __init__(self, inc, outc, ks=3, stride=1, dilation=1): + super().__init__() + self.net = nn.Sequential( + spnn.Conv3d(inc, outc, kernel_size=ks, dilation=dilation, stride=stride), + spnn.BatchNorm(outc), + spnn.ReLU(True), + spnn.Conv3d(outc, outc, kernel_size=ks, dilation=dilation, stride=1), + spnn.BatchNorm(outc), + ) + + if inc == outc and stride == 1: + self.downsample = nn.Identity() + else: + self.downsample = nn.Sequential( + spnn.Conv3d(inc, outc, kernel_size=1, dilation=1, stride=stride), + spnn.BatchNorm(outc), + ) + + self.relu = spnn.ReLU(True) + + def forward(self, x): + out = self.relu(self.net(x) + self.downsample(x)) + return out + + +@MODELS.register_module() +class SPVCNN(nn.Module): + def __init__( + self, + in_channels, + out_channels, + base_channels=32, + channels=(32, 64, 128, 256, 256, 128, 96, 96), + layers=(2, 2, 2, 2, 2, 2, 2, 2), + ): # not implement + super().__init__() + + assert ( + torchsparse is not None + ), "Please follow `README.md` to install torchsparse.`" + assert len(layers) % 2 == 0 + assert len(layers) == len(channels) + self.in_channels = in_channels + self.out_channels = out_channels + self.base_channels = base_channels + self.channels = channels + self.layers = layers + self.num_stages = len(layers) // 2 + + self.stem = nn.Sequential( + spnn.Conv3d(in_channels, base_channels, kernel_size=3, stride=1), + spnn.BatchNorm(base_channels), + spnn.ReLU(True), + spnn.Conv3d(base_channels, base_channels, kernel_size=3, stride=1), + spnn.BatchNorm(base_channels), + spnn.ReLU(True), + ) + + self.stage1 = nn.Sequential( + *[ + BasicConvolutionBlock( + base_channels, base_channels, ks=2, stride=2, dilation=1 + ), + ResidualBlock(base_channels, channels[0], ks=3, stride=1, dilation=1), + ] + + [ + ResidualBlock(channels[0], channels[0], ks=3, stride=1, dilation=1) + for _ in range(layers[0] - 1) + ] + ) + + self.stage2 = nn.Sequential( + *[ + BasicConvolutionBlock( + channels[0], channels[0], ks=2, stride=2, dilation=1 + ), + ResidualBlock(channels[0], channels[1], ks=3, stride=1, dilation=1), + ] + + [ + ResidualBlock(channels[1], channels[1], ks=3, stride=1, dilation=1) + for _ in range(layers[1] - 1) + ] + ) + + self.stage3 = nn.Sequential( + *[ + BasicConvolutionBlock( + channels[1], channels[1], ks=2, stride=2, dilation=1 + ), + ResidualBlock(channels[1], channels[2], ks=3, stride=1, dilation=1), + ] + + [ + ResidualBlock(channels[2], channels[2], ks=3, stride=1, dilation=1) + for _ in range(layers[2] - 1) + ] + ) + + self.stage4 = nn.Sequential( + *[ + BasicConvolutionBlock( + channels[2], channels[2], ks=2, stride=2, dilation=1 + ), + ResidualBlock(channels[2], channels[3], ks=3, stride=1, dilation=1), + ] + + [ + ResidualBlock(channels[3], channels[3], ks=3, stride=1, dilation=1) + for _ in range(layers[3] - 1) + ] + ) + + self.up1 = nn.ModuleList( + [ + BasicDeconvolutionBlock(channels[3], channels[4], ks=2, stride=2), + nn.Sequential( + *[ + ResidualBlock( + channels[4] + channels[2], + channels[4], + ks=3, + stride=1, + dilation=1, + ) + ] + + [ + ResidualBlock( + channels[4], channels[4], ks=3, stride=1, dilation=1 + ) + for _ in range(layers[4] - 1) + ] + ), + ] + ) + + self.up2 = nn.ModuleList( + [ + BasicDeconvolutionBlock(channels[4], channels[5], ks=2, stride=2), + nn.Sequential( + *[ + ResidualBlock( + channels[5] + channels[1], + channels[5], + ks=3, + stride=1, + dilation=1, + ) + ] + + [ + ResidualBlock( + channels[5], channels[5], ks=3, stride=1, dilation=1 + ) + for _ in range(layers[5] - 1) + ] + ), + ] + ) + + self.up3 = nn.ModuleList( + [ + BasicDeconvolutionBlock(channels[5], channels[6], ks=2, stride=2), + nn.Sequential( + *[ + ResidualBlock( + channels[6] + channels[0], + channels[6], + ks=3, + stride=1, + dilation=1, + ) + ] + + [ + ResidualBlock( + channels[6], channels[6], ks=3, stride=1, dilation=1 + ) + for _ in range(layers[6] - 1) + ] + ), + ] + ) + + self.up4 = nn.ModuleList( + [ + BasicDeconvolutionBlock(channels[6], channels[7], ks=2, stride=2), + nn.Sequential( + *[ + ResidualBlock( + channels[7] + base_channels, + channels[7], + ks=3, + stride=1, + dilation=1, + ) + ] + + [ + ResidualBlock( + channels[7], channels[7], ks=3, stride=1, dilation=1 + ) + for _ in range(layers[7] - 1) + ] + ), + ] + ) + + self.classifier = nn.Sequential(nn.Linear(channels[7], out_channels)) + + self.point_transforms = nn.ModuleList( + [ + nn.Sequential( + nn.Linear(base_channels, channels[3]), + nn.BatchNorm1d(channels[3]), + nn.ReLU(True), + ), + nn.Sequential( + nn.Linear(channels[3], channels[5]), + nn.BatchNorm1d(channels[5]), + nn.ReLU(True), + ), + nn.Sequential( + nn.Linear(channels[5], channels[7]), + nn.BatchNorm1d(channels[7]), + nn.ReLU(True), + ), + ] + ) + + self.weight_initialization() + self.dropout = nn.Dropout(0.3, True) + + def weight_initialization(self): + for m in self.modules(): + if isinstance(m, nn.BatchNorm1d): + nn.init.constant_(m.weight, 1) + nn.init.constant_(m.bias, 0) + + def forward(self, data_dict): + grid_coord = data_dict["grid_coord"] + feat = data_dict["feat"] + offset = data_dict["offset"] + batch = offset2batch(offset) + + # x: SparseTensor z: PointTensor + z = PointTensor( + feat, + torch.cat( + [grid_coord.float(), batch.unsqueeze(-1).float()], dim=1 + ).contiguous(), + ) + x0 = initial_voxelize(z) + + x0 = self.stem(x0) + z0 = voxel_to_point(x0, z, nearest=False) + z0.F = z0.F + + x1 = point_to_voxel(x0, z0) + x1 = self.stage1(x1) + x2 = self.stage2(x1) + x3 = self.stage3(x2) + x4 = self.stage4(x3) + z1 = voxel_to_point(x4, z0) + z1.F = z1.F + self.point_transforms[0](z0.F) + + y1 = point_to_voxel(x4, z1) + y1.F = self.dropout(y1.F) + y1 = self.up1[0](y1) + y1 = torchsparse.cat([y1, x3]) + y1 = self.up1[1](y1) + + y2 = self.up2[0](y1) + y2 = torchsparse.cat([y2, x2]) + y2 = self.up2[1](y2) + z2 = voxel_to_point(y2, z1) + z2.F = z2.F + self.point_transforms[1](z1.F) + + y3 = point_to_voxel(y2, z2) + y3.F = self.dropout(y3.F) + y3 = self.up3[0](y3) + y3 = torchsparse.cat([y3, x1]) + y3 = self.up3[1](y3) + + y4 = self.up4[0](y3) + y4 = torchsparse.cat([y4, x0]) + y4 = self.up4[1](y4) + z3 = voxel_to_point(y4, z2) + z3.F = z3.F + self.point_transforms[2](z2.F) + + out = self.classifier(z3.F) + return out diff --git a/pointcept/models/stratified_transformer/__init__.py b/pointcept/models/stratified_transformer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..24712d2bc0fe76a52a273172fe9c4f37c807a6c3 --- /dev/null +++ b/pointcept/models/stratified_transformer/__init__.py @@ -0,0 +1,2 @@ +from .stratified_transformer_v1m1_origin import StratifiedTransformer +from .stratified_transformer_v1m2_refine import StratifiedTransformer diff --git a/pointcept/models/stratified_transformer/stratified_transformer_v1m1_origin.py b/pointcept/models/stratified_transformer/stratified_transformer_v1m1_origin.py new file mode 100644 index 0000000000000000000000000000000000000000..e7d9c5921d9f49386f751dfaef6a28e984876870 --- /dev/null +++ b/pointcept/models/stratified_transformer/stratified_transformer_v1m1_origin.py @@ -0,0 +1,830 @@ +import torch +import torch.nn as nn + +try: + import torch_points_kernels as tp +except ImportError: + tp = None + +try: + from torch_points3d.modules.KPConv.kernels import KPConvLayer + from torch_points3d.core.common_modules import FastBatchNorm1d +except ImportError: + KPConvLayer = None + FastBatchNorm1d = None + +from torch_scatter import scatter_softmax +from timm.layers import DropPath, trunc_normal_ +from torch_geometric.nn.pool import voxel_grid + +try: + import pointops2.pointops as pointops +except ImportError: + pointops = None + +from pointcept.models.builder import MODELS + + +def offset2batch(offset): + return ( + torch.cat( + [ + ( + torch.tensor([i] * (o - offset[i - 1])) + if i > 0 + else torch.tensor([i] * o) + ) + for i, o in enumerate(offset) + ], + dim=0, + ) + .long() + .to(offset.device) + ) + + +def get_indice_pairs( + p2v_map, counts, new_p2v_map, new_counts, downsample_idx, batch, xyz, window_size, i +): + # p2v_map: [n, k] + # counts: [n, ] + + n, k = p2v_map.shape + mask = torch.arange(k).unsqueeze(0).cuda() < counts.unsqueeze(-1) # [n, k] + mask_mat = mask.unsqueeze(-1) & mask.unsqueeze(-2) # [n, k, k] + index_0 = p2v_map.unsqueeze(-1).expand(-1, -1, k)[mask_mat] # [M, ] + index_1 = p2v_map.unsqueeze(1).expand(-1, k, -1)[mask_mat] # [M, ] + + downsample_mask = torch.zeros_like(batch).bool() # [N, ] + downsample_mask[downsample_idx.long()] = True + + downsample_mask = downsample_mask[new_p2v_map] # [n, k] + n, k = new_p2v_map.shape + mask = torch.arange(k).unsqueeze(0).cuda() < new_counts.unsqueeze(-1) # [n, k] + downsample_mask = downsample_mask & mask + mask_mat = mask.unsqueeze(-1) & downsample_mask.unsqueeze(-2) # [n, k, k] + xyz_min = xyz.min(0)[0] + if i % 2 == 0: + window_coord = (xyz[new_p2v_map] - xyz_min) // window_size # [n, k, 3] + else: + window_coord = ( + xyz[new_p2v_map] + 1 / 2 * window_size - xyz_min + ) // window_size # [n, k, 3] + + mask_mat_prev = (window_coord.unsqueeze(2) != window_coord.unsqueeze(1)).any( + -1 + ) # [n, k, k] + mask_mat = mask_mat & mask_mat_prev # [n, k, k] + + new_index_0 = new_p2v_map.unsqueeze(-1).expand(-1, -1, k)[mask_mat] # [M, ] + new_index_1 = new_p2v_map.unsqueeze(1).expand(-1, k, -1)[mask_mat] # [M, ] + + index_0 = torch.cat([index_0, new_index_0], 0) + index_1 = torch.cat([index_1, new_index_1], 0) + return index_0, index_1 + + +def grid_sample(pos, batch, size, start, return_p2v=True): + # pos: float [N, 3] + # batch: long [N] + # size: float [3, ] + # start: float [3, ] / None + + cluster = voxel_grid(pos, batch, size, start=start) # [N, ] + + if return_p2v == False: + unique, cluster = torch.unique(cluster, sorted=True, return_inverse=True) + return cluster + + unique, cluster, counts = torch.unique( + cluster, sorted=True, return_inverse=True, return_counts=True + ) + + # obtain p2v_map + n = unique.shape[0] + k = counts.max().item() + p2v_map = cluster.new_zeros(n, k) # [n, k] + mask = torch.arange(k).cuda().unsqueeze(0) < counts.unsqueeze(-1) # [n, k] + p2v_map[mask] = torch.argsort(cluster) + + return cluster, p2v_map, counts + + +class Mlp(nn.Module): + """Multilayer perceptron.""" + + def __init__( + self, + in_features, + hidden_features=None, + out_features=None, + act_layer=nn.GELU, + drop=0.0, + ): + super().__init__() + out_features = out_features or in_features + hidden_features = hidden_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.act = act_layer() + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop = nn.Dropout(drop, inplace=True) + + def forward(self, x): + x = self.fc1(x) + x = self.act(x) + x = self.drop(x) + x = self.fc2(x) + x = self.drop(x) + return x + + +class TransitionDown(nn.Module): + def __init__(self, in_channels, out_channels, ratio, k, norm_layer=nn.LayerNorm): + super().__init__() + self.ratio = ratio + self.k = k + self.norm = norm_layer(in_channels) if norm_layer else None + self.linear = nn.Linear(in_channels, out_channels, bias=False) + self.pool = nn.MaxPool1d(k) + + def forward(self, feats, xyz, offset): + n_offset, count = [int(offset[0].item() * self.ratio) + 1], int( + offset[0].item() * self.ratio + ) + 1 + for i in range(1, offset.shape[0]): + count += ((offset[i].item() - offset[i - 1].item()) * self.ratio) + 1 + n_offset.append(count) + n_offset = torch.cuda.IntTensor(n_offset) + idx = pointops.furthestsampling(xyz, offset, n_offset) # (m) + n_xyz = xyz[idx.long(), :] # (m, 3) + + feats = pointops.queryandgroup( + self.k, xyz, n_xyz, feats, None, offset, n_offset, use_xyz=False + ) # (m, nsample, 3+c) + m, k, c = feats.shape + feats = ( + self.linear(self.norm(feats.view(m * k, c)).view(m, k, c)) + .transpose(1, 2) + .contiguous() + ) + feats = self.pool(feats).squeeze(-1) # (m, c) + + return feats, n_xyz, n_offset + + +class WindowAttention(nn.Module): + """Window based multi-head self attention (W-MSA) module with relative position bias. + It supports both of shifted and non-shifted window. + + Args: + dim (int): Number of input channels. + window_size (tuple[int]): The height and width of the window. + num_heads (int): Number of attention heads. + qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True + qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set + attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0 + proj_drop (float, optional): Dropout ratio of output. Default: 0.0 + """ + + def __init__( + self, + dim, + window_size, + num_heads, + quant_size, + rel_query=True, + rel_key=False, + rel_value=False, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + ): + super().__init__() + self.dim = dim + self.num_heads = num_heads + head_dim = dim // num_heads + self.scale = qk_scale or head_dim**-0.5 + self.window_size = window_size + + self.quant_size = quant_size + self.rel_query = rel_query + self.rel_key = rel_key + self.rel_value = rel_value + + quant_grid_length = int((2 * window_size + 1e-4) // quant_size) + + if rel_query: + self.relative_pos_query_table = nn.Parameter( + torch.zeros(2 * quant_grid_length, num_heads, head_dim, 3) + ) + trunc_normal_(self.relative_pos_query_table, std=0.02) + if rel_key: + self.relative_pos_key_table = nn.Parameter( + torch.zeros(2 * quant_grid_length, num_heads, head_dim, 3) + ) + trunc_normal_(self.relative_pos_key_table, std=0.02) + if rel_value: + self.relative_pos_value_table = nn.Parameter( + torch.zeros(2 * quant_grid_length, num_heads, head_dim, 3) + ) + trunc_normal_(self.relative_pos_value_table, std=0.02) + + self.quant_grid_length = quant_grid_length + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop, inplace=True) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop, inplace=True) + + self.softmax = nn.Softmax(dim=-1) + + # def forward(self, feats, xyz, index_0, index_1): + def forward(self, feats, xyz, index_0, index_1, index_0_offsets, n_max): + """Forward function. + + Args: + feats: N, C + xyz: N, 3 + index_0: M, + index_1: M, + """ + + N, C = feats.shape + M = index_0.shape[0] + + assert index_0.shape[0] == index_1.shape[0] + + # Query, Key, Value + qkv = ( + self.qkv(feats) + .reshape(N, 3, self.num_heads, C // self.num_heads) + .permute(1, 0, 2, 3) + .contiguous() + ) + query, key, value = qkv[0], qkv[1], qkv[2] # [N, num_heads, C//num_heads] + query = query * self.scale + attn_flat = pointops.attention_step1_v2( + query.float(), key.float(), index_1.int(), index_0_offsets.int(), n_max + ) + + # # Position embedding + relative_position = xyz[index_0] - xyz[index_1] + relative_position = torch.round(relative_position * 100000) / 100000 + relative_position_index = ( + relative_position + 2 * self.window_size - 0.0001 + ) // self.quant_size + assert (relative_position_index >= 0).all() + assert (relative_position_index <= 2 * self.quant_grid_length - 1).all() + + assert self.rel_query and self.rel_key + if self.rel_query and self.rel_key: + relative_position_bias = pointops.dot_prod_with_idx_v3( + query.float(), + index_0_offsets.int(), + n_max, + key.float(), + index_1.int(), + self.relative_pos_query_table.float(), + self.relative_pos_key_table.float(), + relative_position_index.int(), + ) + elif self.rel_query: + relative_position_bias = pointops.dot_prod_with_idx( + query.float(), + index_0.int(), + self.relative_pos_query_table.float(), + relative_position_index.int(), + ) # [M, num_heads] + elif self.rel_key: + relative_position_bias = pointops.dot_prod_with_idx( + key.float(), + index_1.int(), + self.relative_pos_key_table.float(), + relative_position_index.int(), + ) # [M, num_heads] + else: + relative_position_bias = 0.0 + + attn_flat = attn_flat + relative_position_bias # [M, num_heads] + + softmax_attn_flat = scatter_softmax( + src=attn_flat, index=index_0, dim=0 + ) # [M, num_heads] + + if self.rel_value: + x = pointops.attention_step2_with_rel_pos_value_v2( + softmax_attn_flat.float(), + value.float(), + index_0_offsets.int(), + n_max, + index_1.int(), + self.relative_pos_value_table.float(), + relative_position_index.int(), + ) + else: + x = pointops.attention_step2( + softmax_attn_flat.float(), value.float(), index_0.int(), index_1.int() + ) + + x = x.view(N, C) + + x = self.proj(x) + x = self.proj_drop(x) # [N, C] + + return x + + +class SwinTransformerBlock(nn.Module): + def __init__( + self, + dim, + num_heads, + window_size, + quant_size, + rel_query=True, + rel_key=False, + rel_value=False, + drop_path=0.0, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + act_layer=nn.GELU, + norm_layer=nn.LayerNorm, + mode=4, + ): # mode=4:mean + super().__init__() + self.mode = mode + + self.norm1 = norm_layer(dim) + self.attn = WindowAttention( + dim, + window_size, + num_heads=num_heads, + quant_size=quant_size, + rel_query=rel_query, + rel_key=rel_key, + rel_value=rel_value, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + ) + + self.drop_path = DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + self.norm2 = norm_layer(dim) + mlp_hidden_dim = int(dim * mlp_ratio) + self.mlp = Mlp( + in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer + ) + + def forward(self, feats, xyz, index_0, index_1, index_0_offsets, n_max): + # feats: [N, c] + # pos: [N, 3] + + short_cut = feats + + feats = self.norm1(feats) + + feats = self.attn( + feats, xyz, index_0, index_1, index_0_offsets, n_max + ) # index_0 MUST be in ascending order + + feats = short_cut + self.drop_path(feats) + feats = feats + self.drop_path(self.mlp(self.norm2(feats))) + + return feats + + +class BasicLayer(nn.Module): + def __init__( + self, + downsample_scale, + depth, + channel, + num_heads, + window_size, + grid_size, + quant_size, + rel_query=True, + rel_key=False, + rel_value=False, + drop_path=0.0, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + norm_layer=nn.LayerNorm, + downsample=None, + ratio=0.25, + k=16, + out_channels=None, + ): + super().__init__() + self.depth = depth + self.grid_size = grid_size + self.max_window_counts = 64 + self.window_size = window_size + self.downsample_scale = downsample_scale + + self.blocks = nn.ModuleList( + [ + SwinTransformerBlock( + channel, + num_heads, + window_size, + quant_size, + rel_query=rel_query, + rel_key=rel_key, + rel_value=rel_value, + drop_path=( + drop_path[i] if isinstance(drop_path, list) else drop_path + ), + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + norm_layer=norm_layer, + ) + for i in range(depth) + ] + ) + + self.downsample = ( + downsample(channel, out_channels, ratio, k) if downsample else None + ) + + def forward(self, feats, xyz, offset): + # feats: N, C + # xyz: N, 3 + + window_size = torch.tensor([self.window_size] * 3).type_as(xyz).to(xyz.device) + + offset_ = offset.clone() + offset_[1:] = offset_[1:] - offset_[:-1] + batch = ( + torch.cat([torch.tensor([ii] * o) for ii, o in enumerate(offset_)], 0) + .long() + .cuda() + ) + + v2p_map, p2v_map, counts = grid_sample(xyz, batch, window_size, start=None) + + shift_size = 1 / 2 * window_size + shift_v2p_map, shift_p2v_map, shift_counts = grid_sample( + xyz + shift_size, batch, window_size, start=xyz.min(0)[0] + ) + + downsample_scale = self.downsample_scale + new_offset, count = [offset[0].item() // downsample_scale + 1], offset[ + 0 + ].item() // downsample_scale + 1 + for i in range(1, offset.shape[0]): + count += (offset[i].item() - offset[i - 1].item()) // downsample_scale + 1 + new_offset.append(count) + + new_offset = torch.cuda.IntTensor(new_offset) + downsample_idx = pointops.furthestsampling( + xyz, offset.int(), new_offset.int() + ) # [N/16,] + + new_window_size = 2 * torch.tensor([self.window_size] * 3).type_as(xyz).to( + xyz.device + ) + + # offset_ = new_offset.clone() + # offset_[1:] = offset_[1:] - offset_[:-1] + # new_batch = torch.cat([torch.tensor([ii]*o) for ii,o in enumerate(offset_)], 0).long().cuda() + + new_v2p_map, new_p2v_map, new_counts = grid_sample( + xyz, batch, new_window_size, start=None + ) + + shift_size = 1 / 2 * new_window_size + shift_new_v2p_map, shift_new_p2v_map, shift_new_counts = grid_sample( + xyz + shift_size, batch, new_window_size, start=xyz.min(0)[0] + ) + + for i, blk in enumerate(self.blocks): + p2v_map_blk = p2v_map if i % 2 == 0 else shift_p2v_map + counts_blk = counts if i % 2 == 0 else shift_counts + + new_p2v_map_blk = new_p2v_map if i % 2 == 0 else shift_new_p2v_map + new_counts_blk = new_counts if i % 2 == 0 else shift_new_counts + + index_0, index_1 = get_indice_pairs( + p2v_map_blk, + counts_blk, + new_p2v_map_blk, + new_counts_blk, + downsample_idx, + batch, + xyz, + window_size, + i, + ) + + # rearrange index for acceleration + index_0, indices = torch.sort(index_0) # [M,] + index_1 = index_1[indices] # [M,] + index_0_counts = index_0.bincount() + n_max = index_0_counts.max() + index_0_offsets = index_0_counts.cumsum(dim=-1) # [N] + index_0_offsets = torch.cat( + [torch.zeros(1, dtype=torch.long).cuda(), index_0_offsets], 0 + ) # [N+1] + + feats = blk(feats, xyz, index_0, index_1, index_0_offsets, n_max) + + if self.downsample: + feats_down, xyz_down, offset_down = self.downsample(feats, xyz, offset) + else: + feats_down, xyz_down, offset_down = None, None, None + + return feats, xyz, offset, feats_down, xyz_down, offset_down + + +class Upsample(nn.Module): + def __init__(self, k, in_channels, out_channels, bn_momentum=0.02): + super().__init__() + self.k = k + self.in_channels = in_channels + self.out_channels = out_channels + + self.linear1 = nn.Sequential( + nn.LayerNorm(out_channels), nn.Linear(out_channels, out_channels) + ) + self.linear2 = nn.Sequential( + nn.LayerNorm(in_channels), nn.Linear(in_channels, out_channels) + ) + + def forward( + self, feats, xyz, support_xyz, offset, support_offset, support_feats=None + ): + feats = self.linear1(support_feats) + pointops.interpolation( + xyz, support_xyz, self.linear2(feats), offset, support_offset + ) + return feats, support_xyz, support_offset + + +class KPConvSimpleBlock(nn.Module): + def __init__( + self, + in_channels, + out_channels, + prev_grid_size, + sigma=1.0, + negative_slope=0.2, + bn_momentum=0.02, + ): + super().__init__() + self.kpconv = KPConvLayer( + in_channels, + out_channels, + point_influence=prev_grid_size * sigma, + add_one=False, + ) + self.bn = FastBatchNorm1d(out_channels, momentum=bn_momentum) + self.activation = nn.LeakyReLU(negative_slope=negative_slope) + + def forward(self, feats, xyz, batch, neighbor_idx): + # feats: [N, C] + # xyz: [N, 3] + # batch: [N,] + # neighbor_idx: [N, M] + + feats = self.kpconv(xyz, xyz, neighbor_idx, feats) + feats = self.activation(self.bn(feats)) + return feats + + +class KPConvResBlock(nn.Module): + def __init__( + self, + in_channels, + out_channels, + prev_grid_size, + sigma=1.0, + negative_slope=0.2, + bn_momentum=0.02, + ): + super().__init__() + d_2 = out_channels // 4 + activation = nn.LeakyReLU(negative_slope=negative_slope) + self.unary_1 = torch.nn.Sequential( + nn.Linear(in_channels, d_2, bias=False), + FastBatchNorm1d(d_2, momentum=bn_momentum), + activation, + ) + self.unary_2 = torch.nn.Sequential( + nn.Linear(d_2, out_channels, bias=False), + FastBatchNorm1d(out_channels, momentum=bn_momentum), + activation, + ) + self.kpconv = KPConvLayer( + d_2, d_2, point_influence=prev_grid_size * sigma, add_one=False + ) + self.bn = FastBatchNorm1d(out_channels, momentum=bn_momentum) + self.activation = activation + + if in_channels != out_channels: + self.shortcut_op = torch.nn.Sequential( + nn.Linear(in_channels, out_channels, bias=False), + FastBatchNorm1d(out_channels, momentum=bn_momentum), + ) + else: + self.shortcut_op = nn.Identity() + + def forward(self, feats, xyz, batch, neighbor_idx): + # feats: [N, C] + # xyz: [N, 3] + # batch: [N,] + # neighbor_idx: [N, M] + + shortcut = feats + feats = self.unary_1(feats) + feats = self.kpconv(xyz, xyz, neighbor_idx, feats) + feats = self.unary_2(feats) + shortcut = self.shortcut_op(shortcut) + feats += shortcut + return feats + + +@MODELS.register_module("ST-v1m1") +class StratifiedTransformer(nn.Module): + def __init__( + self, + downsample_scale, + depths, + channels, + num_heads, + window_size, + up_k, + grid_sizes, + quant_sizes, + rel_query=True, + rel_key=False, + rel_value=False, + drop_path_rate=0.2, + num_layers=4, + concat_xyz=False, + num_classes=13, + ratio=0.25, + k=16, + prev_grid_size=0.04, + sigma=1.0, + stem_transformer=False, + kp_ball_radius=0.02 * 2.5, + kp_max_neighbor=34, + ): + super().__init__() + assert ( + KPConvLayer is not None and FastBatchNorm1d is not None + ), "Please make sure torch_points3d is installed" + assert tp is not None, "Please make sure torch_points_kernels is installed" + assert pointops is not None, "Please make sure pointops2 is installed" + + dpr = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(depths)) + ] # stochastic depth decay rule + + self.kp_ball_radius = kp_ball_radius + self.kp_max_neighbor = kp_max_neighbor + if stem_transformer: + self.stem_layer = nn.ModuleList( + [ + KPConvSimpleBlock( + 3 if not concat_xyz else 6, + channels[0], + prev_grid_size, + sigma=sigma, + ) + ] + ) + self.layer_start = 0 + else: + self.stem_layer = nn.ModuleList( + [ + KPConvSimpleBlock( + 3 if not concat_xyz else 6, + channels[0], + prev_grid_size, + sigma=sigma, + ), + KPConvResBlock( + channels[0], channels[0], prev_grid_size, sigma=sigma + ), + ] + ) + self.downsample = TransitionDown(channels[0], channels[1], ratio, k) + self.layer_start = 1 + + self.layers = nn.ModuleList( + [ + BasicLayer( + downsample_scale, + depths[i], + channels[i], + num_heads[i], + window_size[i], + grid_sizes[i], + quant_sizes[i], + rel_query=rel_query, + rel_key=rel_key, + rel_value=rel_value, + drop_path=dpr[sum(depths[:i]) : sum(depths[: i + 1])], + downsample=TransitionDown if i < num_layers - 1 else None, + ratio=ratio, + k=k, + out_channels=channels[i + 1] if i < num_layers - 1 else None, + ) + for i in range(self.layer_start, num_layers) + ] + ) + + self.upsamples = nn.ModuleList( + [ + Upsample(up_k, channels[i], channels[i - 1]) + for i in range(num_layers - 1, 0, -1) + ] + ) + + self.classifier = nn.Sequential( + nn.Linear(channels[0], channels[0]), + nn.BatchNorm1d(channels[0]), + nn.ReLU(inplace=True), + nn.Linear(channels[0], num_classes), + ) + + self.init_weights() + + def forward(self, data_dict): + feats = data_dict["feat"] + xyz = data_dict["coord"] + offset = data_dict["offset"].int() + batch = offset2batch(offset) + neighbor_idx = tp.ball_query( + self.kp_ball_radius, + self.kp_max_neighbor, + xyz, + xyz, + mode="partial_dense", + batch_x=batch, + batch_y=batch, + )[0] + + feats_stack = [] + xyz_stack = [] + offset_stack = [] + + for i, layer in enumerate(self.stem_layer): + feats = layer(feats, xyz, batch, neighbor_idx) + + feats = feats.contiguous() + + if self.layer_start == 1: + feats_stack.append(feats) + xyz_stack.append(xyz) + offset_stack.append(offset) + feats, xyz, offset = self.downsample(feats, xyz, offset) + + for i, layer in enumerate(self.layers): + feats, xyz, offset, feats_down, xyz_down, offset_down = layer( + feats, xyz, offset + ) + + feats_stack.append(feats) + xyz_stack.append(xyz) + offset_stack.append(offset) + + feats = feats_down + xyz = xyz_down + offset = offset_down + + feats = feats_stack.pop() + xyz = xyz_stack.pop() + offset = offset_stack.pop() + + for i, upsample in enumerate(self.upsamples): + feats, xyz, offset = upsample( + feats, + xyz, + xyz_stack.pop(), + offset, + offset_stack.pop(), + support_feats=feats_stack.pop(), + ) + + out = self.classifier(feats) + + return out + + def init_weights(self): + """Initialize the weights in backbone.""" + + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.LayerNorm) or isinstance(m, nn.BatchNorm1d): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + self.apply(_init_weights) diff --git a/pointcept/models/stratified_transformer/stratified_transformer_v1m2_refine.py b/pointcept/models/stratified_transformer/stratified_transformer_v1m2_refine.py new file mode 100644 index 0000000000000000000000000000000000000000..04c0717b1b6431c9c0ed3175d014dadda1294b26 --- /dev/null +++ b/pointcept/models/stratified_transformer/stratified_transformer_v1m2_refine.py @@ -0,0 +1,763 @@ +""" +Stratified Transformer + +Modified from https://github.com/dvlab-research/Stratified-Transformer + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from copy import deepcopy +import torch +import torch.nn as nn + +try: + import torch_points_kernels as tp +except ImportError: + tp = None + +try: + from torch_points3d.modules.KPConv.kernels import KPConvLayer + from torch_points3d.core.common_modules import FastBatchNorm1d +except ImportError: + KPConvLayer = None + FastBatchNorm1d = None + +from torch_scatter import scatter_softmax +from timm.layers import DropPath, trunc_normal_ +from torch_geometric.nn.pool import voxel_grid + +try: + import pointops2.pointops as pointops +except ImportError: + pointops = None + +from pointcept.models.builder import MODELS + + +def offset2batch(offset): + return ( + torch.cat( + [ + ( + torch.tensor([i] * (o - offset[i - 1])) + if i > 0 + else torch.tensor([i] * o) + ) + for i, o in enumerate(offset) + ], + dim=0, + ) + .long() + .to(offset.device) + ) + + +def grid_sample(coords, batch, size, start, return_p2v=True): + cluster = voxel_grid(coords, batch, size, start=start) + + if not return_p2v: + unique, cluster = torch.unique(cluster, sorted=True, return_inverse=True) + return cluster + else: + unique, cluster, counts = torch.unique( + cluster, sorted=True, return_inverse=True, return_counts=True + ) + + # obtain p2v_map + n = unique.shape[0] + k = counts.max().item() + p2v_map = cluster.new_zeros(n, k) + mask = torch.arange(k).cuda().unsqueeze(0) < counts.unsqueeze(-1) + p2v_map[mask] = torch.argsort(cluster) + return cluster, p2v_map, counts + + +class WindowAttention(nn.Module): + """Window based multi-head self attention (W-MSA) module with relative position bias. + It supports both of shifted and non-shifted window. + """ + + def __init__( + self, + embed_channels, + num_heads, + window_size, + quant_size, + attn_drop=0.0, + proj_drop=0.0, + scale=None, + rel_query=True, + rel_key=True, + rel_value=True, + qkv_bias=True, + ): + super().__init__() + self.embed_channels = embed_channels + self.head_channels = embed_channels // num_heads + self.num_heads = num_heads + self.scale = scale or self.head_channels**-0.5 + + self.window_size = window_size + self.quant_size = quant_size + + self.rel_query = rel_query + self.rel_key = rel_key + self.rel_value = rel_value + + self.quant_grid_length = int((2 * window_size + 1e-4) // quant_size) + + assert self.rel_query and self.rel_key + if rel_query: + self.relative_pos_query_table = nn.Parameter( + torch.zeros( + 2 * self.quant_grid_length, self.num_heads, self.head_channels, 3 + ) + ) + trunc_normal_(self.relative_pos_query_table, std=0.02) + + if rel_key: + self.relative_pos_key_table = nn.Parameter( + torch.zeros( + 2 * self.quant_grid_length, self.num_heads, self.head_channels, 3 + ) + ) + trunc_normal_(self.relative_pos_query_table, std=0.02) + + if rel_value: + self.relative_pos_value_table = nn.Parameter( + torch.zeros( + 2 * self.quant_grid_length, self.num_heads, self.head_channels, 3 + ) + ) + trunc_normal_(self.relative_pos_query_table, std=0.02) + + self.qkv = nn.Linear(embed_channels, embed_channels * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop, inplace=True) + self.proj = nn.Linear(embed_channels, embed_channels) + self.proj_drop = nn.Dropout(proj_drop, inplace=True) + + self.softmax = nn.Softmax(dim=-1) + + def forward(self, feats, coords, index_0, index_1, index_0_offsets, n_max): + n, c = feats.shape + m = index_0.shape[0] + + assert index_0.shape[0] == index_1.shape[0] + + qkv = ( + self.qkv(feats) + .reshape(n, 3, self.num_heads, c // self.num_heads) + .permute(1, 0, 2, 3) + .contiguous() + ) + query, key, value = qkv[0], qkv[1], qkv[2] + query = query * self.scale + attn_flat = pointops.attention_step1_v2( + query.float(), key.float(), index_1.int(), index_0_offsets.int(), n_max + ) + + # Position embedding + relative_position = coords[index_0] - coords[index_1] + relative_position = torch.round(relative_position * 100000) / 100000 + relative_position_index = torch.div( + relative_position + 2 * self.window_size - 1e-4, + self.quant_size, + rounding_mode="trunc", + ) + # relative_position_index = (relative_position + 2 * self.window_size - 1e-4) // self.quant_size + assert (relative_position_index >= 0).all() + assert (relative_position_index <= 2 * self.quant_grid_length - 1).all() + + if self.rel_query and self.rel_key: + relative_position_bias = pointops.dot_prod_with_idx_v3( + query.float(), + index_0_offsets.int(), + n_max, + key.float(), + index_1.int(), + self.relative_pos_query_table.float(), + self.relative_pos_key_table.float(), + relative_position_index.int(), + ) + elif self.rel_query: + relative_position_bias = pointops.dot_prod_with_idx( + query.float(), + index_0.int(), + self.relative_pos_query_table.float(), + relative_position_index.int(), + ) # [M, num_heads] + elif self.rel_key: + relative_position_bias = pointops.dot_prod_with_idx( + key.float(), + index_1.int(), + self.relative_pos_key_table.float(), + relative_position_index.int(), + ) # [M, num_heads] + else: + relative_position_bias = 0.0 + + attn_flat += relative_position_bias + softmax_attn_flat = scatter_softmax(src=attn_flat, index=index_0, dim=0) + + if self.rel_value: + x = pointops.attention_step2_with_rel_pos_value_v2( + softmax_attn_flat.float(), + value.float(), + index_0_offsets.int(), + n_max, + index_1.int(), + self.relative_pos_value_table.float(), + relative_position_index.int(), + ) + else: + x = pointops.attention_step2( + softmax_attn_flat.float(), value.float(), index_0.int(), index_1.int() + ) + + x = x.view(n, c) + x = self.proj(x) + x = self.proj_drop(x) + return x + + +class MLP(nn.Module): + def __init__(self, in_channels, hidden_channels=None, out_channels=None, drop=0.0): + super().__init__() + out_channels = out_channels or in_channels + hidden_channels = hidden_channels or in_channels + self.fc1 = nn.Linear(in_channels, hidden_channels) + self.act = nn.GELU() + self.fc2 = nn.Linear(hidden_channels, out_channels) + self.drop = nn.Dropout(drop, inplace=True) + + def forward(self, x): + x = self.fc1(x) + x = self.act(x) + x = self.drop(x) + x = self.fc2(x) + x = self.drop(x) + return x + + +class Block(nn.Module): + def __init__( + self, + embed_channels, + num_heads, + window_size, + quant_size, + mlp_expend_ratio=4.0, + drop_path=0.0, + qk_scale=None, + rel_query=True, + rel_key=True, + rel_value=True, + qkv_bias=True, + ): + super().__init__() + self.norm1 = nn.LayerNorm(embed_channels) + self.attn = WindowAttention( + embed_channels, + num_heads, + window_size, + quant_size, + scale=qk_scale, + rel_query=rel_query, + rel_key=rel_key, + rel_value=rel_value, + qkv_bias=qkv_bias, + ) + self.drop_path = DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + self.norm2 = nn.LayerNorm(embed_channels) + self.mlp = MLP( + in_channels=embed_channels, + hidden_channels=int(embed_channels * mlp_expend_ratio), + ) + + def forward(self, feats, coords, index_0, index_1, index_0_offsets, n_max): + short_cut = feats + feats = self.norm1(feats) + feats = self.attn(feats, coords, index_0, index_1, index_0_offsets, n_max) + + feats = short_cut + self.drop_path(feats) + feats += self.drop_path(self.mlp(self.norm2(feats))) + return feats + + +class BasicLayer(nn.Module): + def __init__( + self, + embed_channels, + out_channels, + depth, + num_heads, + window_size, + quant_size, + mlp_expend_ratio=4.0, + down_ratio=0.25, + down_num_sample=16, + drop_path=None, + qk_scale=None, + down=True, + rel_query=True, + rel_key=True, + rel_value=True, + qkv_bias=True, + ): + super().__init__() + self.depth = depth + self.window_size = window_size + self.quant_size = quant_size + self.down_ratio = down_ratio + + if isinstance(drop_path, list): + drop_path = drop_path + assert len(drop_path) == depth + elif isinstance(drop_path, float): + drop_path = [deepcopy(drop_path) for _ in range(depth)] + else: + drop_path = [0.0 for _ in range(depth)] + + self.blocks = nn.ModuleList() + for i in range(depth): + block = Block( + embed_channels, + num_heads, + window_size, + quant_size, + mlp_expend_ratio=mlp_expend_ratio, + drop_path=drop_path[i], + qk_scale=qk_scale, + rel_query=rel_query, + rel_key=rel_key, + rel_value=rel_value, + qkv_bias=qkv_bias, + ) + self.blocks.append(block) + + self.down = ( + TransitionDown(embed_channels, out_channels, down_ratio, down_num_sample) + if down + else None + ) + + def forward(self, feats, coords, offset): + # window_size -> [window_size, window_size, window_size] + window_size = torch.tensor( + [self.window_size] * 3, dtype=coords.dtype, device=coords.device + ) + new_window_size = 2 * torch.tensor( + [self.window_size] * 3, dtype=coords.dtype, device=coords.device + ) + batch = offset2batch(offset) + + # compute new offset + new_offset = [int(offset[0].item() * self.down_ratio) + 1] + count = int(offset[0].item() * self.down_ratio) + 1 + for i in range(1, offset.shape[0]): + count += ( + int((offset[i].item() - offset[i - 1].item()) * self.down_ratio) + 1 + ) + new_offset.append(count) + new_offset = torch.cuda.IntTensor(new_offset) + down_idx = pointops.furthestsampling(coords, offset.int(), new_offset.int()) + + # compute window mapping + coords_min = coords.min(0).values + v2p_map, p2v_map, counts = grid_sample(coords, batch, window_size, start=None) + shift_size = window_size * 1 / 2 + shift_v2p_map, shift_p2v_map, shift_counts = grid_sample( + coords + shift_size, batch, window_size, start=coords_min + ) + + new_v2p_map, new_p2v_map, new_counts = grid_sample( + coords, batch, new_window_size, start=None + ) + shift_size = new_window_size * 1 / 2 + shift_new_v2p_map, shift_new_p2v_map, shift_new_counts = grid_sample( + coords + shift_size, batch, new_window_size, start=coords_min + ) + + # stratified attention + for i, blk in enumerate(self.blocks): + p2v_map_blk = p2v_map if i % 2 == 0 else shift_p2v_map + counts_blk = counts if i % 2 == 0 else shift_counts + + new_p2v_map_blk = new_p2v_map if i % 2 == 0 else shift_new_p2v_map + new_counts_blk = new_counts if i % 2 == 0 else shift_new_counts + + n, k = p2v_map_blk.shape + mask = torch.arange(k).unsqueeze(0).cuda() < counts_blk.unsqueeze(-1) + mask_mat = mask.unsqueeze(-1) & mask.unsqueeze(-2) + index_0 = p2v_map_blk.unsqueeze(-1).expand(-1, -1, k)[mask_mat] + index_1 = p2v_map_blk.unsqueeze(1).expand(-1, k, -1)[mask_mat] + + down_mask = torch.zeros_like(batch).bool() + down_mask[down_idx.long()] = True + down_mask = down_mask[new_p2v_map_blk] # [n, k], down sample mask + n, k = new_p2v_map_blk.shape + mask = torch.arange(k).unsqueeze(0).cuda() < new_counts_blk.unsqueeze( + -1 + ) # [n, k] + down_mask = down_mask & mask # down sample and window mask + # [n, k, k] query: dense point in large windows; key: sparse point in large windows + mask_mat = mask.unsqueeze(-1) & down_mask.unsqueeze(-2) + + if i % 2 == 0: + # [n, k, 3] + # window_coord = (coords[new_p2v_map_blk] - coords_min) // window_size + window_coord = torch.div( + coords[new_p2v_map_blk] - coords_min, + window_size, + rounding_mode="trunc", + ) + else: + # [n, k, 3] + # window_coord = (coords[new_p2v_map_blk] - coords_min + 1/2 * window_size) // window_size + window_coord = torch.div( + coords[new_p2v_map_blk] - coords_min + 1 / 2 * window_size, + window_size, + rounding_mode="trunc", + ) + # [n, k, k], whether pair points are in same small windows + mask_mat_prev = ( + window_coord.unsqueeze(2) != window_coord.unsqueeze(1) + ).any(-1) + mask_mat = mask_mat & mask_mat_prev + + new_index_0 = new_p2v_map_blk.unsqueeze(-1).expand(-1, -1, k)[mask_mat] + new_index_1 = new_p2v_map_blk.unsqueeze(1).expand(-1, k, -1)[mask_mat] + + index_0 = torch.cat([index_0, new_index_0], 0) + index_1 = torch.cat([index_1, new_index_1], 0) + + # rearrange index for acceleration + index_0, indices = torch.sort(index_0) + index_1 = index_1[indices] + index_0_counts = index_0.bincount() + n_max = index_0_counts.max() + index_0_offsets = index_0_counts.cumsum(dim=-1) + index_0_offsets = torch.cat( + [torch.zeros(1, dtype=torch.long).cuda(), index_0_offsets], 0 + ) + + feats = blk(feats, coords, index_0, index_1, index_0_offsets, n_max) + + if self.down: + feats_down, coords_down, offset_down = self.down(feats, coords, offset) + else: + feats_down, coords_down, offset_down = None, None, None + + return feats, coords, offset, feats_down, coords_down, offset_down + + +class TransitionDown(nn.Module): + def __init__(self, in_channels, out_channels, ratio, k, norm_layer=nn.LayerNorm): + super().__init__() + self.ratio = ratio + self.k = k + self.norm = norm_layer(in_channels) if norm_layer else None + self.linear = nn.Linear(in_channels, out_channels, bias=False) + self.pool = nn.MaxPool1d(k) + + def forward(self, feats, coords, offset): + new_offset, count = [int(offset[0].item() * self.ratio) + 1], int( + offset[0].item() * self.ratio + ) + 1 + for i in range(1, offset.shape[0]): + count += ((offset[i].item() - offset[i - 1].item()) * self.ratio) + 1 + new_offset.append(count) + new_offset = torch.cuda.IntTensor(new_offset) + idx = pointops.furthestsampling(coords, offset, new_offset) # (m) + new_coords = coords[idx.long(), :] # (m, 3) + + feats = pointops.queryandgroup( + self.k, coords, new_coords, feats, None, offset, new_offset, use_xyz=False + ) # (m, nsample, 3+c) + m, k, c = feats.shape + feats = ( + self.linear(self.norm(feats.view(m * k, c)).view(m, k, c)) + .transpose(1, 2) + .contiguous() + ) + feats = self.pool(feats).squeeze(-1) # (m, c) + return feats, new_coords, new_offset + + +class TransitionUp(nn.Module): + def __init__(self, in_channels, out_channels): + super().__init__() + self.in_channels = in_channels + self.out_channels = out_channels + + self.linear1 = nn.Sequential( + nn.LayerNorm(out_channels), nn.Linear(out_channels, out_channels) + ) + + self.linear2 = nn.Sequential( + nn.LayerNorm(in_channels), nn.Linear(in_channels, out_channels) + ) + + def forward(self, feats, coords, offset, skip_feats, skip_coords, skip_offset): + feats = self.linear1(skip_feats) + pointops.interpolation( + coords, skip_coords, self.linear2(feats), offset, skip_offset + ) + return feats, skip_coords, skip_offset + + +class KPConvSimpleBlock(nn.Module): + def __init__( + self, + in_channels, + out_channels, + prev_grid_size, + sigma=1.0, + negative_slope=0.2, + bn_momentum=0.02, + ): + super().__init__() + self.kpconv = KPConvLayer( + in_channels, + out_channels, + point_influence=prev_grid_size * sigma, + add_one=False, + ) + self.bn = FastBatchNorm1d(out_channels, momentum=bn_momentum) + self.activation = nn.LeakyReLU(negative_slope=negative_slope) + + def forward(self, feats, xyz, batch, neighbor_idx): + # feats: [N, C] + # coords: [N, 3] + # batch: [N,] + # neighbor_idx: [N, M] + + feats = self.kpconv(xyz, xyz, neighbor_idx, feats) + feats = self.activation(self.bn(feats)) + return feats + + +class KPConvResBlock(nn.Module): + def __init__( + self, + in_channels, + out_channels, + prev_grid_size, + sigma=1.0, + negative_slope=0.2, + bn_momentum=0.02, + ): + super().__init__() + d_2 = out_channels // 4 + activation = nn.LeakyReLU(negative_slope=negative_slope) + self.unary_1 = torch.nn.Sequential( + nn.Linear(in_channels, d_2, bias=False), + FastBatchNorm1d(d_2, momentum=bn_momentum), + activation, + ) + self.unary_2 = torch.nn.Sequential( + nn.Linear(d_2, out_channels, bias=False), + FastBatchNorm1d(out_channels, momentum=bn_momentum), + activation, + ) + self.kpconv = KPConvLayer( + d_2, d_2, point_influence=prev_grid_size * sigma, add_one=False + ) + self.bn = FastBatchNorm1d(out_channels, momentum=bn_momentum) + self.activation = activation + + if in_channels != out_channels: + self.shortcut_op = torch.nn.Sequential( + nn.Linear(in_channels, out_channels, bias=False), + FastBatchNorm1d(out_channels, momentum=bn_momentum), + ) + else: + self.shortcut_op = nn.Identity() + + def forward(self, feats, xyz, batch, neighbor_idx): + # feats: [N, C] + # coords: [N, 3] + # batch: [N,] + # neighbor_idx: [N, M] + + shortcut = feats + feats = self.unary_1(feats) + feats = self.kpconv(xyz, xyz, neighbor_idx, feats) + feats = self.unary_2(feats) + shortcut = self.shortcut_op(shortcut) + feats += shortcut + return feats + + +@MODELS.register_module("ST-v1m2") +class StratifiedTransformer(nn.Module): + def __init__( + self, + in_channels, + num_classes, + channels=(48, 96, 192, 384, 384), + num_heads=(6, 12, 24, 24), + depths=(3, 9, 3, 3), + window_size=(0.2, 0.4, 0.8, 1.6), + quant_size=(0.01, 0.02, 0.04, 0.08), + mlp_expend_ratio=4.0, + down_ratio=0.25, + down_num_sample=16, + kp_ball_radius=2.5 * 0.02, + kp_max_neighbor=34, + kp_grid_size=0.02, + kp_sigma=1.0, + drop_path_rate=0.2, + rel_query=True, + rel_key=True, + rel_value=True, + qkv_bias=True, + stem=True, + ): + super().__init__() + assert ( + KPConvLayer is not None and FastBatchNorm1d is not None + ), "Please make sure torch_points3d is installed" + assert tp is not None, "Please make sure torch_points_kernels is installed" + assert pointops is not None, "Please make sure pointops2 is installed" + # stochastic depth decay rule + dpr = [x.item() for x in torch.linspace(0, drop_path_rate, sum(depths))] + self.kp_ball_radius = kp_ball_radius + self.kp_max_neighbor = kp_max_neighbor + self.stem = stem + if stem: + self.point_embed = nn.ModuleList( + [ + KPConvSimpleBlock( + in_channels, channels[0], kp_grid_size, sigma=kp_sigma + ), + KPConvResBlock( + channels[0], channels[0], kp_grid_size, sigma=kp_sigma + ), + ] + ) + self.down = TransitionDown( + channels[0], channels[1], down_ratio, down_num_sample + ) + else: + assert channels[0] == channels[1] + self.point_embed = nn.ModuleList( + [ + KPConvSimpleBlock( + in_channels, channels[1], kp_grid_size, sigma=kp_sigma + ), + ] + ) + + num_layers = len(depths) + self.layers = nn.ModuleList() + for i in range(num_layers): + layer = BasicLayer( + embed_channels=channels[i + 1], + out_channels=channels[i + 2] if i < num_layers - 1 else channels[i + 1], + depth=depths[i], + num_heads=num_heads[i], + window_size=window_size[i], + quant_size=quant_size[i], + mlp_expend_ratio=mlp_expend_ratio, + down_ratio=down_ratio, + down_num_sample=down_num_sample, + drop_path=dpr[sum(depths[:i]) : sum(depths[: i + 1])], + rel_query=rel_query, + rel_key=rel_key, + rel_value=rel_value, + qkv_bias=qkv_bias, + down=True if i < num_layers - 1 else False, + ) + self.layers.append(layer) + + self.up = nn.ModuleList( + [ + TransitionUp(channels[i + 1], channels[i]) + for i in reversed(range(1, num_layers)) + ] + ) + if self.stem: + self.up.append(TransitionUp(channels[1], channels[0])) + + self.classifier = nn.Sequential( + nn.Linear(channels[0], channels[0]), + nn.BatchNorm1d(channels[0]), + nn.ReLU(inplace=True), + nn.Linear(channels[0], num_classes), + ) + + self.init_weights() + + def forward(self, data_dict): + feats = data_dict["feat"] + coords = data_dict["coord"] + offset = data_dict["offset"].int() + batch = offset2batch(offset) + neighbor_idx = tp.ball_query( + self.kp_ball_radius, + self.kp_max_neighbor, + coords, + coords, + mode="partial_dense", + batch_x=batch, + batch_y=batch, + )[0] + + feats_stack = [] + coords_stack = [] + offset_stack = [] + + for i, layer in enumerate(self.point_embed): + feats = layer(feats, coords, batch, neighbor_idx) + + feats = feats.contiguous() + if self.stem: + feats_stack.append(feats) + coords_stack.append(coords) + offset_stack.append(offset) + feats, coords, offset = self.down(feats, coords, offset) + + for i, layer in enumerate(self.layers): + feats, coords, offset, feats_down, coords_down, offset_down = layer( + feats, coords, offset + ) + + feats_stack.append(feats) + coords_stack.append(coords) + offset_stack.append(offset) + + feats = feats_down + coords = coords_down + offset = offset_down + + feats = feats_stack.pop() + coords = coords_stack.pop() + offset = offset_stack.pop() + + for i, up in enumerate(self.up): + feats, coords, offset = up( + feats, + coords, + offset, + feats_stack.pop(), + coords_stack.pop(), + offset_stack.pop(), + ) + + out = self.classifier(feats) + return out + + def init_weights(self): + """Initialize the weights in backbone.""" + + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.LayerNorm) or isinstance(m, nn.BatchNorm1d): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + self.apply(_init_weights) diff --git a/pointcept/models/swin3d/__init__.py b/pointcept/models/swin3d/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..36050969d9abb027778008e4d6d8f77710f52392 --- /dev/null +++ b/pointcept/models/swin3d/__init__.py @@ -0,0 +1 @@ +from .swin3d_v1m1_base import Swin3DUNet diff --git a/pointcept/models/swin3d/mink_layers.py b/pointcept/models/swin3d/mink_layers.py new file mode 100644 index 0000000000000000000000000000000000000000..ee3e8cfc002e8311ac196335592c337644659612 --- /dev/null +++ b/pointcept/models/swin3d/mink_layers.py @@ -0,0 +1,249 @@ +""" +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +""" + +import torch +import torch.nn as nn +import torch.nn.functional as F +import MinkowskiEngine as ME +import numpy as np + + +def assign_feats(sp, x): + return ME.SparseTensor( + features=x.float(), + coordinate_map_key=sp.coordinate_map_key, + coordinate_manager=sp.coordinate_manager, + ) + + +class MinkConvBN(nn.Module): + def __init__( + self, + in_channels, + out_channels, + kernel_size=3, + stride=1, + dilation=1, + bias=False, + dimension=3, + ): + super().__init__() + self.conv_layers = nn.Sequential( + ME.MinkowskiConvolution( + in_channels=in_channels, + out_channels=out_channels, + kernel_size=kernel_size, + stride=stride, + dilation=dilation, + bias=bias, + dimension=dimension, + ), + ME.MinkowskiBatchNorm(out_channels), + ) + + def forward(self, x): + x = self.conv_layers(x) + return x + + +class MinkConvBNRelu(nn.Module): + def __init__( + self, + in_channels, + out_channels, + kernel_size=3, + stride=1, + dilation=1, + bias=False, + dimension=3, + ): + super().__init__() + self.conv_layers = nn.Sequential( + ME.MinkowskiConvolution( + in_channels=in_channels, + out_channels=out_channels, + kernel_size=kernel_size, + stride=stride, + dilation=dilation, + bias=bias, + dimension=dimension, + ), + ME.MinkowskiBatchNorm(out_channels), + ME.MinkowskiReLU(inplace=True), + ) + + def forward(self, x): + x = self.conv_layers(x) + if x.F.dtype == torch.float16: + x = assign_feats(x, x.F.float()) + return x + + +class MinkDeConvBNRelu(nn.Module): + def __init__( + self, + in_channels, + out_channels, + kernel_size, + stride, + dilation=1, + bias=False, + dimension=3, + ): + super().__init__() + self.conv_layers = nn.Sequential( + ME.MinkowskiConvolutionTranspose( + in_channels=in_channels, + out_channels=out_channels, + kernel_size=kernel_size, + stride=stride, + dilation=dilation, + bias=bias, + dimension=dimension, + ), + ME.MinkowskiBatchNorm(out_channels), + ME.MinkowskiReLU(), + ) + + def forward(self, x): + x = self.conv_layers(x) + return x + + +class MinkResBlock(nn.Module): + def __init__(self, in_channels, out_channels, stride=1, dilation=1): + super(MinkResBlock, self).__init__() + + self.conv1 = ME.MinkowskiConvolution( + in_channels=in_channels, + out_channels=out_channels, + kernel_size=3, + stride=stride, + dilation=dilation, + bias=False, + dimension=3, + ) + self.norm1 = ME.MinkowskiBatchNorm(out_channels) + self.conv2 = ME.MinkowskiConvolution( + in_channels=out_channels, + out_channels=out_channels, + kernel_size=3, + stride=1, + dilation=dilation, + bias=False, + dimension=3, + ) + + self.norm2 = ME.MinkowskiBatchNorm(out_channels) + self.relu = ME.MinkowskiReLU(inplace=True) + + def forward(self, x): + residual = x + + out = self.conv1(x) + out = self.norm1(out) + out = self.relu(out) + + out = self.conv2(out) + out = self.norm2(out) + + out += residual + out = self.relu(out) + + return out + + +class SparseTensorLinear(nn.Module): + def __init__(self, in_channels, out_channels, bias=False): + super().__init__() + self.linear = nn.Linear(in_channels, out_channels, bias=bias) + + def forward(self, sp): + x = self.linear(sp.F) + return assign_feats(sp, x.float()) + + +class SparseTensorLayerNorm(nn.Module): + def __init__(self, dim): + super().__init__() + self.norm = nn.LayerNorm(dim) + + def forward(self, sp): + x = self.norm(sp.F) + return assign_feats(sp, x.float()) + + +class MinkResBlock_v2(nn.Module): + def __init__(self, in_channels, out_channels): + super().__init__() + d_2 = out_channels // 4 + self.conv1 = torch.nn.Sequential( + SparseTensorLinear(in_channels, d_2, bias=False), + ME.MinkowskiBatchNorm(d_2), + ME.MinkowskiReLU(), + ) + self.unary_2 = torch.nn.Sequential( + SparseTensorLinear(d_2, out_channels, bias=False), + ME.MinkowskiBatchNorm(out_channels), + ME.MinkowskiReLU(), + ) + self.spconv = ME.MinkowskiConvolution( + in_channels=d_2, + out_channels=d_2, + kernel_size=5, + stride=1, + dilation=1, + bias=False, + dimension=3, + ) + if in_channels != out_channels: + self.shortcut_op = torch.nn.Sequential( + SparseTensorLinear(in_channels, out_channels, bias=False), + ME.MinkowskiBatchNorm(out_channels), + ) + else: + self.shortcut_op = nn.Identity() + + def forward(self, x): + # feats: [N, C] + # xyz: [N, 3] + # batch: [N,] + # neighbor_idx: [N, M] + shortcut = x + x = self.unary_1(x) + x = self.spconv(x) + x = self.unary_2(x) + shortcut = self.shortcut_op(shortcut) + x += shortcut + return x + + +class MinkResBlock_BottleNeck(nn.Module): + def __init__(self, in_channels, out_channels): + super(MinkResBlock_BottleNeck, self).__init__() + bottle_neck = out_channels // 4 + self.conv1x1a = MinkConvBNRelu( + in_channels, bottle_neck, kernel_size=1, stride=1 + ) + self.conv3x3 = MinkConvBNRelu(bottle_neck, bottle_neck, kernel_size=3, stride=1) + self.conv1x1b = MinkConvBN(bottle_neck, out_channels, kernel_size=1, stride=1) + if in_channels != out_channels: + self.conv1x1c = MinkConvBN( + in_channels, out_channels, kernel_size=1, stride=1 + ) + else: + self.conv1x1c = None + self.relu = ME.MinkowskiReLU(inplace=True) + + def forward(self, x): + residual = x + out = self.conv1x1a(x) + out = self.conv3x3(out) + out = self.conv1x1b(out) + if self.conv1x1c is not None: + residual = self.conv1x1c(residual) + out = self.relu(out + residual) + + return out diff --git a/pointcept/models/swin3d/swin3d_layers.py b/pointcept/models/swin3d/swin3d_layers.py new file mode 100644 index 0000000000000000000000000000000000000000..04809de8b873242d0eae2cf1a1774957ee1794ef --- /dev/null +++ b/pointcept/models/swin3d/swin3d_layers.py @@ -0,0 +1,876 @@ +""" +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +""" + +import numpy as np +import torch +import torch.nn as nn +from timm.layers import DropPath, trunc_normal_ +import MinkowskiEngine as ME +from MinkowskiEngine import SparseTensor +from Swin3D.sparse_dl.attn.attn_coff import ( + SelfAttnAIOFunction, + PosEmb, + TableDims, + IndexMode, + PrecisionMode, +) +import Swin3D.sparse_dl.knn +from Swin3D.sparse_dl.knn import KNN + +from .mink_layers import ( + assign_feats, + SparseTensorLayerNorm, + SparseTensorLinear, +) + + +def query_knn_feature( + K, src_xyz, query_xyz, src_feat, src_offset, query_offset, return_idx=False +): + """ + gather feature in the KNN neighborhood + """ + assert ( + src_xyz.is_contiguous() + and query_xyz.is_contiguous() + and src_feat.is_contiguous() + ) + if query_xyz is None: + query_xyz = src_xyz + query_offset = src_offset + + idx, _ = KNN.apply(K, src_xyz, query_xyz, src_offset, query_offset) + + n, m, c = src_xyz.shape[0], query_xyz.shape[0], src_feat.shape[1] + grouped_feat = src_feat[idx.view(-1).long(), :].view(m, K, c) + + if return_idx: + return grouped_feat, idx + else: + return grouped_feat + + +def knn_linear_interpolation( + src_xyz, query_xyz, src_feat, src_offset, query_offset, K=3 +): + """ + interpolation feature using distance in KNN neighborhood + """ + N, C = query_xyz.shape[0], src_feat.shape[1] + assert ( + src_xyz.is_contiguous() + and query_xyz.is_contiguous() + and src_feat.is_contiguous() + ) + # (N, K) + idx, dist = KNN.apply(K, src_xyz, query_xyz, src_offset, query_offset) + weight = 1.0 / (dist + 1e-8) + norm = torch.sum(weight, dim=1, keepdim=True) + weight = weight / norm + query_feat = torch.zeros((N, C), dtype=src_feat.dtype, device=src_feat.device) + for i in range(K): + query_feat += src_feat[idx[:, i].long(), :] * weight[:, i].unsqueeze(-1) + return query_feat + + +def sparse_self_attention( + w_w_id: torch.Tensor, w_sizes: torch.Tensor, protocol: str = "v1" +): + """ + Args: + indices [torch.Tensor]: sparse window index with shape [N, 2], N is the total + number of non-empty voxels with indices (window_id, within_window_id). window_id + is ordered and starts from 0; within_window_id is a sparse index to indicate the + offset of kernel_size ** 3. + feats [torch.Tensor]: sprase features of each non-empty voxel with shape [N, C] + Outputs: + [M, 3]: sparse indices of cofficient matrix (window_id, att_a_id, att_b_id). att_a_id + and att_b_id are the within_window_id + [M, 1]: the sparse coffient matrix + + Spaces: + W: total number of windows + N: total number of input voxels + M: total number of output cofficients + """ + w_sizes_2 = w_sizes**2 + + # w2n_indices - [W], mapping window index to window global offset in input + # space + w_cumsum = torch.cumsum(w_sizes, dim=-1) + w2n_indices = torch.cat( + [torch.zeros(1, dtype=w_cumsum.dtype, device=w_cumsum.device), w_cumsum[:-1]] + ) + + # w2m indices - [W], mapping window index to window global offset in output + # space + w2_cumsum = torch.cumsum(w_sizes_2, dim=-1) + w2m_indices = torch.cat( + [torch.zeros(1, dtype=w2_cumsum.dtype, device=w2_cumsum.device), w2_cumsum[:-1]] + ) + + # m2w indices - [M], mapping element global offset to the window index + m2w_indices = torch.zeros( + [w2_cumsum[-1]], dtype=w_sizes.dtype, device=w_sizes.device + ) + m2w_offset = torch.zeros( + [w2_cumsum[-1]], dtype=w_sizes.dtype, device=w_sizes.device + ) + m2w_indices[w2m_indices[1:]] = 1 + m2w_offset[w2m_indices[1:]] = w_sizes_2[:-1] + m2w_indices = torch.cumsum(m2w_indices, dim=-1) + m2w_offset = torch.cumsum(m2w_offset, dim=-1) + + # m_indices = [M], element global offset in output space + m_indices = torch.arange( + 0, w2_cumsum[-1], dtype=w_sizes.dtype, device=w_sizes.device + ) + + # m2n_indices - [M], mapping element global offset to the window global offset + # in input space + m2n_indices = w2n_indices[m2w_indices] + + m_offset = m_indices - m2w_offset + m2w_sizes = w_sizes[m2w_indices] + + # print_log_main("m_offset:", m_offset, m_offset.shape) + # print_log_main("m2n_indices:", m2n_indices, m2n_indices.shape) + + y_offset = m2n_indices + m_offset % m2w_sizes + x_offset = m2n_indices + torch.div(m_offset, m2w_sizes, rounding_mode="floor") + + # print_log_main("=================================") + # print_log_main(w_sizes[:5]) + # print_log_main(x_offset[:50]) + # print_log_main(y_offset[:50]) + # coord = torch.stack([m2w_indices, w_w_id[x_offset], w_w_id[y_offset]], axis=-1) + if protocol == "v1": + return x_offset, y_offset + elif protocol == "v2": + return x_offset, y_offset, m2w_indices, w_sizes, w2n_indices, w2m_indices + + +class Mlp(nn.Module): + def __init__( + self, + in_features, + hidden_features=None, + out_features=None, + act_layer=nn.GELU, + drop=0.0, + ): + super().__init__() + out_features = out_features or in_features + hidden_features = hidden_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.act = act_layer() + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop = nn.Dropout(drop) + + def forward(self, x): + x = self.fc1(x) + x = self.act(x) + x = self.drop(x) + x = self.fc2(x) + x = self.drop(x) + return x + + +class GridCoordsDown(nn.Module): + """ + downsample the grid coordinates + keep the nearest point to the average point of the downsampled grid + """ + + def __init__(self, stride): + super().__init__() + self.stride = stride + self.avg_pool = ME.MinkowskiAvgPooling( + kernel_size=self.stride, stride=self.stride, dimension=3 + ) + self.unpool = ME.MinkowskiPoolingTranspose( + kernel_size=stride, stride=stride, dimension=3 + ) + self.max_pool = ME.MinkowskiMaxPooling( + kernel_size=self.stride, stride=self.stride, dimension=3 + ) + + def forward(self, coords_sp, sp, return_map=False): + device = sp.C.device + # is_pool = True means pooling map + # is_pool = False means conv map (query as center) + + N = sp.shape[0] + avg_coords_sp = self.avg_pool(coords_sp) + dist_sp = self.unpool(avg_coords_sp) - coords_sp + dist = dist_sp.F + dist = -torch.sqrt((dist**2).sum(dim=1)).unsqueeze(1) + dist_sp = assign_feats(dist_sp, dist) + min_dist_sp = self.max_pool(dist_sp) + map_pair = sp.coordinate_manager.kernel_map( + dist_sp.coordinate_map_key, + min_dist_sp.coordinate_map_key, + stride=self.stride, + kernel_size=self.stride, + is_pool=True, + )[0] + in_map, out_map = map_pair + broad_min_dist_sp = self.unpool(min_dist_sp) + mask = (broad_min_dist_sp.F == dist_sp.F).squeeze(1) + in_map = in_map[mask].long() + out_map = out_map[mask].long() + downsample_map = torch.zeros(N, dtype=torch.long, device=device) - 1 + downsample_map[out_map] = in_map + assert (downsample_map >= 0).all() + assert (dist_sp.F[downsample_map] == min_dist_sp.F).all() + new_coords = coords_sp.F[downsample_map] + new_coords_sp = assign_feats(sp, new_coords) + if return_map: + return new_coords_sp, downsample_map + else: + return new_coords_sp + + +def get_offset(batch): + offset = [] + bs = batch.max() + 1 + for i in range(bs): + offset.append(torch.sum(batch == i)) + offset = torch.cuda.IntTensor(offset) + offset = offset.cumsum(dim=0).int() + return offset + + +class GridDownsample(nn.Module): + """ + use stride to downsample voxel + use grid maxpooling with kernel_size + """ + + def __init__(self, in_channels, out_channels, kernel_size=2, stride=2): + super().__init__() + self.kernel_size = kernel_size + self.stride = stride + self.in_channels = in_channels + self.out_channels = out_channels + self.sp_pool = ME.MinkowskiMaxPooling( + kernel_size=kernel_size, stride=stride, dimension=3 + ) + self.coords_pool = GridCoordsDown(stride=stride) + self.norm = SparseTensorLayerNorm(in_channels) + self.linear = SparseTensorLinear(in_channels, out_channels) + + def forward(self, sp, coords_sp): + sp_down = self.sp_pool(self.linear(self.norm(sp))) + coords_sp_down = self.coords_pool(coords_sp, sp_down) + return sp_down, coords_sp_down + + def extra_repr(self) -> str: + return f"kernel_size={self.kernel_size}, stride={self.stride}, in_channels={self.in_channels}, out_channels={self.out_channels}" + + +class GridKNNDownsample(nn.Module): + """ + use stride to downsample voxel + use KNN to do maxpooling + """ + + def __init__(self, in_channels, out_channels, kernel_size=2, stride=2): + super().__init__() + self.stride = stride + self.in_channels = in_channels + self.out_channels = out_channels + self.k = 16 + self.sp_pool = ME.MinkowskiMaxPooling( + kernel_size=stride, stride=stride, dimension=3 + ) + self.coords_pool = GridCoordsDown(stride=stride) + self.norm = nn.LayerNorm(in_channels) + self.linear = nn.Linear(in_channels, out_channels, bias=False) + self.pool = nn.MaxPool1d(self.k) + + def forward(self, sp, coords_sp): + # calculate the voxel + sp_down = self.sp_pool(sp) + # for downsampled cRSE + coords_sp_down = self.coords_pool(coords_sp, sp_down) + offset = get_offset(sp.C[:, 0]) + n_offset = get_offset(sp_down.C[:, 0]) + + xyz = coords_sp.F[:, 1:4].detach().contiguous() + n_xyz = coords_sp_down.F[:, 1:4].detach().contiguous() + feats = query_knn_feature(self.k, xyz, n_xyz, sp.F, offset, n_offset) + m, k, c = feats.shape + feats = ( + self.linear(self.norm(feats.view(m * k, c)).view(m, k, c)) + .transpose(1, 2) + .contiguous() + ) + feats = self.pool(feats).squeeze(-1) + sp = assign_feats(sp_down, feats.float()) + coords_sp = coords_sp_down + return sp, coords_sp + + def extra_repr(self) -> str: + return f"kernel_size={self.k}, stride={self.stride}, in_channels={self.in_channels}, out_channels={self.out_channels}" + + +class Upsample(nn.Module): + """ + upsample using trilinear interpolation + follower by attn block according to self.attn + """ + + def __init__( + self, + in_channels, + out_channels, + num_heads, + window_size, + quant_size, + attn=True, + up_k=3, + cRSE="XYZ_RGB", + fp16_mode=0, + ): + super().__init__() + self.in_channels = in_channels + self.out_channels = out_channels + + self.linear1 = nn.Sequential( + nn.LayerNorm(out_channels), nn.Linear(out_channels, out_channels) + ) + self.linear2 = nn.Sequential( + nn.LayerNorm(in_channels), nn.Linear(in_channels, out_channels) + ) + self.up_k = up_k + self.attn = attn and window_size > 0 + if self.attn: + self.block = BasicLayer( + dim=out_channels, + depth=1, + num_heads=num_heads, + window_size=window_size, + quant_size=quant_size, + drop_path=0.1, + downsample=None, + out_channels=None, + cRSE=cRSE, + fp16_mode=fp16_mode, + ) + + def forward(self, sp, coords_sp, sp_up, coords_sp_up): + feats = sp.F + support_feats = sp_up.F + xyz = coords_sp.F[:, 1:4].detach().contiguous() + support_xyz = coords_sp_up.F[:, 1:4].detach().contiguous() + offset = get_offset(sp.C[:, 0]) + support_offset = get_offset(sp_up.C[:, 0]) + + feats = self.linear1(support_feats) + knn_linear_interpolation( + xyz, support_xyz, self.linear2(feats), offset, support_offset, K=self.up_k + ) + sp_up = assign_feats(sp_up, feats) + if self.attn: + sp_up, _, _ = self.block(sp_up, coords_sp_up) + return sp_up + + def extra_repr(self) -> str: + return f"up_k={self.up_k}, in_channels={self.in_channels}, out_channels={self.out_channels}, attn={self.attn}" + + +class WindowAttention(nn.Module): + """ + Window based multi-head self attention (W-MSA) module with cRSE. + Designed for sparse structure + It supports both of shifted and non-shifted window. + + Args: + dim (int): Number of input channels. + window_size (tuple[int]): The height and width of the window. + quant_size (int): quant_size for for finer cRSE table + num_heads (int): Number of attention heads. + qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True + qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set + attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0 + proj_drop (float, optional): Dropout ratio of output. Default: 0.0 + cRSE (str | 'XYZ', 'XYZ_RGB', 'XYZ_RGB_NORM'): cRSE mode. Default: 'XYZ_RGB' + fp16_mode (int | 0, 1, 2): fp16 mode for attention module, Default: 0 + 0: fp32 forward and fp32 backward + 1: fp16 forward and fp32 backward + 2: fp16 forward and fp16 backward + """ + + def __init__( + self, + dim, + window_size, + quant_size, + num_heads, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + cRSE="XYZ_RGB", + fp16_mode=0, + ): + super().__init__() + self.dim = dim + self.window_size = window_size + self.num_heads = num_heads + head_dim = dim // num_heads + self.scale = qk_scale or head_dim**-0.5 + + # color in [-1, 1], color_windowsize = 2 + # normal in [-1, 1], normal_windowsize = 2 + self.color_windowsize = 2 + self.normal_windowsize = 2 + + self.fp16_mode = fp16_mode + + table_offsets = [] + self.cRSE = cRSE + if "XYZ" in cRSE: + self.xyz_quant_size = quant_size + quant_grid_length_xyz = window_size * self.xyz_quant_size + table_shape_xyz = (3, 2 * quant_grid_length_xyz, num_heads, head_dim) + self.query_xyz_table = nn.Parameter(torch.zeros(table_shape_xyz)) + trunc_normal_(self.query_xyz_table, std=0.02) + self.key_xyz_table = nn.Parameter(torch.zeros(table_shape_xyz)) + trunc_normal_(self.key_xyz_table, std=0.02) + self.value_xyz_table = nn.Parameter(torch.zeros(table_shape_xyz)) + trunc_normal_(self.value_xyz_table, std=0.02) + table_offsets += [np.prod(table_shape_xyz[1:])] * 3 + + if "RGB" in cRSE: + self.color_quant_size = quant_size * 2 + quant_grid_length_rgb = self.color_windowsize * self.color_quant_size + table_shape_rgb = (3, 2 * quant_grid_length_rgb, num_heads, head_dim) + self.query_rgb_table = nn.Parameter(torch.zeros(table_shape_rgb)) + trunc_normal_(self.query_rgb_table, std=0.02) + self.key_rgb_table = nn.Parameter(torch.zeros(table_shape_rgb)) + trunc_normal_(self.key_rgb_table, std=0.02) + self.value_rgb_table = nn.Parameter(torch.zeros(table_shape_rgb)) + trunc_normal_(self.value_rgb_table, std=0.02) + table_offsets += [np.prod(table_shape_rgb[1:])] * 3 + + if "NORM" in cRSE: + self.normal_quant_size = quant_size * 2 + quant_grid_length_norm = self.normal_windowsize * self.normal_quant_size + table_shape_norm = (3, 2 * quant_grid_length_norm, num_heads, head_dim) + self.query_norm_table = nn.Parameter(torch.zeros(table_shape_norm)) + trunc_normal_(self.query_norm_table, std=0.02) + self.key_norm_table = nn.Parameter(torch.zeros(table_shape_norm)) + trunc_normal_(self.key_norm_table, std=0.02) + self.value_norm_table = nn.Parameter(torch.zeros(table_shape_norm)) + trunc_normal_(self.value_norm_table, std=0.02) + table_offsets += [np.prod(table_shape_norm[1:])] * 3 + + self.table_offsets = table_offsets + + self.quant_size = quant_size + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop, inplace=True) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop, inplace=True) + + self.softmax = nn.Softmax(dim=-1) + + def forward(self, feats: torch.Tensor, attn_args): + """Forward function. + + Args: + feats: N, C + attn_args: arguments for computing attention + """ + num_v, _ = feats.shape + num_sc = self.dim // self.num_heads + + ( + x_offset, + y_offset, + m2w_indices, + w_sizes, + w2n_indices, + n2n_indices, + w2m_indices, + n_coords, + ) = attn_args + + # Query, Key, Value + qkv = self.qkv(feats) + qkv = ( + qkv.reshape(num_v, 3, self.num_heads, num_sc) + .permute(1, 0, 2, 3) + .contiguous() + ) + query, key, value = qkv[0], qkv[1], qkv[2] # [N, num_heads, C//num_heads] + query = query * self.scale + + table_offsets = torch.IntTensor(self.table_offsets).cuda() + query_table, key_table, value_table = [], [], [] + n_cRSE = [] + if "XYZ" in self.cRSE: + n_xyz = n_coords[:, 0:3] + n_xyz = n_xyz * self.quant_size + n_cRSE.append(n_xyz) + query_table.append(self.query_xyz_table.view(-1)) + key_table.append(self.key_xyz_table.view(-1)) + value_table.append(self.value_xyz_table.view(-1)) + if "RGB" in self.cRSE: + n_rgb = n_coords[:, 3:6] + n_rgb = n_rgb * self.color_quant_size + n_cRSE.append(n_rgb) + query_table.append(self.query_rgb_table.view(-1)) + key_table.append(self.key_rgb_table.view(-1)) + value_table.append(self.value_rgb_table.view(-1)) + if "NORM" in self.cRSE: + n_norm = n_coords[:, 6:9] + n_norm = n_norm * self.normal_quant_size + n_cRSE.append(n_norm) + query_table.append(self.query_norm_table.view(-1)) + key_table.append(self.key_norm_table.view(-1)) + value_table.append(self.value_norm_table.view(-1)) + + n_cRSE = torch.cat(n_cRSE, dim=1) + + indices = [m2w_indices, w_sizes, w2m_indices, w2n_indices, n2n_indices, n_cRSE] + query_table = torch.cat(query_table) + key_table = torch.cat(key_table) + value_table = torch.cat(value_table) + + if self.fp16_mode == 0: + # do not use fp16 + # cast q,k,v to fp32 in forward and backward + fp16_mode = PrecisionMode.HALF_NONE + elif self.fp16_mode == 1: + # use fp16 only in forward + fp16_mode = PrecisionMode.HALF_FORWARD + elif self.fp16_mode == 2: + # use fp16 both in forward and backward + fp16_mode = PrecisionMode.HALF_ALL + + updated_values = SelfAttnAIOFunction.apply( + query, + key, + value, + query_table, + key_table, + value_table, + table_offsets, + indices, + PosEmb.SEPARATE, + TableDims.D0, + IndexMode.INDIRECT, + fp16_mode, + ) + + updated_values = updated_values.flatten(1) + updated_feats = updated_values.view(num_v, self.dim) + + updated_feats = self.proj(updated_feats) + updated_feats = self.proj_drop(updated_feats) # [N, C] + + return updated_feats + + +class SwinTransformerBlock(nn.Module): + def __init__( + self, + dim, + num_heads, + window_size, + quant_size, + drop_path=0.0, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + act_layer=nn.GELU, + norm_layer=nn.LayerNorm, + cRSE="XYZ_RGB", + fp16_mode=0, + ): + super().__init__() + self.window_size = window_size + + self.norm1 = norm_layer(dim) + self.attn = WindowAttention( + dim, + window_size=self.window_size, + quant_size=quant_size, + num_heads=num_heads, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + cRSE=cRSE, + fp16_mode=fp16_mode, + ) + + self.drop_path = DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + self.norm2 = norm_layer(dim) + mlp_hidden_dim = int(dim * mlp_ratio) + self.mlp = Mlp( + in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer + ) + + def forward(self, feats, attn_args): + # feats: [N, c] + short_cut = feats + feats = self.norm1(feats) + feats = self.attn(feats, attn_args) # [N, c] + + feats = short_cut + self.drop_path(feats) + feats = feats + self.drop_path(self.mlp(self.norm2(feats))) + + return feats + + +class BasicLayer(nn.Module): + """A basic Swin3D layer for one stage. + + Args: + dim (int): Number of input channels. + depth (int): Number of blocks. + num_heads (int): Number of attention heads. + window_size (int): Local window size. + quant_size (int): quant_size for for finer cRSE table + mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. + qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True + qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set. + drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0 + norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm + downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None + cRSE (str | 'XYZ', 'XYZ_RGB', 'XYZ_RGB_NORM'): cRSE mode. Default: 'XYZ_RGB' + fp16_mode (int | 0, 1, 2): fp16 mode for attention module, Default: 0 + 0: fp32 forward and fp32 backward + 1: fp16 forward and fp32 backward + 2: fp16 forward and fp16 backward + """ + + def __init__( + self, + dim, + depth, + num_heads, + window_size, + quant_size, + out_channels=None, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + drop_path=0.0, + norm_layer=nn.LayerNorm, + downsample=None, + down_stride=2, + cRSE="XYZ_RGB", + fp16_mode=0, + ): + super().__init__() + self.window_size = window_size + self.depth = depth + self.dim = dim + self.num_heads = num_heads + self.quant_size = quant_size + self.cRSE = cRSE + self.fp16_mode = fp16_mode + + self.shift_size = window_size // 2 + # build blocks + self.blocks = nn.ModuleList( + [ + SwinTransformerBlock( + dim, + num_heads, + window_size, + quant_size, + drop_path=( + drop_path[i] if isinstance(drop_path, list) else drop_path + ), + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + norm_layer=norm_layer, + cRSE=cRSE, + fp16_mode=fp16_mode, + ) + for i in range(depth) + ] + ) + + self.pool = ME.MinkowskiMaxPooling( + kernel_size=self.window_size, stride=self.window_size, dimension=3 + ) + + if downsample is not None: + if out_channels is None: + out_channels = dim * 2 + self.downsample = downsample( + dim, out_channels, kernel_size=down_stride, stride=down_stride + ) + else: + self.downsample = None + + def get_map_pair(self, sp): + """ + use minkowski pool to calculate windows + get the mapping from voxel to window + """ + window_size = [self.window_size] * 3 + pool_sp = self.pool(sp) + windows = pool_sp.C + window_N = windows.shape[0] + + stride_in = sp.coordinate_map_key.get_tensor_stride() + x, y, z = [ + torch.arange(window_size[i], device=self.device) * stride_in[i] + for i in range(3) + ] + x, y, z = torch.meshgrid(x, y, z) + i = torch.zeros_like(x, device=self.device) + local_window = torch.stack([i, x, y, z], dim=-1).flatten(0, -2) + all_windows = windows.unsqueeze(1) + local_window.unsqueeze(0) + all_windows = all_windows.flatten(0, -2).int() + cm = sp.coordinate_manager + query_key, (map, inverse_map) = cm.insert_and_map( + all_windows, tensor_stride=stride_in + ) + map_pair = cm.kernel_map(query_key, sp.coordinate_map_key, kernel_size=1)[0] + return map_pair, window_N + + def get_window_mapping(self, sp): + """ + calculate the relationshape in the window: + w_w_id: non-empty idx inside the window(sorted by window) + w_w_xyz: xyz inside the window(sorted by window) + nempty_num: non-empty voxel number in each window + sort_idx: sort voxel according to window_id, to gather the point inside the same window + inv_sort_idx: inverse sort index + """ + map_pair, window_N = self.get_map_pair(sp) + window_size = self.window_size + nW = window_size**3 + in_map, out_map = map_pair + in_map, sort_idx = torch.sort(in_map) + # assert out_map == arange(out_map.shape[0]) + out_map = out_map[sort_idx] + sort_idx = out_map.long() + inv_sort_idx = torch.zeros_like(sort_idx) + inv_sort_idx[sort_idx] = torch.arange( + sort_idx.shape[0], dtype=sort_idx.dtype, device=self.device + ) + N = window_N * nW + v2w_mask = torch.zeros(N, dtype=torch.bool, device=self.device) + w_id = ( + torch.arange(window_N, dtype=torch.long, device=self.device) + .unsqueeze(1) + .repeat(1, nW) + .view(-1) + ) + w_w_id = ( + torch.arange(nW, dtype=torch.long, device=self.device) + .unsqueeze(0) + .repeat(window_N, 1) + .view(-1) + ) + v2w_mask[in_map.long()] = True + nempty_num = v2w_mask.view(-1, nW).sum(dim=-1) + w_id = w_id[in_map.long()] + w_w_id = w_w_id[in_map.long()] + w_w_xyz = torch.stack( + [ + w_w_id // window_size // window_size, + w_w_id // window_size % window_size, + w_w_id % window_size, + ], + dim=-1, + ) + return w_w_id, w_w_xyz, nempty_num, sort_idx, inv_sort_idx + + def get_index01(self, sp, local_xyz, colors): + """ + calculate the arguments for sparse attention + """ + ( + w_w_id, + w_w_xyz, + nempty_num, + n2n_indices, + inv_sort_idx, + ) = self.get_window_mapping(sp) + local_xyz = local_xyz[n2n_indices] + colors = colors[n2n_indices] + # recover the relative pos in the voxel + n_coords = w_w_xyz + local_xyz + n_coords = torch.cat([n_coords, colors], dim=1) + ( + x_offset, + y_offset, + m2w_indices, + w_sizes, + w2n_indices, + w2m_indices, + ) = sparse_self_attention(w_w_id, nempty_num, protocol="v2") + return ( + x_offset, + y_offset, + m2w_indices, + w_sizes, + w2n_indices, + n2n_indices, + w2m_indices, + n_coords, + ) + + def get_shifted_sp(self, sp): + """ + get the shifted sparse tensor for shift-window + """ + stride_in = sp.coordinate_map_key.get_tensor_stride() + shift_size = self.shift_size * stride_in[0] + shifted_C = sp.C.clone() + shifted_C[:, 1:] += shift_size + shifted_sp = SparseTensor( + features=sp.F, + coordinates=shifted_C, + device=self.device, + tensor_stride=stride_in, + ) + return shifted_sp + + def get_window_pos(self, sp): + stride_in = sp.coordinate_map_key.get_tensor_stride() + return (sp.C[:, 1:] / stride_in[0]) % self.window_size + + def forward(self, sp, coords_sp): + """ + xyz: position of point inside voxel + colors: other signal for cRSE, include colors and normals + local_xyz: relative position of point indide voxel(using for finer cRSE table) + """ + colors = coords_sp.F[:, 4:] + xyz = coords_sp.F[:, :4] + local_xyz = (xyz - coords_sp.C)[ + :, 1: + ] / coords_sp.coordinate_map_key.get_tensor_stride()[0] + self.device = sp.device + sp_shift = self.get_shifted_sp(sp) + + attn_args = self.get_index01(sp, local_xyz, colors) + attn_args_shift = self.get_index01(sp_shift, local_xyz, colors) + + feats = sp.F + for i, blk in enumerate(self.blocks): + attn_args_blk = attn_args if i % 2 == 0 else attn_args_shift + feats = blk(feats, attn_args_blk) # [N, C] + + sp = assign_feats(sp, feats) + if self.downsample is not None: + sp_down, coords_sp = self.downsample(sp, coords_sp) + return sp, sp_down, coords_sp + else: + return sp, sp, coords_sp + + def extra_repr(self) -> str: + return f"window_size={self.window_size}, depth={self.depth}, channel={self.dim}, num_heads={self.num_heads}, quant_size={self.quant_size}, cRSE={self.cRSE}, fp16_mode={self.fp16_mode}" diff --git a/pointcept/models/swin3d/swin3d_v1m1_base.py b/pointcept/models/swin3d/swin3d_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..7c415b55fc836c91f0943c786e947bff158d123b --- /dev/null +++ b/pointcept/models/swin3d/swin3d_v1m1_base.py @@ -0,0 +1,190 @@ +import torch +import torch.nn as nn +import MinkowskiEngine as ME +from MinkowskiEngine import SparseTensor +from timm.layers import trunc_normal_ + +from .mink_layers import MinkConvBNRelu, MinkResBlock +from .swin3d_layers import GridDownsample, GridKNNDownsample, BasicLayer, Upsample +from pointcept.models.builder import MODELS +from pointcept.models.utils import offset2batch, batch2offset + + +@MODELS.register_module("Swin3D-v1m1") +class Swin3DUNet(nn.Module): + def __init__( + self, + in_channels, + num_classes, + base_grid_size, + depths, + channels, + num_heads, + window_sizes, + quant_size, + drop_path_rate=0.2, + up_k=3, + num_layers=5, + stem_transformer=True, + down_stride=2, + upsample="linear", + knn_down=True, + cRSE="XYZ_RGB", + fp16_mode=0, + ): + super().__init__() + dpr = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(depths)) + ] # stochastic depth decay rule + if knn_down: + downsample = GridKNNDownsample + else: + downsample = GridDownsample + + self.cRSE = cRSE + if stem_transformer: + self.stem_layer = MinkConvBNRelu( + in_channels=in_channels, + out_channels=channels[0], + kernel_size=3, + stride=1, + ) + self.layer_start = 0 + else: + self.stem_layer = nn.Sequential( + MinkConvBNRelu( + in_channels=in_channels, + out_channels=channels[0], + kernel_size=3, + stride=1, + ), + MinkResBlock(in_channels=channels[0], out_channels=channels[0]), + ) + self.downsample = downsample( + channels[0], channels[1], kernel_size=down_stride, stride=down_stride + ) + self.layer_start = 1 + self.layers = nn.ModuleList( + [ + BasicLayer( + dim=channels[i], + depth=depths[i], + num_heads=num_heads[i], + window_size=window_sizes[i], + quant_size=quant_size, + drop_path=dpr[sum(depths[:i]) : sum(depths[: i + 1])], + downsample=downsample if i < num_layers - 1 else None, + down_stride=down_stride if i == 0 else 2, + out_channels=channels[i + 1] if i < num_layers - 1 else None, + cRSE=cRSE, + fp16_mode=fp16_mode, + ) + for i in range(self.layer_start, num_layers) + ] + ) + + if "attn" in upsample: + up_attn = True + else: + up_attn = False + + self.upsamples = nn.ModuleList( + [ + Upsample( + channels[i], + channels[i - 1], + num_heads[i - 1], + window_sizes[i - 1], + quant_size, + attn=up_attn, + up_k=up_k, + cRSE=cRSE, + fp16_mode=fp16_mode, + ) + for i in range(num_layers - 1, 0, -1) + ] + ) + + self.classifier = nn.Sequential( + nn.Linear(channels[0], channels[0]), + nn.BatchNorm1d(channels[0]), + nn.ReLU(inplace=True), + nn.Linear(channels[0], num_classes), + ) + self.num_classes = num_classes + self.base_grid_size = base_grid_size + self.init_weights() + + def forward(self, data_dict): + grid_coord = data_dict["grid_coord"] + feat = data_dict["feat"] + coord_feat = data_dict["coord_feat"] + coord = data_dict["coord"] + offset = data_dict["offset"] + batch = offset2batch(offset) + in_field = ME.TensorField( + features=torch.cat( + [ + batch.unsqueeze(-1), + coord / self.base_grid_size, + coord_feat / 1.001, + feat, + ], + dim=1, + ), + coordinates=torch.cat([batch.unsqueeze(-1).int(), grid_coord.int()], dim=1), + quantization_mode=ME.SparseTensorQuantizationMode.UNWEIGHTED_AVERAGE, + minkowski_algorithm=ME.MinkowskiAlgorithm.SPEED_OPTIMIZED, + device=feat.device, + ) + + sp = in_field.sparse() + coords_sp = SparseTensor( + features=sp.F[:, : coord_feat.shape[-1] + 4], + coordinate_map_key=sp.coordinate_map_key, + coordinate_manager=sp.coordinate_manager, + ) + sp = SparseTensor( + features=sp.F[:, coord_feat.shape[-1] + 4 :], + coordinate_map_key=sp.coordinate_map_key, + coordinate_manager=sp.coordinate_manager, + ) + sp_stack = [] + coords_sp_stack = [] + sp = self.stem_layer(sp) + if self.layer_start > 0: + sp_stack.append(sp) + coords_sp_stack.append(coords_sp) + sp, coords_sp = self.downsample(sp, coords_sp) + + for i, layer in enumerate(self.layers): + coords_sp_stack.append(coords_sp) + sp, sp_down, coords_sp = layer(sp, coords_sp) + sp_stack.append(sp) + assert (coords_sp.C == sp_down.C).all() + sp = sp_down + + sp = sp_stack.pop() + coords_sp = coords_sp_stack.pop() + for i, upsample in enumerate(self.upsamples): + sp_i = sp_stack.pop() + coords_sp_i = coords_sp_stack.pop() + sp = upsample(sp, coords_sp, sp_i, coords_sp_i) + coords_sp = coords_sp_i + + output = self.classifier(sp.slice(in_field).F) + return output + + def init_weights(self): + """Initialize the weights in backbone.""" + + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.LayerNorm) or isinstance(m, nn.BatchNorm1d): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + self.apply(_init_weights) diff --git a/pointcept/models/utils/__init__.py b/pointcept/models/utils/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..da0c47b6a1b482e823bacf176d13ae693c79f436 --- /dev/null +++ b/pointcept/models/utils/__init__.py @@ -0,0 +1,9 @@ +from .misc import ( + offset2batch, + offset2bincount, + bincount2offset, + batch2offset, + off_diagonal, +) +from .checkpoint import checkpoint +from .serialization import encode, decode diff --git a/pointcept/models/utils/checkpoint.py b/pointcept/models/utils/checkpoint.py new file mode 100644 index 0000000000000000000000000000000000000000..58820352bd5d1b37b3905b038816323253ffd3de --- /dev/null +++ b/pointcept/models/utils/checkpoint.py @@ -0,0 +1,57 @@ +""" +Checkpoint Utils for Models + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch + + +class CheckpointFunction(torch.autograd.Function): + @staticmethod + def forward(ctx, run_function, length, *args): + ctx.run_function = run_function + ctx.input_tensors = list(args[:length]) + ctx.input_params = list(args[length:]) + + with torch.no_grad(): + output_tensors = ctx.run_function(*ctx.input_tensors) + return output_tensors + + @staticmethod + def backward(ctx, *output_grads): + ctx.input_tensors = [x.detach().requires_grad_(True) for x in ctx.input_tensors] + with torch.enable_grad(): + # Fixes a bug where the first op in run_function modifies the + # Tensor storage in place, which is not allowed for detach()'d + # Tensors. + shallow_copies = [x.view_as(x) for x in ctx.input_tensors] + output_tensors = ctx.run_function(*shallow_copies) + input_grads = torch.autograd.grad( + output_tensors, + ctx.input_tensors + ctx.input_params, + output_grads, + allow_unused=True, + ) + del ctx.input_tensors + del ctx.input_params + del output_tensors + return (None, None) + input_grads + + +def checkpoint(func, inputs, params, flag): + """ + Evaluate a function without caching intermediate activations, allowing for + reduced memory at the expense of extra compute in the backward pass. + :param func: the function to evaluate. + :param inputs: the argument sequence to pass to `func`. + :param params: a sequence of parameters `func` depends on but does not + explicitly take as arguments. + :param flag: if False, disable gradient checkpointing. + """ + if flag: + args = tuple(inputs) + tuple(params) + return CheckpointFunction.apply(func, len(inputs), *args) + else: + return func(*inputs) diff --git a/pointcept/models/utils/misc.py b/pointcept/models/utils/misc.py new file mode 100644 index 0000000000000000000000000000000000000000..4eef9ebf66c4fb67625c903ccc156b0784aa4e7a --- /dev/null +++ b/pointcept/models/utils/misc.py @@ -0,0 +1,41 @@ +""" +General Utils for Models + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import torch +from itertools import chain + + +@torch.no_grad() +def offset2bincount(offset): + return torch.diff( + offset, prepend=torch.tensor([0], device=offset.device, dtype=torch.long) + ) + + +@torch.no_grad() +def bincount2offset(bincount): + return torch.cumsum(bincount, dim=0) + + +@torch.no_grad() +def offset2batch(offset): + bincount = offset2bincount(offset) + return torch.arange( + len(bincount), device=offset.device, dtype=torch.long + ).repeat_interleave(bincount) + + +@torch.no_grad() +def batch2offset(batch): + return torch.cumsum(batch.bincount(), dim=0).long() + + +def off_diagonal(x): + # return a flattened view of the off-diagonal elements of a square matrix + n, m = x.shape + assert n == m + return x.flatten()[:-1].view(n - 1, n + 1)[:, 1:].flatten() diff --git a/pointcept/models/utils/serialization/__init__.py b/pointcept/models/utils/serialization/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..058c5e1001c76d9c7014bf0bbb824eec4f54f476 --- /dev/null +++ b/pointcept/models/utils/serialization/__init__.py @@ -0,0 +1,8 @@ +from .default import ( + encode, + decode, + z_order_encode, + z_order_decode, + hilbert_encode, + hilbert_decode, +) diff --git a/pointcept/models/utils/serialization/default.py b/pointcept/models/utils/serialization/default.py new file mode 100644 index 0000000000000000000000000000000000000000..275d269aeed20c31e25fe87f8fd2d0cee04a5b59 --- /dev/null +++ b/pointcept/models/utils/serialization/default.py @@ -0,0 +1,59 @@ +import torch +from .z_order import xyz2key as z_order_encode_ +from .z_order import key2xyz as z_order_decode_ +from .hilbert import encode as hilbert_encode_ +from .hilbert import decode as hilbert_decode_ + + +@torch.inference_mode() +def encode(grid_coord, batch=None, depth=16, order="z"): + assert order in {"z", "z-trans", "hilbert", "hilbert-trans"} + if order == "z": + code = z_order_encode(grid_coord, depth=depth) + elif order == "z-trans": + code = z_order_encode(grid_coord[:, [1, 0, 2]], depth=depth) + elif order == "hilbert": + code = hilbert_encode(grid_coord, depth=depth) + elif order == "hilbert-trans": + code = hilbert_encode(grid_coord[:, [1, 0, 2]], depth=depth) + else: + raise NotImplementedError + if batch is not None: + batch = batch.long() + code = batch << depth * 3 | code + return code + + +@torch.inference_mode() +def decode(code, depth=16, order="z"): + assert order in {"z", "hilbert"} + batch = code >> depth * 3 + code = code & ((1 << depth * 3) - 1) + if order == "z": + grid_coord = z_order_decode(code, depth=depth) + elif order == "hilbert": + grid_coord = hilbert_decode(code, depth=depth) + else: + raise NotImplementedError + return grid_coord, batch + + +def z_order_encode(grid_coord: torch.Tensor, depth: int = 16): + x, y, z = grid_coord[:, 0].long(), grid_coord[:, 1].long(), grid_coord[:, 2].long() + # we block the support to batch, maintain batched code in Point class + code = z_order_encode_(x, y, z, b=None, depth=depth) + return code + + +def z_order_decode(code: torch.Tensor, depth): + x, y, z, _ = z_order_decode_(code, depth=depth) + grid_coord = torch.stack([x, y, z], dim=-1) # (N, 3) + return grid_coord + + +def hilbert_encode(grid_coord: torch.Tensor, depth: int = 16): + return hilbert_encode_(grid_coord, num_dims=3, num_bits=depth) + + +def hilbert_decode(code: torch.Tensor, depth: int = 16): + return hilbert_decode_(code, num_dims=3, num_bits=depth) diff --git a/pointcept/models/utils/serialization/hilbert.py b/pointcept/models/utils/serialization/hilbert.py new file mode 100644 index 0000000000000000000000000000000000000000..afca34a4e68bff9e4f1c652e23488fed94f748f5 --- /dev/null +++ b/pointcept/models/utils/serialization/hilbert.py @@ -0,0 +1,294 @@ +""" +Hilbert Order +Modified from https://github.com/PrincetonLIPS/numpy-hilbert-curve + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Kaixin Xu +Please cite our work if the code is helpful to you. +""" + +import torch + + +def right_shift(binary, k=1, axis=-1): + """Right shift an array of binary values. + + Parameters: + ----------- + binary: An ndarray of binary values. + + k: The number of bits to shift. Default 1. + + axis: The axis along which to shift. Default -1. + + Returns: + -------- + Returns an ndarray with zero prepended and the ends truncated, along + whatever axis was specified.""" + + # If we're shifting the whole thing, just return zeros. + if binary.shape[axis] <= k: + return torch.zeros_like(binary) + + # Determine the padding pattern. + # padding = [(0,0)] * len(binary.shape) + # padding[axis] = (k,0) + + # Determine the slicing pattern to eliminate just the last one. + slicing = [slice(None)] * len(binary.shape) + slicing[axis] = slice(None, -k) + shifted = torch.nn.functional.pad( + binary[tuple(slicing)], (k, 0), mode="constant", value=0 + ) + + return shifted + + +def binary2gray(binary, axis=-1): + """Convert an array of binary values into Gray codes. + + This uses the classic X ^ (X >> 1) trick to compute the Gray code. + + Parameters: + ----------- + binary: An ndarray of binary values. + + axis: The axis along which to compute the gray code. Default=-1. + + Returns: + -------- + Returns an ndarray of Gray codes. + """ + shifted = right_shift(binary, axis=axis) + + # Do the X ^ (X >> 1) trick. + gray = torch.logical_xor(binary, shifted) + + return gray + + +def gray2binary(gray, axis=-1): + """Convert an array of Gray codes back into binary values. + + Parameters: + ----------- + gray: An ndarray of gray codes. + + axis: The axis along which to perform Gray decoding. Default=-1. + + Returns: + -------- + Returns an ndarray of binary values. + """ + + # Loop the log2(bits) number of times necessary, with shift and xor. + shift = 2 ** (torch.Tensor([gray.shape[axis]]).log2().ceil().int() - 1) + while shift > 0: + gray = torch.logical_xor(gray, right_shift(gray, shift)) + shift = torch.div(shift, 2, rounding_mode="floor") + return gray + + +def encode(locs, num_dims, num_bits): + """Decode an array of locations in a hypercube into a Hilbert integer. + + This is a vectorized-ish version of the Hilbert curve implementation by John + Skilling as described in: + + Skilling, J. (2004, April). Programming the Hilbert curve. In AIP Conference + Proceedings (Vol. 707, No. 1, pp. 381-387). American Institute of Physics. + + Params: + ------- + locs - An ndarray of locations in a hypercube of num_dims dimensions, in + which each dimension runs from 0 to 2**num_bits-1. The shape can + be arbitrary, as long as the last dimension of the same has size + num_dims. + + num_dims - The dimensionality of the hypercube. Integer. + + num_bits - The number of bits for each dimension. Integer. + + Returns: + -------- + The output is an ndarray of uint64 integers with the same shape as the + input, excluding the last dimension, which needs to be num_dims. + """ + + # Keep around the original shape for later. + orig_shape = locs.shape + bitpack_mask = 1 << torch.arange(0, 8).to(locs.device) + bitpack_mask_rev = bitpack_mask.flip(-1) + + if orig_shape[-1] != num_dims: + raise ValueError(""" + The shape of locs was surprising in that the last dimension was of size + %d, but num_dims=%d. These need to be equal. + """ % (orig_shape[-1], num_dims)) + + if num_dims * num_bits > 63: + raise ValueError(""" + num_dims=%d and num_bits=%d for %d bits total, which can't be encoded + into a int64. Are you sure you need that many points on your Hilbert + curve? + """ % (num_dims, num_bits, num_dims * num_bits)) + + # Treat the location integers as 64-bit unsigned and then split them up into + # a sequence of uint8s. Preserve the association by dimension. + locs_uint8 = locs.long().view(torch.uint8).reshape((-1, num_dims, 8)).flip(-1) + + # Now turn these into bits and truncate to num_bits. + gray = ( + locs_uint8.unsqueeze(-1) + .bitwise_and(bitpack_mask_rev) + .ne(0) + .byte() + .flatten(-2, -1)[..., -num_bits:] + ) + + # Run the decoding process the other way. + # Iterate forwards through the bits. + for bit in range(0, num_bits): + # Iterate forwards through the dimensions. + for dim in range(0, num_dims): + # Identify which ones have this bit active. + mask = gray[:, dim, bit] + + # Where this bit is on, invert the 0 dimension for lower bits. + gray[:, 0, bit + 1 :] = torch.logical_xor( + gray[:, 0, bit + 1 :], mask[:, None] + ) + + # Where the bit is off, exchange the lower bits with the 0 dimension. + to_flip = torch.logical_and( + torch.logical_not(mask[:, None]).repeat(1, gray.shape[2] - bit - 1), + torch.logical_xor(gray[:, 0, bit + 1 :], gray[:, dim, bit + 1 :]), + ) + gray[:, dim, bit + 1 :] = torch.logical_xor( + gray[:, dim, bit + 1 :], to_flip + ) + gray[:, 0, bit + 1 :] = torch.logical_xor(gray[:, 0, bit + 1 :], to_flip) + + # Now flatten out. + gray = gray.swapaxes(1, 2).reshape((-1, num_bits * num_dims)) + + # Convert Gray back to binary. + hh_bin = gray2binary(gray) + + # Pad back out to 64 bits. + extra_dims = 64 - num_bits * num_dims + padded = torch.nn.functional.pad(hh_bin, (extra_dims, 0), "constant", 0) + + # Convert binary values into uint8s. + hh_uint8 = ( + (padded.flip(-1).reshape((-1, 8, 8)) * bitpack_mask) + .sum(2) + .squeeze() + .type(torch.uint8) + ) + + # Convert uint8s into uint64s. + hh_uint64 = hh_uint8.view(torch.int64).squeeze() + + return hh_uint64 + + +def decode(hilberts, num_dims, num_bits): + """Decode an array of Hilbert integers into locations in a hypercube. + + This is a vectorized-ish version of the Hilbert curve implementation by John + Skilling as described in: + + Skilling, J. (2004, April). Programming the Hilbert curve. In AIP Conference + Proceedings (Vol. 707, No. 1, pp. 381-387). American Institute of Physics. + + Params: + ------- + hilberts - An ndarray of Hilbert integers. Must be an integer dtype and + cannot have fewer bits than num_dims * num_bits. + + num_dims - The dimensionality of the hypercube. Integer. + + num_bits - The number of bits for each dimension. Integer. + + Returns: + -------- + The output is an ndarray of unsigned integers with the same shape as hilberts + but with an additional dimension of size num_dims. + """ + + if num_dims * num_bits > 64: + raise ValueError(""" + num_dims=%d and num_bits=%d for %d bits total, which can't be encoded + into a uint64. Are you sure you need that many points on your Hilbert + curve? + """ % (num_dims, num_bits)) + + # Handle the case where we got handed a naked integer. + hilberts = torch.atleast_1d(hilberts) + + # Keep around the shape for later. + orig_shape = hilberts.shape + bitpack_mask = 2 ** torch.arange(0, 8).to(hilberts.device) + bitpack_mask_rev = bitpack_mask.flip(-1) + + # Treat each of the hilberts as a s equence of eight uint8. + # This treats all of the inputs as uint64 and makes things uniform. + hh_uint8 = ( + hilberts.ravel().type(torch.int64).view(torch.uint8).reshape((-1, 8)).flip(-1) + ) + + # Turn these lists of uints into lists of bits and then truncate to the size + # we actually need for using Skilling's procedure. + hh_bits = ( + hh_uint8.unsqueeze(-1) + .bitwise_and(bitpack_mask_rev) + .ne(0) + .byte() + .flatten(-2, -1)[:, -num_dims * num_bits :] + ) + + # Take the sequence of bits and Gray-code it. + gray = binary2gray(hh_bits) + + # There has got to be a better way to do this. + # I could index them differently, but the eventual packbits likes it this way. + gray = gray.reshape((-1, num_bits, num_dims)).swapaxes(1, 2) + + # Iterate backwards through the bits. + for bit in range(num_bits - 1, -1, -1): + # Iterate backwards through the dimensions. + for dim in range(num_dims - 1, -1, -1): + # Identify which ones have this bit active. + mask = gray[:, dim, bit] + + # Where this bit is on, invert the 0 dimension for lower bits. + gray[:, 0, bit + 1 :] = torch.logical_xor( + gray[:, 0, bit + 1 :], mask[:, None] + ) + + # Where the bit is off, exchange the lower bits with the 0 dimension. + to_flip = torch.logical_and( + torch.logical_not(mask[:, None]), + torch.logical_xor(gray[:, 0, bit + 1 :], gray[:, dim, bit + 1 :]), + ) + gray[:, dim, bit + 1 :] = torch.logical_xor( + gray[:, dim, bit + 1 :], to_flip + ) + gray[:, 0, bit + 1 :] = torch.logical_xor(gray[:, 0, bit + 1 :], to_flip) + + # Pad back out to 64 bits. + extra_dims = 64 - num_bits + padded = torch.nn.functional.pad(gray, (extra_dims, 0), "constant", 0) + + # Now chop these up into blocks of 8. + locs_chopped = padded.flip(-1).reshape((-1, num_dims, 8, 8)) + + # Take those blocks and turn them unto uint8s. + # from IPython import embed; embed() + locs_uint8 = (locs_chopped * bitpack_mask).sum(3).squeeze().type(torch.uint8) + + # Finally, treat these as uint64s. + flat_locs = locs_uint8.view(torch.int64) + + # Return them in the expected shape. + return flat_locs.reshape((*orig_shape, num_dims)) diff --git a/pointcept/models/utils/serialization/z_order.py b/pointcept/models/utils/serialization/z_order.py new file mode 100644 index 0000000000000000000000000000000000000000..6fd01a5bcf4b6c76c5d75db4999326e174409ee3 --- /dev/null +++ b/pointcept/models/utils/serialization/z_order.py @@ -0,0 +1,126 @@ +# -------------------------------------------------------- +# Octree-based Sparse Convolutional Neural Networks +# Copyright (c) 2022 Peng-Shuai Wang +# Licensed under The MIT License [see LICENSE for details] +# Written by Peng-Shuai Wang +# -------------------------------------------------------- + +import torch +from typing import Optional, Union + + +class KeyLUT: + def __init__(self): + r256 = torch.arange(256, dtype=torch.int64) + r512 = torch.arange(512, dtype=torch.int64) + zero = torch.zeros(256, dtype=torch.int64) + device = torch.device("cpu") + + self._encode = { + device: ( + self.xyz2key(r256, zero, zero, 8), + self.xyz2key(zero, r256, zero, 8), + self.xyz2key(zero, zero, r256, 8), + ) + } + self._decode = {device: self.key2xyz(r512, 9)} + + def encode_lut(self, device=torch.device("cpu")): + if device not in self._encode: + cpu = torch.device("cpu") + self._encode[device] = tuple(e.to(device) for e in self._encode[cpu]) + return self._encode[device] + + def decode_lut(self, device=torch.device("cpu")): + if device not in self._decode: + cpu = torch.device("cpu") + self._decode[device] = tuple(e.to(device) for e in self._decode[cpu]) + return self._decode[device] + + def xyz2key(self, x, y, z, depth): + key = torch.zeros_like(x) + for i in range(depth): + mask = 1 << i + key = ( + key + | ((x & mask) << (2 * i + 2)) + | ((y & mask) << (2 * i + 1)) + | ((z & mask) << (2 * i + 0)) + ) + return key + + def key2xyz(self, key, depth): + x = torch.zeros_like(key) + y = torch.zeros_like(key) + z = torch.zeros_like(key) + for i in range(depth): + x = x | ((key & (1 << (3 * i + 2))) >> (2 * i + 2)) + y = y | ((key & (1 << (3 * i + 1))) >> (2 * i + 1)) + z = z | ((key & (1 << (3 * i + 0))) >> (2 * i + 0)) + return x, y, z + + +_key_lut = KeyLUT() + + +def xyz2key( + x: torch.Tensor, + y: torch.Tensor, + z: torch.Tensor, + b: Optional[Union[torch.Tensor, int]] = None, + depth: int = 16, +): + r"""Encodes :attr:`x`, :attr:`y`, :attr:`z` coordinates to the shuffled keys + based on pre-computed look up tables. The speed of this function is much + faster than the method based on for-loop. + + Args: + x (torch.Tensor): The x coordinate. + y (torch.Tensor): The y coordinate. + z (torch.Tensor): The z coordinate. + b (torch.Tensor or int): The batch index of the coordinates, and should be + smaller than 32768. If :attr:`b` is :obj:`torch.Tensor`, the size of + :attr:`b` must be the same as :attr:`x`, :attr:`y`, and :attr:`z`. + depth (int): The depth of the shuffled key, and must be smaller than 17 (< 17). + """ + + EX, EY, EZ = _key_lut.encode_lut(x.device) + x, y, z = x.long(), y.long(), z.long() + + mask = 255 if depth > 8 else (1 << depth) - 1 + key = EX[x & mask] | EY[y & mask] | EZ[z & mask] + if depth > 8: + mask = (1 << (depth - 8)) - 1 + key16 = EX[(x >> 8) & mask] | EY[(y >> 8) & mask] | EZ[(z >> 8) & mask] + key = key16 << 24 | key + + if b is not None: + b = b.long() + key = b << 48 | key + + return key + + +def key2xyz(key: torch.Tensor, depth: int = 16): + r"""Decodes the shuffled key to :attr:`x`, :attr:`y`, :attr:`z` coordinates + and the batch index based on pre-computed look up tables. + + Args: + key (torch.Tensor): The shuffled key. + depth (int): The depth of the shuffled key, and must be smaller than 17 (< 17). + """ + + DX, DY, DZ = _key_lut.decode_lut(key.device) + x, y, z = torch.zeros_like(key), torch.zeros_like(key), torch.zeros_like(key) + + b = key >> 48 + key = key & ((1 << 48) - 1) + + n = (depth + 2) // 3 + for i in range(n): + k = key >> (i * 9) & 511 + x = x | (DX[k] << (i * 3)) + y = y | (DY[k] << (i * 3)) + z = z | (DZ[k] << (i * 3)) + + return x, y, z, b diff --git a/pointcept/models/utils/structure.py b/pointcept/models/utils/structure.py new file mode 100644 index 0000000000000000000000000000000000000000..1e8e80a9b95446005b771cd18aecd490d7ffb723 --- /dev/null +++ b/pointcept/models/utils/structure.py @@ -0,0 +1,209 @@ +import torch +import spconv.pytorch as spconv + +try: + import ocnn +except ImportError: + ocnn = None +from addict import Dict +from typing import List + +from pointcept.models.utils.serialization import encode +from pointcept.models.utils import ( + offset2batch, + batch2offset, + offset2bincount, + bincount2offset, +) + + +class Point(Dict): + """ + Point Structure of Pointcept + + A Point (point cloud) in Pointcept is a dictionary that contains various properties of + a batched point cloud. The property with the following names have a specific definition + as follows: + + - "coord": original coordinate of point cloud; + - "grid_coord": grid coordinate for specific grid size (related to GridSampling); + Point also support the following optional attributes: + - "offset": if not exist, initialized as batch size is 1; + - "batch": if not exist, initialized as batch size is 1; + - "feat": feature of point cloud, default input of model; + - "grid_size": Grid size of point cloud (related to GridSampling); + (related to Serialization) + - "serialized_depth": depth of serialization, 2 ** depth * grid_size describe the maximum of point cloud range; + - "serialized_code": a list of serialization codes; + - "serialized_order": a list of serialization order determined by code; + - "serialized_inverse": a list of inverse mapping determined by code; + (related to Sparsify: SpConv) + - "sparse_shape": Sparse shape for Sparse Conv Tensor; + - "sparse_conv_feat": SparseConvTensor init with information provide by Point; + """ + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + # If one of "offset" or "batch" do not exist, generate by the existing one + if "batch" not in self.keys() and "offset" in self.keys(): + self["batch"] = offset2batch(self.offset) + elif "offset" not in self.keys() and "batch" in self.keys(): + self["offset"] = batch2offset(self.batch) + + def serialization(self, order="z", depth=None, shuffle_orders=False): + """ + Point Cloud Serialization + + relay on ["grid_coord" or "coord" + "grid_size", "batch", "feat"] + """ + self["order"] = order + assert "batch" in self.keys() + if "grid_coord" not in self.keys(): + # if you don't want to operate GridSampling in data augmentation, + # please add the following augmentation into your pipline: + # dict(type="Copy", keys_dict={"grid_size": 0.01}), + # (adjust `grid_size` to what your want) + assert {"grid_size", "coord"}.issubset(self.keys()) + + self["grid_coord"] = torch.div( + self.coord - self.coord.min(0)[0], self.grid_size, rounding_mode="trunc" + ).int() + + if depth is None: + # Adaptive measure the depth of serialization cube (length = 2 ^ depth) + depth = int(self.grid_coord.max() + 1).bit_length() + self["serialized_depth"] = depth + # Maximum bit length for serialization code is 63 (int64) + assert depth * 3 + len(self.offset).bit_length() <= 63 + # Here we follow OCNN and set the depth limitation to 16 (48bit) for the point position. + # Although depth is limited to less than 16, we can encode a 655.36^3 (2^16 * 0.01) meter^3 + # cube with a grid size of 0.01 meter. We consider it is enough for the current stage. + # We can unlock the limitation by optimizing the z-order encoding function if necessary. + assert depth <= 16 + + # The serialization codes are arranged as following structures: + # [Order1 ([n]), + # Order2 ([n]), + # ... + # OrderN ([n])] (k, n) + code = [ + encode(self.grid_coord, self.batch, depth, order=order_) for order_ in order + ] + code = torch.stack(code) + order = torch.argsort(code) + inverse = torch.zeros_like(order).scatter_( + dim=1, + index=order, + src=torch.arange(0, code.shape[1], device=order.device).repeat( + code.shape[0], 1 + ), + ) + + if shuffle_orders: + perm = torch.randperm(code.shape[0]) + code = code[perm] + order = order[perm] + inverse = inverse[perm] + + self["serialized_code"] = code + self["serialized_order"] = order + self["serialized_inverse"] = inverse + + def sparsify(self, pad=96): + """ + Point Cloud Serialization + + Point cloud is sparse, here we use "sparsify" to specifically refer to + preparing "spconv.SparseConvTensor" for SpConv. + + relay on ["grid_coord" or "coord" + "grid_size", "batch", "feat"] + + pad: padding sparse for sparse shape. + """ + assert {"feat", "batch"}.issubset(self.keys()) + if "grid_coord" not in self.keys(): + # if you don't want to operate GridSampling in data augmentation, + # please add the following augmentation into your pipline: + # dict(type="Copy", keys_dict={"grid_size": 0.01}), + # (adjust `grid_size` to what your want) + assert {"grid_size", "coord"}.issubset(self.keys()) + self["grid_coord"] = torch.div( + self.coord - self.coord.min(0)[0], self.grid_size, rounding_mode="trunc" + ).int() + if "sparse_shape" in self.keys(): + sparse_shape = self.sparse_shape + else: + sparse_shape = torch.add( + torch.max(self.grid_coord, dim=0).values, pad + ).tolist() + sparse_conv_feat = spconv.SparseConvTensor( + features=self.feat, + indices=torch.cat( + [self.batch.unsqueeze(-1).int(), self.grid_coord.int()], dim=1 + ).contiguous(), + spatial_shape=sparse_shape, + batch_size=self.batch[-1].tolist() + 1, + ) + self["sparse_shape"] = sparse_shape + self["sparse_conv_feat"] = sparse_conv_feat + + def octreelization(self, depth=None, full_depth=None): + """ + Point Cloud Octreelization + + Generate octree with OCNN + relay on ["grid_coord", "batch", "feat"] + """ + assert ( + ocnn is not None + ), "Please follow https://github.com/octree-nn/ocnn-pytorch install ocnn." + assert {"feat", "batch"}.issubset(self.keys()) + # add 1 to make grid space support shift order + if "grid_coord" not in self.keys(): + # if you don't want to operate GridSampling in data augmentation, + # please add the following augmentation into your pipline: + # dict(type="Copy", keys_dict={"grid_size": 0.01}), + # (adjust `grid_size` to what your want) + assert {"grid_size", "coord"}.issubset(self.keys()) + self["grid_coord"] = torch.div( + self.coord - self.coord.min(0)[0], self.grid_size, rounding_mode="trunc" + ).int() + if depth is None: + if "depth" in self.keys(): + depth = self.depth + else: + depth = int(self.grid_coord.max() + 1).bit_length() + if full_depth is None: + full_depth = 1 + self["depth"] = depth + assert depth <= 16 # maximum in ocnn + + # [0, 2**depth] -> [0, 2] -> [-1, 1] + coord = self.grid_coord / 2 ** (self.depth - 1) - 1.0 + point = ocnn.octree.Points( + points=coord, + features=self.feat, + batch_id=self.batch.unsqueeze(-1), + batch_size=self.batch[-1] + 1, + ) + octree = ocnn.octree.Octree( + depth=depth, + full_depth=full_depth, + batch_size=self.batch[-1] + 1, + device=coord.device, + ) + octree.build_octree(point) + octree.construct_all_neigh() + + query_pts = torch.cat([self.grid_coord, point.batch_id], dim=1).contiguous() + inverse = octree.search_xyzb(query_pts, depth, True) + assert torch.sum(inverse < 0) == 0 # all mapping should be valid + inverse_ = torch.unique(inverse) + order = torch.zeros_like(inverse_).scatter_( + dim=0, + index=inverse, + src=torch.arange(0, inverse.shape[0], device=inverse.device), + ) + self["octree"] = octree + self["octree_order"] = order + self["octree_inverse"] = inverse diff --git a/pointcept/models/utonia/README.md b/pointcept/models/utonia/README.md new file mode 100644 index 0000000000000000000000000000000000000000..4f271a98efe46cd8c8ff1d3db89b1e470bfe6cbe --- /dev/null +++ b/pointcept/models/utonia/README.md @@ -0,0 +1,534 @@ +# Utonia +This repo is the official training codebase of the paper **_Utonia: Toward One Encoder for All Point Clouds_**. Utonia is a step toward one-from-all and one-for-all point cloud encoder, developed from [Concerto](https://github.com/Pointcept/Concerto) and [Sonata](https://github.com/facebookresearch/sonata). It pretrains a single encoder on diverse point cloud data and reuses it as a reliable backbone for downstream tasks. + +*We recommend beginning with our inference [demo](https://github.com/Pointcept/Utonia), and the data transform process is different from Concerto and Sonata. This section is designed for users interested in reproducing our pre-training.* + +
+teaser +
+ +## Quick start + +#### 1. Installation +Follow the instructions [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#installation). + +#### 2. Dataset +Besides the same indoor and outdoor point cloud datasets as Concerto, Utonia includes object datasets into pretraining. Each dataset has image assets and their correspondence to point clouds if available (HKRemote, ScanObjectNN, PartNet lack image information). + +The standard data is organized as follows: +``` +└── Default Dataset + ├── images + │ ├── train + │ │ ├── color + │ │ ├── correspondence + │ │ ├── depth (optional) + │ │ ├── intrinsic + │ │ └── pose + │ ├── val + │ └── test + ├── splits + │ ├── train.json + │ ├── val.json + │ └── test.json + ├── train + ├── val + └── test +``` +The `color`, `correspondence`, `depth`, `intrinsic`, and `pose` folders contain RGB images, point cloud correspondences, depth maps, camera intrinsics, and camera poses, respectively. The `train.json`, `val.json`, and `test.json` files in the _splits_ folder provide the indices for the training, validation, and test splits. + +For the _intrinsic_ folder, if only a single file is presented, it indicates that all images share the same intrinsic parameters. If multiple files exist, each corresponds to a different image. In the case of the HM3D dataset, where camera parameters are simulated, all images share identical intrinsics across the entire dataset. + +##### 2.1 [Option A] Preprocess Dataset from Scratch +For those who wish to do data preprocessing locally, please follow the instructions below: +###### 2.1.1 Indoor Datasets +- If you do not have the point cloud datasets for Sonata, you can process the next step without any modification. If you have already downloaded the processed Sonata datasets below, you can remove "-c" in the next step to only output image assets. + - ScanNet v2 - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#scannet-v2) ] + - ScanNet++ v2 - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#scannet) ] + - S3DIS - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#s3dis) ] + - ArkitScenes - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#arkitscenes) ] + - HM3D - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#habitat---matterport-3d-hm3d) ] + - Structured3D - [ [here](https://github.com/Pointcept/Pointcept?tab=readme-ov-file#structured3d) ] + + +- Now you can follow the instructions below to processe the image assets. + - ScanNet v2 + - Download the [ScanNet](http://www.scan-net.org/) v2 dataset. + - Run preprocessing code for raw ScanNet as follows: + ```bash + # RAW_SCANNET_DIR: the directory of downloaded ScanNet v2 raw dataset. PROCESSED_SCANNET_DIR: the directory of the processed ScanNet dataset (output dir). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/scannet/preprocess_scannet.sh -d ${RAW_SCANNET_DIR} -o ${PROCESSED_SCANNET_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/scannet/splits.py --dataset_root ${PROCESSED_SCANNET_DIR} + ``` + - ScanNet++ + - Download the [ScanNet++](https://kaldir.vc.in.tum.de/scannetpp/) dataset. + - Run preprocessing code for raw ScanNet++ as follows: + ```bash + # For Concerto, we use the unsplitted point cloud data. There is no need to split the large scene because we crop the scenes in code according to the selected camera views. + # RAW_SCANNETPP_DIR: the directory of downloaded ScanNet++ raw dataset. PROCESSED_SCANNETPP_DIR: the directory of the processed ScanNet++ dataset (output dir). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/scannetpp/preprocess_scannetpp.sh -d ${RAW_SCANNETPP_DIR} -o ${PROCESSED_SCANNETPP_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/scannetpp/splits.py --dataset_root ${PROCESSED_SCANNETPP_DIR} + ``` + - S3DIS + - Download S3DIS data by filling this [Google form](https://docs.google.com/forms/d/e/1FAIpQLScDimvNMCGhy_rmBA2gHfDu3naktRm6A8BPwAWWDv-Uhm6Shw/viewform?c=0&w=1). Download the `Stanford3dDataset_v1.2.zip` file and unzip it. + - Fix error in `Area_5/office_19/Annotations/ceiling` Line 323474 (103.0�0000 => 103.000000). + - (Optional) Download Full 2D-3D S3DIS dataset (no XYZ) from [here](https://github.com/alexsax/2D-3D-Semantics) for parsing normal. + - Run preprocessing code for S3DIS as follows: + ```bash + # Default not use --align_angle + # S3DIS_DIR: the directory of downloaded Stanford3dDataset_v1.2 dataset. RAW_S3DIS_DIR: the directory of the Stanford2d3dDataset_noXYZ dataset. PROCESSED_S3DIS_DIR: the directory of the processed S3DIS dataset (output dir). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/s3dis/preprocess_s3dis.sh -d ${S3DIS_DIR} -r ${RAW_S3DIS_DIR} -o ${PROCESSED_S3DIS_DIR} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/s3dis/splits.py --dataset_root ${PROCESSED_S3DIS_DIR} + ``` + - ARKitScenes + - Download ArkitScenes 3DOD split with the following commands: + ```bash + # RAW_AS_DIR: the directory of downloaded Raw ArkitScenes dataset. + git clone https://github.com/apple/ARKitScenes.git + cd ARKitScenes + python download_data.py 3dod --download_dir $RAW_AS_DIR --video_id_csv threedod/3dod_train_val_splits.csv + ``` + - Run preprocessing code for ArkitScenes as follows: + ```bash + # RAW_AS_DIR: the directory of the downloaded ARKitScenes dataset. PROCESSED_AS_DIR: the directory of processed ArkitScenes dataset (output dir). + # NUM_WORKERS: Number of workers for preprocessing, default same as CPU count (might OOM). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/arkitscenes/preprocess_arkitscenes.sh -d ${RAW_AS_DIR} -o ${PROCESSED_AS_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/arkitscenes/splits.py --dataset_root ${PROCESSED_ARKITSCENES_DIR} + ``` + - Habitat - Matterport 3D (HM3D) + - Download HM3D `hm3d-train-glb-v0.2.tar` and `hm3d-val-glb-v0.2.tar` with instuction [here](https://github.com/facebookresearch/habitat-sim/blob/main/DATASETS.md#habitat-matterport-3d-research-dataset-hm3d) and unzip them. + - Run preprocessing code for HM3D as follows: + ```bash + # We leverage the Habitat-Sim to simulate the camera views. The detailed installation for Habitat-Sim can be found at Habitat-Sim(https://github.com/facebookresearch/habitat-sim) + # RAW_HM_DIR: the directory of downloaded HM3D dataset. PROCESSED_HM_DIR: the directory of processed HM3D dataset (output dir). + # NUM_WORKERS: Number of workers for preprocessing, default same as CPU count (might OOM). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/hm3d/preprocess_hm3d.sh -d ${RAW_HM3D_DIR} -o ${PROCESSED_HM3D_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/hm3d/splits.py --dataset_root ${PROCESSED_HM3D_DIR} + ``` + - Structured3D + - Download Structured3D panorama related and perspective (full) related zip files by filling this [Google form](https://docs.google.com/forms/d/e/1FAIpQLSc0qtvh4vHSoZaW6UvlXYy79MbcGdZfICjh4_t4bYofQIVIdw/viewform?pli=1) (no need to unzip them). + - Organize all downloaded zip file in one folder (`${STRUCT3D_DIR}`). + - Run preprocessing code for Structured3D as follows: + ```bash + # RAW_STRUCT3D_DIR: the directory of downloaded Structured3D dataset. PROCESSED_STRUCT3D_DIR: the directory of processed Structured3D dataset (output dir). + # NUM_WORKERS: Number of workers for preprocessing, default same as CPU count (might OOM). + # Use -p to parse depths of the selected images. + bash pointcept/datasets/preprocessing/concerto/structured3d/preprocess_structured3d.sh -d ${RAW_STRUCT3D_DIR} -o ${PROCESSED_STRUCT3D_DIR} -n ${NUM_WORKERS} -c + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/structured3d/splits.py --dataset_root ${PROCESSED_STRUCT3D_DIR} + ``` + - RealEstate10K (RE10K) + Additionally, we leverage video lifted RE10K by [VGGT](https://github.com/facebookresearch/vggt). Currently, the python file below only supports single data inference through VGGT. However, it is OK for you to start several python processes by setting num_workers and thread_id. + - Install VGGT as a package + ```bash + git clone https://github.com/facebookresearch/vggt.git + pip install -e . + ``` + - Download RealEstate10K inPixelSplat format[ [Here](http://schadenfreude.csail.mit.edu:8000/)] + - Preprocess the RE10K dataset + ```bash + # RAW_RE10K_DIR: the directory of downloaded RE10K dataset. + # PROCESSED_RE10K_DIR: the directory of processed RE10K dataset (output dir). + # using --parse_depths in the shell script to parse depths of the selected images. + python pointcept/datasets/preprocessing/concerto/re10k/preprocess_re10k.py --dataset_root ${RAW_RE10K_DIR} --output_root ${PROCESSED_RE10K_DIR} --num_workers ${NUM_WORKERS} --thread_id ${THREAD_ID} + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/re10k/splits.py --dataset_root ${PROCESSED_RE10K_DIR} + ``` + - After all the preprocessing, link processed dataset above to the codebase. + ```bash + # PROCESSED_DIR: the directory of the processed dataset (output dir). + # DATASET_NAME: the dataset name, which should be consistent with 'data_root' in the corresponding config, such as scannet, scannetpp, s3dis, arkitscenes, hm3d, structured3d, re10k, semantic_kitti, nuscenes, waymo + mkdir data + ln -s ${PROCESSED_DIR} ${CODEBASE_DIR}/data/${DATASET_NAME} + ``` + +###### 2.1.2 Outdoor Datasets +Follow the instructions below to processe the outdoor datasets. +- SemanticKITTI + - Download [SemanticKITTI](http://www.semantic-kitti.org/dataset.html#download) dataset. + - SemanticKitti does not need any preprocessing. + ```bash + # SEMANTIC_KITTI_DIR: the directory of SemanticKITTI dataset. + # |- SEMANTIC_KITTI_DIR + # |- dataset + # |- sequences + # |- 00 + # |- 01 + # |- ... + ``` + +- nuScenes + - Download the official [NuScene](https://www.nuscenes.org/nuscenes#download) dataset (with Lidar Segmentation) and organize the downloaded files as follows: + ```bash + NUSCENES_DIR + │── samples + │── sweeps + │── lidarseg + ... + │── v1.0-trainval + │── v1.0-test + ``` + - Run information preprocessing code (modified from OpenPCDet) for nuScenes as follows: + ```bash + # NUSCENES_DIR: the directory of downloaded nuScenes dataset. + # PROCESSED_NUSCENES_DIR: the directory of processed nuScenes dataset (output dir). + # MAX_SWEEPS: Max number of sweeps. Default: 10. + pip install nuscenes-devkit pyquaternion + python pointcept/datasets/preprocessing/concerto/nuscenes/preprocess_nuscenes_info.py --dataset_root ${NUSCENES_DIR} --output_root ${PROCESSED_NUSCENES_DIR} --max_sweeps ${MAX_SWEEPS} --with_camera + ``` + + - Link raw dataset to processed NuScene dataset folder: + ```bash + # NUSCENES_DIR: the directory of downloaded nuScenes dataset. + # PROCESSED_NUSCENES_DIR: the directory of processed nuScenes dataset (output dir). + ln -s ${NUSCENES_DIR} {PROCESSED_NUSCENES_DIR}/raw + ``` + then the processed nuscenes folder is organized as follows: + ```bash + nuscene + |── raw + │── samples + │── sweeps + │── lidarseg + ... + │── v1.0-trainval + │── v1.0-test + |── info + ``` +- Waymo + - Download the official [Waymo](https://waymo.com/open/download/) dataset (v1.4.3) and organize the downloaded files as follows: + ```bash + WAYMO_RAW_DIR + │── training + │── validation + │── testing + ``` + - Install the following dependence: + ```bash + # If shows "No matching distribution found", download whl directly from Pypi and install the package. + conda create -n waymo python=3.10 -y + conda activate waymo + pip install waymo-open-dataset-tf-2-12-0 + ``` + - Run the preprocessing code as follows: + ```bash + # WAYMO_DIR: the directory of the downloaded Waymo dataset. + # PROCESSED_WAYMO_DIR: the directory of the processed Waymo dataset (output dir). + # NUM_WORKERS: num workers for preprocessing + bash pointcept/datasets/preprocessing/concerto/waymo/preprocess_waymo.sh -d ${WAYMO_DIR} -o ${PROCESSED_WAYMO_DIR} -n ${NUM_WORKERS} -s "training validation" + # To generate the JSON file for the Concerto dataloader + python pointcept/datasets/preprocessing/concerto/waymo/splits.py --dataset_root ${PROCESSED_WAYMO_DIR} + ``` +- HK Remote + - Download the official data using script at `pointcept/datasets/preprocessing/concerto/hk/hkremote_download.sh` and unzip all the .zip files to ${HK_3D_MAPS}. + - Run the preprocessing code as follows: + ``` + # HK_3D_MAPS_DIR: the directory of the downloaded HK Remote dataset. + # PROCESSED_HK_DIR: the directory of the processed Waymo dataset (output dir). + # NUM_WORKERS: num workers for preprocessing + bash pointcept/datasets/preprocessing/concerto/hk/preprocess_hk.sh -d ${HK_3D_MAPS_DIR} -o ${PROCESSED_HK_DIR} -n ${NUM_WORKERS} + python pointcept/datasets/preprocessing/concerto/hk/splits.py --dataset_root ${PROCESSED_HK_DIR} + ``` + +- After all the preprocessing, link processed dataset above to the codebase. +```bash +# PROCESSED_DIR: the directory of the processed dataset (output dir). +# DATASET_NAME: the dataset name, which should be consistent with 'data_root' in the corresponding config, such as scannet, scannetpp, s3dis, arkitscenes, hm3d, structured3d, re10k, semantic_kitti, nuscenes, waymo +mkdir data +ln -s ${PROCESSED_DIR} ${CODEBASE_DIR}/data/${DATASET_NAME} +``` + +###### 2.1.3 Object Datasets +- Cap3D + - Download the [Cap3D](https://huggingface.co/datasets/tiange/Cap3D) dataset to ${RAW_CAP3D_DIR}. + - Run preprocessing code for Cap3D as follows to produce the correspondence between images and point clouds: + ```bash + # RAW_CAP3D_DIR: the directory of downloaded Cap3D raw dataset. + # CORRESPONDENCE_CAP3D_DIR: the directory of the correspondence between images and point clouds. + # NUM_WORKERS: num workers for preprocessing + bash pointcept/datasets/preprocessing/concerto/cap3d/preprocess_cap3d.sh -c ${RAW_CAP3D_DIR}/misc/RenderedImage_CamMatrix_zips -p ${RAW_CAP3D_DIR}/misc/PointCloud_pt_zips -o ${CORRESPONDENCE_CAP3D_DIR} -n ${NUM_WORKERS} + # To generate the JSON file for the Concerto dataloader + ln -s ${CORRESPONDENCE_CAP3D_DIR} data/cap3d/correspondences + ln -s ${RAW_CAP3D_DIR}/misc/PointCloud_pt_zips/Cap3D_pcs_pt data/cap3d/train + ln -s ${RAW_CAP3D_DIR}/RenderedImage_perobj_zips/Cap3D_Objaverse_renderimgs data/cap3d/images/train + python pointcept/datasets/preprocessing/concerto/cap3d/splits.py --dataset_root data/cap3d + ``` + The data directory will be like: + ``` + └── Cap3D Dataset + ├── images + │ └── train + ├── correspondences + ├── splits + │ └── train.json + └── train + ``` +- GraspNet + - Download the [GraspNet](https://graspnet.net/datasets.html) dataset, including both train and test. and unzip them to ${GRASPNET_DATASET_DIR} + - Run the preprocessing code as follows: + ``` + python pointcept/datasets/preprocessing/concerto/graspnet/preprocess_graspnet_poses.py --dataset_root ${GRASPNET_DATASET_DIR} + python pointcept/datasets/preprocessing/concerto/graspnet/splits.py --dataset_root ${GRASPNET_DATASET_DIR} + ln -s ${GRASPNET_DATASET_DIR} data/graspnet + ``` + The data directory will be like: + ``` + └── GraspNet Dataset + ├── splits + │ ├── train.json + │ └── val.json + └── scenes + ``` +- ParNet + - Download the [PartNet](https://www.shapenet.org/download/parts) dataset, including `data_v0.zip` and unzip it to ${PARTNET_DATA_0_DIR}. + ``` + mkdir data/partnet_data_v0 + ln -s ${PARTNET_DATA_0_DIR} data/partnet_data_v0/train + ``` +- ScanObjectNN + - Download the [ScanObjectNN](https://forms.gle/ZZRnnmaUdwfRucoy7) dataset, including `h5_files.zip` and `raw/object_dataset.zip`. Unzip them to \${BENCHMARK_SCANOBJECTNN_DIR} and ${RAW_SCANOBJECTNN_DIR} + ``` + ln -s ${BENCHMARK_SCANOBJECTNN_DIR} data/scanobject_eval + mkdir data/scanobject_raw + ln -s ${RAW_SCANOBJECTNN_DIR} data/scanobject_raw/train + ``` + + +##### 2.2 [Option B] Download Preprocessed Dataset +You can download the preprocessed indoor datasets from Huggingface. Other datasets are currently not ready. The downloaded datasets should be put in the folders named after the ${DATASET_NAME}, which is the string in front of .tar.gz. + +- ScanNet v2 - [ [here](https://huggingface.co/datasets/Pointcept/concerto_scannet_compressed) ] +- ScanNet++ v2 - [ [here](https://huggingface.co/datasets/Pointcept/concerto_scannetpp_compressed) ] +- S3DIS - [ [here](https://huggingface.co/datasets/Pointcept/concerto_s3dis_compressed) ] +- ArkitScenes - [ [here](https://huggingface.co/datasets/Pointcept/concerto_arkitscenes_compressed) ] +- HM3D - [ [here](https://huggingface.co/datasets/Pointcept/concerto_hm3d_compressed) ] +- Structured3D - [ [here](https://huggingface.co/datasets/Pointcept/concerto_structured3d_compressed) ] +- RE10K - [ [here](https://huggingface.co/datasets/Pointcept/concerto_re10k_compressed) ] + +Then you need to decompress the tar.gz using: +```bash +cat ${DATASET_NAME}/${DATASET_NAME}.tar.gz.* | tar -xzvf - +``` +After decompressing, link processed dataset above to the codebase. +```bash +# PROCESSED_DIR: the directory of the processed dataset (output dir). +# DATASET_NAME: the dataset name, which should be consistent with 'data_root' in the corresponding config, such as scannet, scannetpp, s3dis, arkitscenes, hm3d, structured3d, re10k, semantic_kitti, nuscenes, waymo +mkdir data +ln -s ${PROCESSED_DIR} ${CODEBASE_DIR}/data/${DATASET_NAME} +``` + +#### 3. Pre-training +Enable pre-training with 32 GPUs (can be adjusted according to your hardware resources) by running the following script: +```bash +# default config stage 1 +sh scripts/train.sh -m 8 -g 8 -d utonia -c pretrain-utonia-v1m1-0-base_stagev1 -n pretrain-utonia-v1m1-0-base_stagev1 + +# default config stage 2 +sh scripts/train.sh -m 8 -g 8 -d utonia -c pretrain-utonia-v1m1-0-base_stagev2 -n pretrain-utonia-v1m1-0-base_stagev2 -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev1/model/model_last.pth + +# (or) if OOM +# half the batch size and half the base learning rate + +# (or) if Nan appears +# try making grad clip lower as 2.0 or 1.0 +# try making enable_amp=False +``` + +#### 4. Probing and Tuning +Our pre-trained model weight can be downloaded [here](https://huggingface.co/Pointcept/Utonia/blob/main/utonia.pth). The original probing and fine-tuning scripts are designed for locally pretrained weight. For the `utonia.pth` on HuggingFace, replace the lines below in the config to correctly load the model weight: +```bash +dict( + type="CheckpointLoader", + keywords="module.student.backbone", + replacement="module.backbone", + ), +``` +to +```bash +dict( + type="CheckpointLoader", + keywords="module", + replacement="module.backbone", + ), +``` +Or directly, use `pretrain-utonia-v1m1-0-base.pth` without any modification. + +Here are the example commands for probing and tuning on large model: +```bash +# Assume the pre-trained experiment is recorded in: +# exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# ModelNet40 +# linear probing +sh scripts/train.sh -m 1 -g 8 -d utonia -c cls-utonia-v1m1-7a-modelnet40-lin -n cls-utonia-v1m1-7a-modelnet40-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +# full fine-tuning +sh scripts/train.sh -m 1 -g 8 -d utonia -c cls-utonia-v1m1-7b-modelnet40-ft -n cls-utonia-v1m1-7b-modelnet40-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# ScanObjectNN +sh scripts/train.sh -m 1 -g 8 -d utonia -c cls-utonia-v1m1-8a-scanobjectnn-lin -n cls-utonia-v1m1-8a-scanobjectnn-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c cls-utonia-v1m1-8b-scanobjectnn-ft -n cls-utonia-v1m1-8b-scanobjectnn-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c cls-utonia-v1m1-8c-scanobjectnn-lin_hard -n cls-utonia-v1m1-8c-scanobjectnn-lin_hard -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c cls-utonia-v1m1-8d-scanobjectnn-ft_hard -n cls-utonia-v1m1-8d-scanobjectnn-ft_hard -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# ShapeNetPart +sh scripts/train.sh -m 1 -g 8 -d utonia -c partseg-utonia-v1m1-9a-shapenet-lin -n partseg-utonia-v1m1-9a-shapenet-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c partseg-utonia-v1m1-9b-shapenet-ft -n partseg-utonia-v1m1-9b-shapenet-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# PartNetE +sh scripts/train.sh -m 1 -g 8 -d utonia -c partseg-utonia-v1m1-10a-partnete-lin -n partseg-utonia-v1m1-10a-partnete-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c partseg-utonia-v1m1-10b-partnete-ft -n partseg-utonia-v1m1-10b-partnete-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# ScanNet +# linear probing +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0a-scannet-lin -n semseg-utonia-v1m1-0a-scannet-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +# decoder probing +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0b-scannet-dec -n semseg-utonia-v1m1-0b-scannet-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +# full fine-tuning +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0c-scannet-ft -n semseg-utonia-v1m1-0c-scannet-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +# w/o color +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0d-scannet-nocolor-lin -n semseg-utonia-v1m1-0d-scannet-nocolor-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0e-scannet-nocolor-dec -n semseg-utonia-v1m1-0e-scannet-nocolor-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0f-scannet-nocolor-ft -n semseg-utonia-v1m1-0f-scannet-nocolor-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +# w/o normal +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0g-scannet-nonormal-lin -n semseg-utonia-v1m1-0g-scannet-nonormal-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0h-scannet-nonormal-dec -n semseg-utonia-v1m1-0h-scannet-nonormal-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-0i-scannet-nonormal-ft -n semseg-utonia-v1m1-0i-scannet-nonormal-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# ScanNet200 +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-1a-scannet200-lin -n semseg-utonia-v1m1-1a-scannet200-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-1b-scannet200-dec -n semseg-utonia-v1m1-1b-scannet200-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-1c-scannet200-ft -n semseg-utonia-v1m1-1c-scannet200-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# ScanNetpp +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-2a-scannetpp-lin -n semseg-utonia-v1m1-2a-scannetpp-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-2b-scannetpp-dec -n semseg-utonia-v1m1-2b-scannetpp-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-2c-scannetpp-ft -n semseg-utonia-v1m1-2c-scannetpp-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# S3DIS Area 5 +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-3a-s3dis-lin -n semseg-utonia-v1m1-3a-s3dis-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-3b-s3dis-dec -n semseg-utonia-v1m1-3b-s3dis-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-3c-s3dis-ft -n semseg-utonia-v1m1-3c-s3dis-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# NuScenes +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4a-nuscenes-lin -n semseg-utonia-v1m1-4a-nuscenes-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4b-nuscenes-dec -n semseg-utonia-v1m1-4b-nuscenes-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4c-nuscenes-ft -n semseg-utonia-v1m1-4c-nuscenes-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +# w/o color +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4d-nuscenes-nocolor-lin -n semseg-utonia-v1m1-0d-scannet-nocolor-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4e-nuscenes-nocolor-dec -n semseg-utonia-v1m1-4e-nuscenes-nocolor-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4f-nuscenes-nocolor-ft -n semseg-utonia-v1m1-4f-nuscenes-nocolor-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +# w/o normal +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4g-nuscenes-nonormal-lin -n semseg-utonia-v1m1-4g-nuscenes-nonormal-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4h-nuscenes-nonormal-dec -n semseg-utonia-v1m1-4h-nuscenes-nonormal-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-4i-nuscenes-nonormal-ft -n semseg-utonia-v1m1-4i-nuscenes-nonormal-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# Waymo +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-5a-waymo-lin -n semseg-utonia-v1m1-5a-waymo-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-5b-waymo-dec -n semseg-utonia-v1m1-5b-waymo-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-5c-waymo-ft -n semseg-utonia-v1m1-5c-waymo-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth + +# Kitti +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-6a-kitti-lin -n semseg-utonia-v1m1-6a-kitti-lin -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-6b-kitti-dec -n semseg-utonia-v1m1-6b-kitti-dec -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +sh scripts/train.sh -m 1 -g 8 -d utonia -c semseg-utonia-v1m1-6c-kitti-ft -n semseg-utonia-v1m1-6c-kitti-ft -w exp/utonia/pretrain-utonia-v1m1-0-base_stagev2/model/model_last.pth +``` +The above configs have not been verified with one additional run. If you encounter any problems, please feel free to let me know. Also, comparing changes between each configuration would be helpful in handling the configuration system of Pointcept. + +#### 5. Distillation +Here, we provide some distillation examples: +```bash +# you can adjust the backbone size according to your requirements, replace the "teacher_pretrained_path" in the configs with the path of "pretrain-utonia-v1m1-0-base" on the huggingface. +# we here provide the distillation with DINO branch in Concerto. Also youcan also try pure Sonata distillation. The model is in pointcept/models/sonata/sonata_v1m3_distill.py +sh scripts/train.sh -m 4 -g 8 -d utonia -c distill-utonia-v1m2-0-tiny -n distill-utonia-v1m2-0-tiny +sh scripts/train.sh -m 4 -g 8 -d utonia -c distill-utonia-v1m2-1-small -n distill-utonia-v1m2-1-small +``` + +## Citation +If you find _Utonia_ useful to your research, please consider citing our line of works as an acknowledgment. (੭ˊ꒳​ˋ)੭✧ +```bib +@misc{zhang2026utonia, + title={Utonia: Toward One Encoder for All Point Clouds}, + author={Yujia Zhang and Xiaoyang Wu and Yunhan Yang and Xianzhe Fan and Han Li and Yuechen Zhang and Zehao Huang and Naiyan Wang and Hengshuang Zhao}, + year={2026}, + eprint={2603.03283}, + archivePrefix={arXiv}, + primaryClass={cs.CV}, + url={https://arxiv.org/abs/2603.03283}, +} + +``` + +```bib +@inproceedings{zhang2025concerto, + title={Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations}, + author={Zhang, Yujia and Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Tian, Zhuotao and Wang, Naiyan and Zhao, Hengshuang}, + booktitle={NeurIPS}, + year={2025} +} +``` + +```bib +@inproceedings{wu2025sonata, + title={Sonata: Self-Supervised Learning of Reliable Point Representations}, + author={Wu, Xiaoyang and DeTone, Daniel and Frost, Duncan and Shen, Tianwei and Xie, Chris and Yang, Nan and Engel, Jakob and Newcombe, Richard and Zhao, Hengshuang and Straub, Julian}, + booktitle={CVPR}, + year={2025} +} +``` + +```bib +@inproceedings{wu2024ptv3, + title={Point Transformer V3: Simpler, Faster, Stronger}, + author={Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang}, + booktitle={CVPR}, + year={2024} +} +``` +```bib +@inproceedings{wu2024ppt, + title={Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training}, + author={Wu, Xiaoyang and Tian, Zhuotao and Wen, Xin and Peng, Bohao and Liu, Xihui and Yu, Kaicheng and Zhao, Hengshuang}, + booktitle={CVPR}, + year={2024} +} +``` +```bib +@inproceedings{wu2023masked, + title={Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning}, + author={Wu, Xiaoyang and Wen, Xin and Liu, Xihui and Zhao, Hengshuang}, + journal={CVPR}, + year={2023} +} +``` +```bib +@inproceedings{wu2022ptv2, + title={Point transformer V2: Grouped Vector Attention and Partition-based Pooling}, + author={Wu, Xiaoyang and Lao, Yixing and Jiang, Li and Liu, Xihui and Zhao, Hengshuang}, + booktitle={NeurIPS}, + year={2022} +} +``` +```bib +@misc{pointcept2023, + title={Pointcept: A Codebase for Point Cloud Perception Research}, + author={Pointcept Contributors}, + howpublished={\url{https://github.com/Pointcept/Pointcept}}, + year={2023} +} +``` + +## License +- Utonia code is based on Sonata, which is released by Meta under the [Apache 2.0 license](https://github.com/facebookresearch/sonata/blob/main/LICENSE), +- Utonia weight is released under the [CC-BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en) +(restricted by NC of datasets like HM3D, ArkitScenes). +- For commercial usage, please removing datasets restrict by non-commercial license. diff --git a/pointcept/models/utonia/__init__.py b/pointcept/models/utonia/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..d3529f76df5872af298e5fb4bc7e802936cd571b --- /dev/null +++ b/pointcept/models/utonia/__init__.py @@ -0,0 +1 @@ +from .utonia_v1m1_base import * diff --git a/pointcept/models/utonia/utonia_v1m1_base.py b/pointcept/models/utonia/utonia_v1m1_base.py new file mode 100644 index 0000000000000000000000000000000000000000..f8e22c45e4f01eca42ffe2117b19851871566eb1 --- /dev/null +++ b/pointcept/models/utonia/utonia_v1m1_base.py @@ -0,0 +1,875 @@ +""" +Utonia V1M1 + +Author: Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +from itertools import chain +from packaging import version +from functools import partial +import torch +import torch.nn as nn +import torch.nn.functional as F +import torch.distributed as dist +import torch_scatter +import torchvision.transforms +from timm.layers import trunc_normal_ +from torch.nn.utils import weight_norm +from transformers import AutoConfig, ViTModel, ViTConfig +from transformers import AutoModel, AutoProcessor +from copy import deepcopy + +import pointops +from pointcept.models.utils.structure import Point +from pointcept.models.builder import MODELS, build_model +from pointcept.models.modules import PointModel +from pointcept.models.utils import ( + offset2batch, + offset2bincount, + batch2offset, + bincount2offset, +) +from pointcept.utils.comm import get_world_size, all_gather +from pointcept.utils.scheduler import CosineScheduler + + +class OnlineCluster(nn.Module): + def __init__( + self, + in_channels, + hidden_channels=4096, + embed_channels=512, + num_prototypes=4096, + enable_mlp=True, + ): + super().__init__() + if enable_mlp: + self.mlp = nn.Sequential( + nn.Linear(in_channels, hidden_channels), + nn.GELU(), + nn.Linear(hidden_channels, embed_channels), + ) + self.apply(self._init_weights) + if version.parse(torch.__version__) >= version.parse("2.1.0"): + self.prototype = torch.nn.utils.parametrizations.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.parametrizations.weight.original0.data.fill_(1) + self.prototype.parametrizations.weight.original0.requires_grad = False + + else: + self.prototype = torch.nn.utils.weight_norm( + nn.Linear(embed_channels, num_prototypes, bias=False) + ) + self.prototype.weight_g.data.fill_(1) + self.prototype.weight_g.requires_grad = False + + @staticmethod + def _init_weights(m): + if isinstance(m, nn.Linear): + trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + + def forward(self, feat): + if hasattr(self, "mlp"): + feat = self.mlp(feat) + eps = 1e-6 if feat.dtype == torch.float16 else 1e-12 + feat = nn.functional.normalize(feat, dim=-1, p=2, eps=eps) + similarity = self.prototype(feat) + return similarity + + +@MODELS.register_module("Utonia-v1m1") +class Utonia(PointModel): + def __init__( + self, + image_weight_name, + image_weight_path, + backbone, + head_in_channels, + backbone_out_channels, + embedding_channels, + patch_w, + patch_h, + student_pretrained_path=None, + teacher_pretrained_path=None, + student_pretrained=False, + head_hidden_channels=4096, + head_embed_channels=512, + head_num_prototypes=4096, + enc2d_head_in_channels=384, + enc2d_head_hidden_channels=4096, + enc2d_head_embed_channels=256, + enc2d_head_num_prototypes=384, + teacher_custom=None, + num_global_view=2, + num_local_view=4, + mask_size_start=5, + mask_size_base=20, + mask_size_warmup_ratio=0.05, + mask_ratio_start=0.3, + mask_ratio_base=0.7, + mask_ratio_warmup_ratio=0.05, + mask_jitter=None, + teacher_temp_start=0.04, + teacher_temp_base=0.07, + teacher_temp_warmup_ratio=0.05, + student_temp=0.1, + mask_loss_weight=2 / 10, + roll_mask_loss_weight=2 / 10, + unmask_loss_weight=4 / 10, + enc2d_loss_weight=2 / 10, + momentum_base=0.996, + momentum_final=1, + match_max_k=8, + match_max_r=0.08, + up_cast_level=2, + enc2d_upcast_level=4, + enc2d_cos_shift=True, + sonata_model_type="offline", + ): + super(Utonia, self).__init__() + assert sonata_model_type in ["online", "offline"] + self.mask_loss_weight = mask_loss_weight + self.roll_mask_loss_weight = roll_mask_loss_weight + self.unmask_loss_weight = unmask_loss_weight + self.enc2d_loss_weight = enc2d_loss_weight + + self.num_global_view = num_global_view + self.num_local_view = num_local_view + + # masking and scheduler + self.mask_size = mask_size_start + self.mask_size_start = mask_size_start + self.mask_size_base = mask_size_base + self.mask_size_warmup_ratio = mask_size_warmup_ratio + self.mask_size_scheduler = None + + self.mask_ratio = mask_ratio_start + self.mask_ratio_start = mask_ratio_start + self.mask_ratio_base = mask_ratio_base + self.mask_ratio_warmup_ratio = mask_ratio_warmup_ratio + self.mask_ratio_scheduler = None + + self.mask_jitter = mask_jitter + + # temperature and scheduler + self.teacher_temp = teacher_temp_start + self.teacher_temp_start = teacher_temp_start + self.teacher_temp_base = teacher_temp_base + self.teacher_temp_warmup_ratio = teacher_temp_warmup_ratio + self.teacher_temp_scheduler = None + self.student_temp = student_temp + + # momentum and scheduler + self.momentum = momentum_base + self.momentum_base = momentum_base + self.momentum_final = momentum_final + self.momentum_scheduler = None + + # dynamic matching + self.match_max_k = match_max_k + self.match_max_r = match_max_r + + # up cast level + self.up_cast_level = up_cast_level + self.enc2d_upcast_level = enc2d_upcast_level + + # one of unmask, mask, roll mask loss enable + assert ( + unmask_loss_weight + + mask_loss_weight + + roll_mask_loss_weight + + enc2d_loss_weight + > 0 + ) + # roll mask loss need more than one global view + assert num_global_view > 1 or roll_mask_loss_weight == 0 + # current roll mask only support two global views + assert num_global_view == 1 or num_global_view == 2 + + student_model_dict = dict() + teacher_model_dict = dict() + if teacher_custom is None: + teacher_custom = {} + student_backbone = build_model(backbone) + if student_pretrained_path != None: + print("Load pretrained student model") + student_backbone = self.load_sonata( + student_backbone, path=student_pretrained_path + ) + + # turn off parameters like drop path for teacher model + backbone.update(teacher_custom) + + teacher_backbone = build_model(backbone) + if sonata_model_type == "offline": + teacher_backbone = self.load_sonata( + teacher_backbone, path=teacher_pretrained_path + ) + student_model_dict["backbone"] = student_backbone + teacher_model_dict["backbone"] = teacher_backbone + + if self.enc2d_loss_weight > 0: + self.patch_h = patch_h + self.patch_w = patch_w + self.image_weight_name = image_weight_name + + # Load Model + self.enc2d_model = self.load_enc2d(image_weight_name, image_weight_path) + self.enc2d_model.requires_grad_(False) + self._num_channels = enc2d_head_in_channels + self.patch_proj = torch.nn.Linear(backbone_out_channels, self._num_channels) + + enc2d_head = partial( + OnlineCluster, + in_channels=enc2d_head_in_channels, + hidden_channels=enc2d_head_hidden_channels, + embed_channels=enc2d_head_in_channels, + num_prototypes=enc2d_head_num_prototypes, + ) + enc2d_head_ = partial( + OnlineCluster, + in_channels=enc2d_head_in_channels, + hidden_channels=enc2d_head_hidden_channels, + embed_channels=enc2d_head_in_channels, + num_prototypes=enc2d_head_num_prototypes, + enable_mlp=False, + ) + + self.enc2d_head_student = enc2d_head() + self.enc2d_head_teacher = enc2d_head_() + self.enc2d_head_student.prototype.load_state_dict( + self.enc2d_head_teacher.prototype.state_dict() + ) + for p in self.enc2d_head_teacher.parameters(): + p.requires_grad = False + + head = partial( + OnlineCluster, + in_channels=head_in_channels, + hidden_channels=head_hidden_channels, + embed_channels=head_embed_channels, + num_prototypes=head_num_prototypes, + ) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + student_model_dict["mask_head"] = head() + teacher_model_dict["mask_head"] = head() + if self.unmask_loss_weight > 0: + student_model_dict["unmask_head"] = head() + teacher_model_dict["unmask_head"] = head() + if ( + self.enc2d_loss_weight > 0 + and self.unmask_loss_weight + + self.mask_loss_weight + + self.roll_mask_loss_weight + == 0 + ): + student_model_dict["unmask_head"] = head() + teacher_model_dict["unmask_head"] = head() + + self.student = nn.ModuleDict(student_model_dict) + self.teacher = nn.ModuleDict(teacher_model_dict) + + for k, v in self.student.items(): + if "head" in k: + self.teacher[k].load_state_dict(self.student[k].state_dict()) + if sonata_model_type == "online": + self.teacher.backbone.load_state_dict(self.student.backbone.state_dict()) + for n, p in self.teacher.named_parameters(): + p.requires_grad = False + + self.enc2d_cos_shift = enc2d_cos_shift + self.sonata_model_type = sonata_model_type + + def load_enc2d(self, model_name, model_weight): + model = AutoModel.from_pretrained(model_weight, trust_remote_code=True) + return model.eval() + + def load_sonata(self, model, path): + checkpoint = torch.load(path, map_location=lambda storage, loc: storage.cuda()) + weight = {} + whether_weight = False + if "state_dict" in checkpoint.keys(): + checkpoint = checkpoint["state_dict"] + for key, value in checkpoint.items(): + if "module.student.backbone." in key: + whether_weight = True + key = key.replace("module.student.backbone.", "module.") + key = key[7:] # module.xxx.xxx -> xxx.xxx + weight[key] = value + if whether_weight: + load_state_info = model.load_state_dict(weight) + else: + load_state_info = model.load_state_dict(checkpoint) + print(f"Missing keys: {load_state_info[0]}") + print(f"Unexpected keys: {load_state_info[1]}") + return model + + @torch.no_grad() + def ENC2D_forward(self, x): + # RADIO + if "radio" in self.image_weight_name: + summary, features = self.enc2d_model(x) + features = features.reshape( + -1, self.patch_h * self.patch_w, self._num_channels + ) + return features + # SigLIPv2 + if hasattr(self.enc2d_model, "vision_model"): + outputs = self.enc2d_model.vision_model(x) + features = outputs.last_hidden_state + # DINOv2.5 + else: + outputs = self.enc2d_model(x) + features = outputs.last_hidden_state[:, -self.patch_h * self.patch_w :, :] + return features + + def before_train(self): + # make ModelHook after CheckPointLoader + total_steps = self.trainer.cfg.scheduler.total_steps + curr_step = self.trainer.start_epoch * len(self.trainer.train_loader) + # mask size scheduler + self.mask_size_scheduler = CosineScheduler( + start_value=self.mask_size_start, + base_value=self.mask_size_base, + final_value=self.mask_size_base, + warmup_iters=int(total_steps * self.mask_size_warmup_ratio), + total_iters=total_steps, + ) + self.mask_size_scheduler.iter = curr_step + + # mask ratio scheduler + self.mask_ratio_scheduler = CosineScheduler( + start_value=self.mask_ratio_start, + base_value=self.mask_ratio_base, + final_value=self.mask_ratio_base, + warmup_iters=int(total_steps * self.mask_ratio_warmup_ratio), + total_iters=total_steps, + ) + self.mask_ratio_scheduler.iter = curr_step + + # teacher temperature scheduler + self.teacher_temp_scheduler = CosineScheduler( + start_value=self.teacher_temp_start, + base_value=self.teacher_temp_base, + final_value=self.teacher_temp_base, + warmup_iters=int(total_steps * self.teacher_temp_warmup_ratio), + total_iters=total_steps, + ) + self.teacher_temp_scheduler.iter = curr_step + + # momentum scheduler + self.momentum_scheduler = CosineScheduler( + base_value=self.momentum_base, + final_value=self.momentum_final, + total_iters=total_steps, + ) + self.momentum_scheduler.iter = curr_step + + def before_step(self): + # update parameters from schedulers + self.mask_size = self.mask_size_scheduler.step() + self.mask_ratio = self.mask_ratio_scheduler.step() + self.teacher_temp = self.teacher_temp_scheduler.step() + self.momentum = self.momentum_scheduler.step() + + if self.trainer.writer is not None: + self.trainer.writer.add_scalar( + "params/mask_size", + self.mask_size, + self.mask_size_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/mask_ratio", + self.mask_ratio, + self.mask_ratio_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/teacher_temp", + self.teacher_temp, + self.teacher_temp_scheduler.iter, + ) + self.trainer.writer.add_scalar( + "params/momentum", + self.momentum, + self.momentum_scheduler.iter, + ) + + def after_step(self): + # pass + # EMA update teacher + with torch.no_grad(): + m = self.momentum + if self.sonata_model_type == "online": + student_param_list = list(self.student.backbone.parameters()) + teacher_param_list = list(self.teacher.backbone.parameters()) + torch._foreach_mul_(teacher_param_list, m) + torch._foreach_add_(teacher_param_list, student_param_list, alpha=1 - m) + + student_param_list = [ + p for n, p in self.student.named_parameters() if "head" in n + ] + teacher_param_list = [ + p for n, p in self.teacher.named_parameters() if "head" in n + ] + torch._foreach_mul_(teacher_param_list, m) + torch._foreach_add_(teacher_param_list, student_param_list, alpha=1 - m) + + if self.enc2d_loss_weight > 0: + enc2d_student_param_list = [ + p + for n, p in self.enc2d_head_student.named_parameters() + if "prototype" in n + ] + enc2d_teacher_param_list = [ + p + for n, p in self.enc2d_head_teacher.named_parameters() + if "prototype" in n + ] + torch._foreach_copy_(enc2d_teacher_param_list, enc2d_student_param_list) + + @staticmethod + def sinkhorn_knopp(feat, temp, num_iter=3): + feat = feat.float() + q = torch.exp(feat / temp).t() + n = sum(all_gather(q.shape[1])) # number of samples to assign + k = q.shape[0] # number of prototypes + + # make the matrix sums to 1 + sum_q = q.sum() + if get_world_size() > 1: + dist.all_reduce(sum_q) + q = q / sum_q + + for i in range(num_iter): + # normalize each row: total weight per prototype must be 1/k + q_row_sum = q.sum(dim=1, keepdim=True) + if get_world_size() > 1: + dist.all_reduce(q_row_sum) + q = q / q_row_sum / k + + # normalize each column: total weight per sample must be 1/n + q = q / q.sum(dim=0, keepdim=True) / n + + q *= n # the columns must sum to 1 so that Q is an assignment + return q.t() + + def generate_mask(self, coord, offset, grid_size): + batch = offset2batch(offset) + mask_size = self.mask_size * grid_size + mask_ratio = self.mask_ratio + + # Grouping points with grid patch + min_coord = torch_scatter.segment_coo(coord, batch, reduce="min") + grid_coord = ((coord - min_coord[batch]) // mask_size).int() + grid_coord = torch.cat([batch.unsqueeze(-1), grid_coord], dim=-1) + unique, point_cluster, counts = torch.unique( + grid_coord, dim=0, sorted=True, return_inverse=True, return_counts=True + ) + patch_num = unique.shape[0] + mask_patch_num = int(patch_num * mask_ratio) + patch_index = torch.randperm(patch_num, device=coord.device) + mask_patch_index = patch_index[:mask_patch_num] + point_mask = torch.isin(point_cluster, mask_patch_index) + return point_mask, point_cluster + + @torch.no_grad() + def match_neighbour( + self, + view1_coord, + view1_offset, + view2_coord, + view2_offset, + ): + index2, distance = pointops.knn_query( + 1, + view2_coord.float(), + view2_offset.int(), + view1_coord.float(), + view1_offset.int(), + ) + index1 = torch.arange( + index2.shape[0], device=index2.device, dtype=torch.long + ).unsqueeze(-1) + index = torch.cat([index1, index2], dim=-1)[ + distance.squeeze(-1) < self.match_max_r + ] + return index + + @torch.no_grad() + def roll_point(self, point): + n = self.num_global_view + # [pc1, pc1', pc2, pc2'] -> [pc1', pc1, pc2', pc2], only support num_global_view == 2 + bs = len(point.offset) // self.num_global_view + data_dict = {} + for key in point.keys(): + if key in ["feat", "coord", "origin_coord", "batch"]: + value = point[key].split(offset2bincount(point.offset).tolist()) + value = chain(*[value[n * b : n * (b + 1)][::-1] for b in range(bs)]) + if key == "batch": + value = [torch.ones_like(v) * i for i, v in enumerate(value)] + data_dict[key] = torch.cat(list(value), dim=0) + return Point(data_dict) + + def up_cast(self, point, upcast_level=None): + if upcast_level is None: + upcast_level = self.up_cast_level + else: + upcast_level = upcast_level + for _ in range(upcast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return point + + @staticmethod + def pool_corr(point, correspondence): + inverse_list = [] + idx_ptr_list = [] + point_feat = dict(offset=point.offset, feat=point.feat) + while "pooling_parent" in point.keys(): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + assert "idx_ptr" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + idx_ptr = point.pop("idx_ptr") + inverse_list.append(inverse) + idx_ptr_list.append(idx_ptr) + point = parent + inverse_list.reverse() + idx_ptr_list.reverse() + for inverse, idx_ptr in zip(inverse_list, idx_ptr_list): + _, indices = torch.sort(inverse) + img_num = correspondence.shape[1] + correspondence_all = [] + if img_num == 0: + correspondence_all = -torch.ones((idx_ptr.shape[0] - 1, 0, 2)).cuda() + else: + for img_id in range(img_num): + mask = torch.all( + correspondence[:, img_id] != torch.tensor([-1, -1]).cuda(), + dim=1, + ).float() + counts = torch_scatter.segment_csr( + mask[indices], idx_ptr, reduce="sum" + ) + counts[counts == 0] = 100000 + correspondence_img = deepcopy(correspondence[:, img_id]) + correspondence_img[correspondence_img == -1] = 0 + mask_sum = torch_scatter.segment_csr( + correspondence_img[indices], idx_ptr, reduce="sum" + ) + mask_sum = mask_sum / counts.unsqueeze(1) + mask_sum[counts == 100000] = -1 + correspondence_all.append(mask_sum) + correspondence_all = torch.stack(correspondence_all, dim=1) + correspondence = correspondence_all + point_feat["correspondence"] = correspondence + point_feat = Point(point_feat) + return point_feat + + def forward(self, data_dict, return_point=False): + if return_point: + point = self.teacher.backbone(data_dict) + for _ in range(self.up_cast_level): + assert "pooling_parent" in point.keys() + assert "pooling_inverse" in point.keys() + parent = point.pop("pooling_parent") + inverse = point.pop("pooling_inverse") + parent.feat = torch.cat([parent.feat, point.feat[inverse]], dim=-1) + point = parent + return dict(point=point) + + # prepare global_point, mask_global_point, local_point + with torch.no_grad(): + # global_point & masking + global_point = Point( + feat=data_dict["global_feat"], + coord=data_dict["global_coord"], + origin_coord=data_dict["global_origin_coord"], + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + + global_mask, global_cluster = self.generate_mask( + global_point.coord, global_point.offset, global_point.grid_size + ) + mask_global_coord = global_point.coord.clone().detach() + if self.mask_jitter is not None: + mask_global_coord[global_mask] += torch.clip( + torch.randn_like(mask_global_coord[global_mask]).mul( + self.mask_jitter * data_dict["grid_size"][0] + ), + max=(self.mask_jitter * data_dict["grid_size"][0]) * 2, + ) + + mask_global_point = Point( + feat=data_dict["global_feat"], + coord=mask_global_coord, + origin_coord=data_dict["global_origin_coord"], + mask=global_mask, + offset=data_dict["global_offset"], + grid_size=data_dict["grid_size"][0], + ) + major_view_correspondence = data_dict["global_correspondence"] + + # local point & matching + local_point = Point( + feat=data_dict["local_feat"], + coord=data_dict["local_coord"], + origin_coord=data_dict["local_origin_coord"], + offset=data_dict["local_offset"], + grid_size=data_dict["grid_size"][0], + ) + + # create result dictionary for return + result_dict = dict(loss=[]) + # teacher forward + global_point_ = self.teacher.backbone(global_point) + global_point_ = self.up_cast(global_point_) + # teacher head forward + # only use one shared head for both mask and unmask + # priority: mask (global) > unmask (local) + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + global_point_.feat = self.teacher.mask_head(global_point_.feat) + else: + global_point_.feat = self.teacher.unmask_head(global_point_.feat) + + if self.mask_loss_weight > 0 or self.roll_mask_loss_weight > 0: + # student forward + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_pred_sim = self.student.mask_head(mask_global_point_.feat) + + if self.mask_loss_weight > 0: + with torch.no_grad(): + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + global_point_.origin_coord, + global_point_.offset, + ) + # teacher forward + mask_target_sim = self.sinkhorn_knopp( + global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + # loss + mask_loss = -torch.sum( + mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + + mask_loss = torch_scatter.segment_coo( + mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["mask_loss"] = mask_loss + result_dict["loss"].append(mask_loss * self.mask_loss_weight) + + if self.roll_mask_loss_weight > 0: + roll_global_point_ = self.roll_point(global_point_) + with torch.no_grad(): + # match index for pred and roll target + match_index = self.match_neighbour( + mask_global_point_.origin_coord, + mask_global_point_.offset, + roll_global_point_.origin_coord, + roll_global_point_.offset, + ) + # teacher forward + roll_mask_target_sim = self.sinkhorn_knopp( + roll_global_point_.feat[match_index[:, 1]], + self.teacher_temp, + ) + + roll_mask_loss = -torch.sum( + roll_mask_target_sim + * F.log_softmax( + mask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + roll_mask_loss = torch_scatter.segment_coo( + roll_mask_loss, + index=mask_global_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["roll_mask_loss"] = roll_mask_loss + result_dict["loss"].append(roll_mask_loss * self.roll_mask_loss_weight) + if self.unmask_loss_weight > 0: + # student forward + local_point_ = self.student.backbone(local_point) + local_point_ = self.up_cast(local_point_) + unmask_pred_sim = self.student.unmask_head(local_point_.feat) + with torch.no_grad(): + principal_view_mask = global_point_.batch % self.num_global_view == 0 + principal_view_batch = ( + global_point_.batch[principal_view_mask] // self.num_global_view + ) + match_index = self.match_neighbour( + local_point_.origin_coord, + local_point_.offset[self.num_local_view - 1 :: self.num_local_view], + global_point_.origin_coord[principal_view_mask], + batch2offset(principal_view_batch), + ) + # teacher forward + unmask_target_sim = self.sinkhorn_knopp( + global_point_.feat[principal_view_mask][match_index[:, 1]], + self.teacher_temp, + ) + # loss + unmask_loss = -torch.sum( + unmask_target_sim + * F.log_softmax( + unmask_pred_sim[match_index[:, 0]] / self.student_temp, dim=-1 + ), + dim=-1, + ) + unmask_loss = torch_scatter.segment_coo( + unmask_loss, + index=local_point_.batch[match_index[:, 0]], + reduce="mean", + ).mean() + result_dict["unmask_loss"] = unmask_loss + result_dict["loss"].append(unmask_loss * self.unmask_loss_weight) + if self.enc2d_loss_weight > 0: + if self.mask_loss_weight == 0 or self.roll_mask_loss_weight == 0: + mask_global_point_ = self.student.backbone(mask_global_point) + mask_global_point_ = self.up_cast(mask_global_point_) + mask_global_point_enc2d = self.up_cast( + mask_global_point_, + upcast_level=self.enc2d_upcast_level - self.up_cast_level, + ) + to_feature = self.pool_corr( + mask_global_point_enc2d, major_view_correspondence + ) + data_dict_global_offset = torch.cat( + [torch.tensor([0]).cuda(), to_feature["offset"]], dim=0 + ) + enc2d_count = ( + data_dict_global_offset[ + 1 : len(data_dict_global_offset) : self.num_global_view + ] + - data_dict_global_offset[ + 0 : len(data_dict_global_offset) - 1 : self.num_global_view + ] + ) + enc2d_offset = torch.cat( + [torch.tensor([0]).cuda(), torch.cumsum(enc2d_count, dim=0)] + ) + enc2d_mask = torch.cat( + [ + torch.arange(0, c, device=enc2d_count.device) + + data_dict_global_offset[i * self.num_global_view] + for i, c in enumerate(enc2d_count) + ], + dim=0, + ) + + offset_points_3d = enc2d_offset[1:] + batch_points_3d = offset2batch(offset_points_3d) + imgs = data_dict["images"] + feature3d = to_feature["feat"][enc2d_mask] + enc2d_global_mask = global_mask[enc2d_mask] + correspondence = to_feature["correspondence"][enc2d_mask] + v0 = correspondence.shape[1] + mask = torch.any(correspondence != torch.tensor([-1, -1]).cuda(), dim=2) + enc2d_global_mask = enc2d_global_mask.unsqueeze(1).expand(-1, v0) + valid_index = torch.where(mask) # 0: 3d points index, 1: view index + + bincount_img_num = data_dict["img_num"] + offset_img_num = bincount2offset(bincount_img_num) + total_img_num = offset_img_num[-1] + + if total_img_num > 0: + # expand + with torch.no_grad(): + feature2d = self.ENC2D_forward(imgs) + feature2d = feature2d.contiguous().view(-1, feature2d.shape[-1]) + feature2d_mask = feature2d + + offset_img_num = torch.cat([torch.tensor([0]).cuda(), offset_img_num])[ + :-1 + ] + batch_index = batch_points_3d[valid_index[0]] + batch_img_num = offset_img_num[batch_index] + + feature3d_mask = feature3d[valid_index[0]] + + feature_index = torch.cat( + [ + batch_img_num.unsqueeze(-1), + valid_index[1].unsqueeze(-1), + correspondence[valid_index], + ], + dim=-1, + ) + feature_index = feature_index.long() + feature_index = ( + feature_index[:, 0] * self.patch_h * self.patch_w + + feature_index[:, 1] * self.patch_h * self.patch_w + + feature_index[:, 2] * self.patch_w + + feature_index[:, 3] + ) + + feature_index = feature_index.long() + feature3d_mask = torch_scatter.scatter_mean( + feature3d_mask, feature_index, dim=0, dim_size=feature2d.shape[0] + ) + feature3d_mask = self.patch_proj(feature3d_mask) + feature_index = torch.unique(feature_index) + feature2d_mask = feature2d_mask[feature_index] + feature3d_mask = feature3d_mask[feature_index] + + if self.enc2d_cos_shift: + feature2d_mask = feature2d_mask - feature2d_mask.mean( + dim=-1, keepdim=True + ) + feature3d_mask = feature3d_mask - feature3d_mask.mean( + dim=-1, keepdim=True + ) + cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6) + loss = (1 - cos(feature2d_mask, feature3d_mask)).mean() * 10 + + result_dict["enc2d_loss"] = loss + result_dict["loss"].append(loss * self.enc2d_loss_weight) + del ( + feature2d, + feature3d, + feature2d_mask, + feature3d_mask, + correspondence, + feature_index, + ) + elif ( + self.mask_loss_weight + + self.unmask_loss_weight + + self.roll_mask_loss_weight + > 0 + ): + result_ssl_loss = sum(result_dict["loss"]) / ( + self.mask_loss_weight + + self.unmask_loss_weight + + self.roll_mask_loss_weight + ) + result_dict["enc2d_loss"] = result_ssl_loss + result_dict["loss"].append(result_ssl_loss * self.enc2d_loss_weight) + result_dict["loss"] = sum(result_dict["loss"]) + + if get_world_size() > 1: + for loss_id, loss in result_dict.items(): + dist.all_reduce(loss, op=dist.ReduceOp.AVG) + return result_dict diff --git a/pointcept/utils/__init__.py b/pointcept/utils/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/pointcept/utils/cache.py b/pointcept/utils/cache.py new file mode 100644 index 0000000000000000000000000000000000000000..c7aec25f730b476bf0655c95c9a53272cc133f7a --- /dev/null +++ b/pointcept/utils/cache.py @@ -0,0 +1,60 @@ +""" +Data Cache Utils + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os + +try: + import SharedArray +except ImportError: + SharedArray = None + +try: + from multiprocessing.shared_memory import ShareableList +except ImportError: + import warnings + + warnings.warn("Please update python version >= 3.8 to enable shared_memory") +import numpy as np + + +def shared_array(name, var=None): + if var is not None: + # check exist + if os.path.exists(f"/dev/shm/{name}"): + return SharedArray.attach(f"shm://{name}") + # create shared_array + data = SharedArray.create(f"shm://{name}", var.shape, dtype=var.dtype) + data[...] = var[...] + data.flags.writeable = False + else: + data = SharedArray.attach(f"shm://{name}").copy() + return data + + +def shared_dict(name, var=None): + name = str(name) + assert "." not in name # '.' is used as sep flag + data = {} + if var is not None: + assert isinstance(var, dict) + keys = var.keys() + # current version only cache np.array + keys_valid = [] + for key in keys: + if isinstance(var[key], np.ndarray): + keys_valid.append(key) + keys = keys_valid + + ShareableList(sequence=keys, name=name + ".keys") + for key in keys: + if isinstance(var[key], np.ndarray): + data[key] = shared_array(name=f"{name}.{key}", var=var[key]) + else: + keys = list(ShareableList(name=name + ".keys")) + for key in keys: + data[key] = shared_array(name=f"{name}.{key}") + return data diff --git a/pointcept/utils/comm.py b/pointcept/utils/comm.py new file mode 100644 index 0000000000000000000000000000000000000000..69e29e7c690fe0500d3d9a84b6a8749e2f4f4655 --- /dev/null +++ b/pointcept/utils/comm.py @@ -0,0 +1,198 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +""" +This file contains primitives for multi-gpu communication. +This is useful when doing distributed training. +Modified from detectron2(https://github.com/facebookresearch/detectron2) + +Copyright (c) Xiaoyang Wu (xiaoyang.wu@connect.hku.hk). All Rights Reserved. +Please cite our work if you use any part of the code. +""" + +import functools +import numpy as np +import torch +import torch.distributed as dist + +_LOCAL_PROCESS_GROUP = None +""" +A torch process group which only includes processes that on the same machine as the current process. +This variable is set when processes are spawned by `launch()` in "engine/launch.py". +""" + + +def get_world_size() -> int: + if not dist.is_available(): + return 1 + if not dist.is_initialized(): + return 1 + return dist.get_world_size() + + +def get_rank() -> int: + if not dist.is_available(): + return 0 + if not dist.is_initialized(): + return 0 + return dist.get_rank() + + +def get_local_rank() -> int: + """ + Returns: + The rank of the current process within the local (per-machine) process group. + """ + if not dist.is_available(): + return 0 + if not dist.is_initialized(): + return 0 + assert ( + _LOCAL_PROCESS_GROUP is not None + ), "Local process group is not created! Please use launch() to spawn processes!" + return dist.get_rank(group=_LOCAL_PROCESS_GROUP) + + +def get_local_size() -> int: + """ + Returns: + The size of the per-machine process group, + i.e. the number of processes per machine. + """ + if not dist.is_available(): + return 1 + if not dist.is_initialized(): + return 1 + return dist.get_world_size(group=_LOCAL_PROCESS_GROUP) + + +def is_main_process() -> bool: + return get_rank() == 0 + + +def synchronize(): + """ + Helper function to synchronize (barrier) among all processes when + using distributed training + """ + if not dist.is_available(): + return + if not dist.is_initialized(): + return + world_size = dist.get_world_size() + if world_size == 1: + return + if dist.get_backend() == dist.Backend.NCCL: + # This argument is needed to avoid warnings. + # It's valid only for NCCL backend. + dist.barrier(device_ids=[torch.cuda.current_device()]) + else: + dist.barrier() + + +@functools.lru_cache() +def _get_global_gloo_group(): + """ + Return a process group based on gloo backend, containing all the ranks + The result is cached. + """ + if dist.get_backend() == "nccl": + return dist.new_group(backend="gloo") + else: + return dist.group.WORLD + + +def all_gather(data, group=None): + """ + Run all_gather on arbitrary picklable data (not necessarily tensors). + Args: + data: any picklable object + group: a torch process group. By default, will use a group which + contains all ranks on gloo backend. + Returns: + list[data]: list of data gathered from each rank + """ + if get_world_size() == 1: + return [data] + if group is None: + group = ( + _get_global_gloo_group() + ) # use CPU group by default, to reduce GPU RAM usage. + world_size = dist.get_world_size(group) + if world_size == 1: + return [data] + + output = [None for _ in range(world_size)] + dist.all_gather_object(output, data, group=group) + return output + + +def gather(data, dst=0, group=None): + """ + Run gather on arbitrary picklable data (not necessarily tensors). + Args: + data: any picklable object + dst (int): destination rank + group: a torch process group. By default, will use a group which + contains all ranks on gloo backend. + Returns: + list[data]: on dst, a list of data gathered from each rank. Otherwise, + an empty list. + """ + if get_world_size() == 1: + return [data] + if group is None: + group = _get_global_gloo_group() + world_size = dist.get_world_size(group=group) + if world_size == 1: + return [data] + rank = dist.get_rank(group=group) + + if rank == dst: + output = [None for _ in range(world_size)] + dist.gather_object(data, output, dst=dst, group=group) + return output + else: + dist.gather_object(data, None, dst=dst, group=group) + return [] + + +def shared_random_seed(): + """ + Returns: + int: a random number that is the same across all workers. + If workers need a shared RNG, they can use this shared seed to + create one. + All workers must call this function, otherwise it will deadlock. + """ + ints = np.random.randint(2**31) + all_ints = all_gather(ints) + return all_ints[0] + + +def reduce_dict(input_dict, average=True): + """ + Reduce the values in the dictionary from all processes so that process with rank + 0 has the reduced results. + Args: + input_dict (dict): inputs to be reduced. All the values must be scalar CUDA Tensor. + average (bool): whether to do average or sum + Returns: + a dict with the same keys as input_dict, after reduction. + """ + world_size = get_world_size() + if world_size < 2: + return input_dict + with torch.no_grad(): + names = [] + values = [] + # sort the keys so that they are consistent across processes + for k in sorted(input_dict.keys()): + names.append(k) + values.append(input_dict[k]) + values = torch.stack(values, dim=0) + dist.reduce(values, dst=0) + if dist.get_rank() == 0 and average: + # only main process gets accumulated, so only divide by + # world_size in this case + values /= world_size + reduced_dict = {k: v for k, v in zip(names, values)} + return reduced_dict diff --git a/pointcept/utils/config.py b/pointcept/utils/config.py new file mode 100644 index 0000000000000000000000000000000000000000..762ebf4f151543e81d041d870cbd507feff6a02c --- /dev/null +++ b/pointcept/utils/config.py @@ -0,0 +1,694 @@ +# Copyright (c) OpenMMLab. All rights reserved. +import ast +import copy +import os +import os.path as osp +import platform +import shutil +import sys +import tempfile +import uuid +import warnings +from argparse import Action, ArgumentParser +from collections import abc +from importlib import import_module + +from addict import Dict +from yapf.yapflib.yapf_api import FormatCode + +from .misc import import_modules_from_strings +from .path import check_file_exist + +if platform.system() == "Windows": + import regex as re +else: + import re + +BASE_KEY = "_base_" +DELETE_KEY = "_delete_" +DEPRECATION_KEY = "_deprecation_" +RESERVED_KEYS = ["filename", "text", "pretty_text"] + + +class ConfigDict(Dict): + def __missing__(self, name): + raise KeyError(name) + + def __getattr__(self, name): + try: + value = super(ConfigDict, self).__getattr__(name) + except KeyError: + ex = AttributeError( + f"'{self.__class__.__name__}' object has no " f"attribute '{name}'" + ) + except Exception as e: + ex = e + else: + return value + raise ex + + +def add_args(parser, cfg, prefix=""): + for k, v in cfg.items(): + if isinstance(v, str): + parser.add_argument("--" + prefix + k) + elif isinstance(v, int): + parser.add_argument("--" + prefix + k, type=int) + elif isinstance(v, float): + parser.add_argument("--" + prefix + k, type=float) + elif isinstance(v, bool): + parser.add_argument("--" + prefix + k, action="store_true") + elif isinstance(v, dict): + add_args(parser, v, prefix + k + ".") + elif isinstance(v, abc.Iterable): + parser.add_argument("--" + prefix + k, type=type(v[0]), nargs="+") + else: + print(f"cannot parse key {prefix + k} of type {type(v)}") + return parser + + +class Config: + """A facility for config and config files. + + It supports common file formats as configs: python/json/yaml. The interface + is the same as a dict object and also allows access config values as + attributes. + + Example: + >>> cfg = Config(dict(a=1, b=dict(b1=[0, 1]))) + >>> cfg.a + 1 + >>> cfg.b + {'b1': [0, 1]} + >>> cfg.b.b1 + [0, 1] + >>> cfg = Config.fromfile('tests/data/config/a.py') + >>> cfg.filename + "/home/kchen/projects/mmcv/tests/data/config/a.py" + >>> cfg.item4 + 'test' + >>> cfg + "Config [path: /home/kchen/projects/mmcv/tests/data/config/a.py]: " + "{'item1': [1, 2], 'item2': {'a': 0}, 'item3': True, 'item4': 'test'}" + """ + + @staticmethod + def _validate_py_syntax(filename): + with open(filename, "r", encoding="utf-8") as f: + # Setting encoding explicitly to resolve coding issue on windows + content = f.read() + try: + ast.parse(content) + except SyntaxError as e: + raise SyntaxError( + "There are syntax errors in config " f"file {filename}: {e}" + ) + + @staticmethod + def _substitute_predefined_vars(filename, temp_config_name): + file_dirname = osp.dirname(filename) + file_basename = osp.basename(filename) + file_basename_no_extension = osp.splitext(file_basename)[0] + file_extname = osp.splitext(filename)[1] + support_templates = dict( + fileDirname=file_dirname, + fileBasename=file_basename, + fileBasenameNoExtension=file_basename_no_extension, + fileExtname=file_extname, + ) + with open(filename, "r", encoding="utf-8") as f: + # Setting encoding explicitly to resolve coding issue on windows + config_file = f.read() + for key, value in support_templates.items(): + regexp = r"\{\{\s*" + str(key) + r"\s*\}\}" + value = value.replace("\\", "/") + config_file = re.sub(regexp, value, config_file) + with open(temp_config_name, "w", encoding="utf-8") as tmp_config_file: + tmp_config_file.write(config_file) + + @staticmethod + def _pre_substitute_base_vars(filename, temp_config_name): + """Substitute base variable placehoders to string, so that parsing + would work.""" + with open(filename, "r", encoding="utf-8") as f: + # Setting encoding explicitly to resolve coding issue on windows + config_file = f.read() + base_var_dict = {} + regexp = r"\{\{\s*" + BASE_KEY + r"\.([\w\.]+)\s*\}\}" + base_vars = set(re.findall(regexp, config_file)) + for base_var in base_vars: + randstr = f"_{base_var}_{uuid.uuid4().hex.lower()[:6]}" + base_var_dict[randstr] = base_var + regexp = r"\{\{\s*" + BASE_KEY + r"\." + base_var + r"\s*\}\}" + config_file = re.sub(regexp, f'"{randstr}"', config_file) + with open(temp_config_name, "w", encoding="utf-8") as tmp_config_file: + tmp_config_file.write(config_file) + return base_var_dict + + @staticmethod + def _substitute_base_vars(cfg, base_var_dict, base_cfg): + """Substitute variable strings to their actual values.""" + cfg = copy.deepcopy(cfg) + + if isinstance(cfg, dict): + for k, v in cfg.items(): + if isinstance(v, str) and v in base_var_dict: + new_v = base_cfg + for new_k in base_var_dict[v].split("."): + new_v = new_v[new_k] + cfg[k] = new_v + elif isinstance(v, (list, tuple, dict)): + cfg[k] = Config._substitute_base_vars(v, base_var_dict, base_cfg) + elif isinstance(cfg, tuple): + cfg = tuple( + Config._substitute_base_vars(c, base_var_dict, base_cfg) for c in cfg + ) + elif isinstance(cfg, list): + cfg = [ + Config._substitute_base_vars(c, base_var_dict, base_cfg) for c in cfg + ] + elif isinstance(cfg, str) and cfg in base_var_dict: + new_v = base_cfg + for new_k in base_var_dict[cfg].split("."): + new_v = new_v[new_k] + cfg = new_v + + return cfg + + @staticmethod + def _file2dict(filename, use_predefined_variables=True): + filename = osp.abspath(osp.expanduser(filename)) + check_file_exist(filename) + fileExtname = osp.splitext(filename)[1] + if fileExtname not in [".py", ".json", ".yaml", ".yml"]: + raise IOError("Only py/yml/yaml/json type are supported now!") + + with tempfile.TemporaryDirectory() as temp_config_dir: + temp_config_file = tempfile.NamedTemporaryFile( + dir=temp_config_dir, suffix=fileExtname + ) + if platform.system() == "Windows": + temp_config_file.close() + temp_config_name = osp.basename(temp_config_file.name) + # Substitute predefined variables + if use_predefined_variables: + Config._substitute_predefined_vars(filename, temp_config_file.name) + else: + shutil.copyfile(filename, temp_config_file.name) + # Substitute base variables from placeholders to strings + base_var_dict = Config._pre_substitute_base_vars( + temp_config_file.name, temp_config_file.name + ) + + if filename.endswith(".py"): + temp_module_name = osp.splitext(temp_config_name)[0] + sys.path.insert(0, temp_config_dir) + Config._validate_py_syntax(filename) + mod = import_module(temp_module_name) + sys.path.pop(0) + cfg_dict = { + name: value + for name, value in mod.__dict__.items() + if not name.startswith("__") + } + # delete imported module + del sys.modules[temp_module_name] + elif filename.endswith((".yml", ".yaml", ".json")): + raise NotImplementedError + # close temp file + temp_config_file.close() + + # check deprecation information + if DEPRECATION_KEY in cfg_dict: + deprecation_info = cfg_dict.pop(DEPRECATION_KEY) + warning_msg = ( + f"The config file {filename} will be deprecated " "in the future." + ) + if "expected" in deprecation_info: + warning_msg += f' Please use {deprecation_info["expected"]} ' "instead." + if "reference" in deprecation_info: + warning_msg += ( + " More information can be found at " + f'{deprecation_info["reference"]}' + ) + warnings.warn(warning_msg) + + cfg_text = filename + "\n" + with open(filename, "r", encoding="utf-8") as f: + # Setting encoding explicitly to resolve coding issue on windows + cfg_text += f.read() + + if BASE_KEY in cfg_dict: + cfg_dir = osp.dirname(filename) + base_filename = cfg_dict.pop(BASE_KEY) + base_filename = ( + base_filename if isinstance(base_filename, list) else [base_filename] + ) + + cfg_dict_list = list() + cfg_text_list = list() + for f in base_filename: + _cfg_dict, _cfg_text = Config._file2dict(osp.join(cfg_dir, f)) + cfg_dict_list.append(_cfg_dict) + cfg_text_list.append(_cfg_text) + + base_cfg_dict = dict() + for c in cfg_dict_list: + duplicate_keys = base_cfg_dict.keys() & c.keys() + if len(duplicate_keys) > 0: + raise KeyError( + "Duplicate key is not allowed among bases. " + f"Duplicate keys: {duplicate_keys}" + ) + base_cfg_dict.update(c) + + # Substitute base variables from strings to their actual values + cfg_dict = Config._substitute_base_vars( + cfg_dict, base_var_dict, base_cfg_dict + ) + + base_cfg_dict = Config._merge_a_into_b(cfg_dict, base_cfg_dict) + cfg_dict = base_cfg_dict + + # merge cfg_text + cfg_text_list.append(cfg_text) + cfg_text = "\n".join(cfg_text_list) + + return cfg_dict, cfg_text + + @staticmethod + def _merge_a_into_b(a, b, allow_list_keys=False): + """merge dict ``a`` into dict ``b`` (non-inplace). + + Values in ``a`` will overwrite ``b``. ``b`` is copied first to avoid + in-place modifications. + + Args: + a (dict): The source dict to be merged into ``b``. + b (dict): The origin dict to be fetch keys from ``a``. + allow_list_keys (bool): If True, int string keys (e.g. '0', '1') + are allowed in source ``a`` and will replace the element of the + corresponding index in b if b is a list. Default: False. + + Returns: + dict: The modified dict of ``b`` using ``a``. + + Examples: + # Normally merge a into b. + >>> Config._merge_a_into_b( + ... dict(obj=dict(a=2)), dict(obj=dict(a=1))) + {'obj': {'a': 2}} + + # Delete b first and merge a into b. + >>> Config._merge_a_into_b( + ... dict(obj=dict(_delete_=True, a=2)), dict(obj=dict(a=1))) + {'obj': {'a': 2}} + + # b is a list + >>> Config._merge_a_into_b( + ... {'0': dict(a=2)}, [dict(a=1), dict(b=2)], True) + [{'a': 2}, {'b': 2}] + """ + b = b.copy() + for k, v in a.items(): + if allow_list_keys and k.isdigit() and isinstance(b, list): + k = int(k) + if len(b) <= k: + raise KeyError(f"Index {k} exceeds the length of list {b}") + b[k] = Config._merge_a_into_b(v, b[k], allow_list_keys) + elif isinstance(v, dict) and k in b and not v.pop(DELETE_KEY, False): + allowed_types = (dict, list) if allow_list_keys else dict + if not isinstance(b[k], allowed_types): + raise TypeError( + f"{k}={v} in child config cannot inherit from base " + f"because {k} is a dict in the child config but is of " + f"type {type(b[k])} in base config. You may set " + f"`{DELETE_KEY}=True` to ignore the base config" + ) + b[k] = Config._merge_a_into_b(v, b[k], allow_list_keys) + else: + b[k] = v + return b + + @staticmethod + def fromfile(filename, use_predefined_variables=True, import_custom_modules=True): + cfg_dict, cfg_text = Config._file2dict(filename, use_predefined_variables) + if import_custom_modules and cfg_dict.get("custom_imports", None): + import_modules_from_strings(**cfg_dict["custom_imports"]) + return Config(cfg_dict, cfg_text=cfg_text, filename=filename) + + @staticmethod + def fromstring(cfg_str, file_format): + """Generate config from config str. + + Args: + cfg_str (str): Config str. + file_format (str): Config file format corresponding to the + config str. Only py/yml/yaml/json type are supported now! + + Returns: + obj:`Config`: Config obj. + """ + if file_format not in [".py", ".json", ".yaml", ".yml"]: + raise IOError("Only py/yml/yaml/json type are supported now!") + if file_format != ".py" and "dict(" in cfg_str: + # check if users specify a wrong suffix for python + warnings.warn('Please check "file_format", the file format may be .py') + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", suffix=file_format, delete=False + ) as temp_file: + temp_file.write(cfg_str) + # on windows, previous implementation cause error + # see PR 1077 for details + cfg = Config.fromfile(temp_file.name) + os.remove(temp_file.name) + return cfg + + @staticmethod + def auto_argparser(description=None): + """Generate argparser from config file automatically (experimental)""" + partial_parser = ArgumentParser(description=description) + partial_parser.add_argument("config", help="config file path") + cfg_file = partial_parser.parse_known_args()[0].config + cfg = Config.fromfile(cfg_file) + parser = ArgumentParser(description=description) + parser.add_argument("config", help="config file path") + add_args(parser, cfg) + return parser, cfg + + def __init__(self, cfg_dict=None, cfg_text=None, filename=None): + if cfg_dict is None: + cfg_dict = dict() + elif not isinstance(cfg_dict, dict): + raise TypeError("cfg_dict must be a dict, but " f"got {type(cfg_dict)}") + for key in cfg_dict: + if key in RESERVED_KEYS: + raise KeyError(f"{key} is reserved for config file") + + super(Config, self).__setattr__("_cfg_dict", ConfigDict(cfg_dict)) + super(Config, self).__setattr__("_filename", filename) + if cfg_text: + text = cfg_text + elif filename: + with open(filename, "r") as f: + text = f.read() + else: + text = "" + super(Config, self).__setattr__("_text", text) + + @property + def filename(self): + return self._filename + + @property + def text(self): + return self._text + + @property + def pretty_text(self): + indent = 4 + + def _indent(s_, num_spaces): + s = s_.split("\n") + if len(s) == 1: + return s_ + first = s.pop(0) + s = [(num_spaces * " ") + line for line in s] + s = "\n".join(s) + s = first + "\n" + s + return s + + def _format_basic_types(k, v, use_mapping=False): + if isinstance(v, str): + v_str = f"'{v}'" + else: + v_str = str(v) + + if use_mapping: + k_str = f"'{k}'" if isinstance(k, str) else str(k) + attr_str = f"{k_str}: {v_str}" + else: + attr_str = f"{str(k)}={v_str}" + attr_str = _indent(attr_str, indent) + + return attr_str + + def _format_list(k, v, use_mapping=False): + # check if all items in the list are dict + if all(isinstance(_, dict) for _ in v): + v_str = "[\n" + v_str += "\n".join( + f"dict({_indent(_format_dict(v_), indent)})," for v_ in v + ).rstrip(",") + if use_mapping: + k_str = f"'{k}'" if isinstance(k, str) else str(k) + attr_str = f"{k_str}: {v_str}" + else: + attr_str = f"{str(k)}={v_str}" + attr_str = _indent(attr_str, indent) + "]" + else: + attr_str = _format_basic_types(k, v, use_mapping) + return attr_str + + def _contain_invalid_identifier(dict_str): + contain_invalid_identifier = False + for key_name in dict_str: + contain_invalid_identifier |= not str(key_name).isidentifier() + return contain_invalid_identifier + + def _format_dict(input_dict, outest_level=False): + r = "" + s = [] + + use_mapping = _contain_invalid_identifier(input_dict) + if use_mapping: + r += "{" + for idx, (k, v) in enumerate(input_dict.items()): + is_last = idx >= len(input_dict) - 1 + end = "" if outest_level or is_last else "," + if isinstance(v, dict): + v_str = "\n" + _format_dict(v) + if use_mapping: + k_str = f"'{k}'" if isinstance(k, str) else str(k) + attr_str = f"{k_str}: dict({v_str}" + else: + attr_str = f"{str(k)}=dict({v_str}" + attr_str = _indent(attr_str, indent) + ")" + end + elif isinstance(v, list): + attr_str = _format_list(k, v, use_mapping) + end + else: + attr_str = _format_basic_types(k, v, use_mapping) + end + + s.append(attr_str) + r += "\n".join(s) + if use_mapping: + r += "}" + return r + + cfg_dict = self._cfg_dict.to_dict() + text = _format_dict(cfg_dict, outest_level=True) + # copied from setup.cfg + yapf_style = dict( + based_on_style="pep8", + blank_line_before_nested_class_or_def=True, + split_before_expression_after_opening_paren=True, + ) + text, _ = FormatCode(text, style_config=yapf_style) + + return text + + def __repr__(self): + return f"Config (path: {self.filename}): {self._cfg_dict.__repr__()}" + + def __len__(self): + return len(self._cfg_dict) + + def __getattr__(self, name): + return getattr(self._cfg_dict, name) + + def __getitem__(self, name): + return self._cfg_dict.__getitem__(name) + + def __setattr__(self, name, value): + if isinstance(value, dict): + value = ConfigDict(value) + self._cfg_dict.__setattr__(name, value) + + def __setitem__(self, name, value): + if isinstance(value, dict): + value = ConfigDict(value) + self._cfg_dict.__setitem__(name, value) + + def __iter__(self): + return iter(self._cfg_dict) + + def __getstate__(self): + return (self._cfg_dict, self._filename, self._text) + + def __setstate__(self, state): + _cfg_dict, _filename, _text = state + super(Config, self).__setattr__("_cfg_dict", _cfg_dict) + super(Config, self).__setattr__("_filename", _filename) + super(Config, self).__setattr__("_text", _text) + + def dump(self, file=None): + cfg_dict = super(Config, self).__getattribute__("_cfg_dict").to_dict() + if self.filename.endswith(".py"): + if file is None: + return self.pretty_text + else: + with open(file, "w", encoding="utf-8") as f: + f.write(self.pretty_text) + else: + import mmcv + + if file is None: + file_format = self.filename.split(".")[-1] + return mmcv.dump(cfg_dict, file_format=file_format) + else: + mmcv.dump(cfg_dict, file) + + def merge_from_dict(self, options, allow_list_keys=True): + """Merge list into cfg_dict. + + Merge the dict parsed by MultipleKVAction into this cfg. + + Examples: + >>> options = {'models.backbone.depth': 50, + ... 'models.backbone.with_cp':True} + >>> cfg = Config(dict(models=dict(backbone=dict(type='ResNet')))) + >>> cfg.merge_from_dict(options) + >>> cfg_dict = super(Config, self).__getattribute__('_cfg_dict') + >>> assert cfg_dict == dict( + ... models=dict(backbone=dict(depth=50, with_cp=True))) + + # Merge list element + >>> cfg = Config(dict(pipeline=[ + ... dict(type='LoadImage'), dict(type='LoadAnnotations')])) + >>> options = dict(pipeline={'0': dict(type='SelfLoadImage')}) + >>> cfg.merge_from_dict(options, allow_list_keys=True) + >>> cfg_dict = super(Config, self).__getattribute__('_cfg_dict') + >>> assert cfg_dict == dict(pipeline=[ + ... dict(type='SelfLoadImage'), dict(type='LoadAnnotations')]) + + Args: + options (dict): dict of configs to merge from. + allow_list_keys (bool): If True, int string keys (e.g. '0', '1') + are allowed in ``options`` and will replace the element of the + corresponding index in the config if the config is a list. + Default: True. + """ + option_cfg_dict = {} + for full_key, v in options.items(): + d = option_cfg_dict + key_list = full_key.split(".") + for subkey in key_list[:-1]: + d.setdefault(subkey, ConfigDict()) + d = d[subkey] + subkey = key_list[-1] + d[subkey] = v + + cfg_dict = super(Config, self).__getattribute__("_cfg_dict") + super(Config, self).__setattr__( + "_cfg_dict", + Config._merge_a_into_b( + option_cfg_dict, cfg_dict, allow_list_keys=allow_list_keys + ), + ) + + +class DictAction(Action): + """ + argparse action to split an argument into KEY=VALUE form + on the first = and append to a dictionary. List options can + be passed as comma separated values, i.e 'KEY=V1,V2,V3', or with explicit + brackets, i.e. 'KEY=[V1,V2,V3]'. It also support nested brackets to build + list/tuple values. e.g. 'KEY=[(V1,V2),(V3,V4)]' + """ + + @staticmethod + def _parse_int_float_bool(val): + try: + return int(val) + except ValueError: + pass + try: + return float(val) + except ValueError: + pass + if val.lower() in ["true", "false"]: + return True if val.lower() == "true" else False + return val + + @staticmethod + def _parse_iterable(val): + """Parse iterable values in the string. + + All elements inside '()' or '[]' are treated as iterable values. + + Args: + val (str): Value string. + + Returns: + list | tuple: The expanded list or tuple from the string. + + Examples: + >>> DictAction._parse_iterable('1,2,3') + [1, 2, 3] + >>> DictAction._parse_iterable('[a, b, c]') + ['a', 'b', 'c'] + >>> DictAction._parse_iterable('[(1, 2, 3), [a, b], c]') + [(1, 2, 3), ['a', 'b'], 'c'] + """ + + def find_next_comma(string): + """Find the position of next comma in the string. + + If no ',' is found in the string, return the string length. All + chars inside '()' and '[]' are treated as one element and thus ',' + inside these brackets are ignored. + """ + assert (string.count("(") == string.count(")")) and ( + string.count("[") == string.count("]") + ), f"Imbalanced brackets exist in {string}" + end = len(string) + for idx, char in enumerate(string): + pre = string[:idx] + # The string before this ',' is balanced + if ( + (char == ",") + and (pre.count("(") == pre.count(")")) + and (pre.count("[") == pre.count("]")) + ): + end = idx + break + return end + + # Strip ' and " characters and replace whitespace. + val = val.strip("'\"").replace(" ", "") + is_tuple = False + if val.startswith("(") and val.endswith(")"): + is_tuple = True + val = val[1:-1] + elif val.startswith("[") and val.endswith("]"): + val = val[1:-1] + elif "," not in val: + # val is a single value + return DictAction._parse_int_float_bool(val) + + values = [] + while len(val) > 0: + comma_idx = find_next_comma(val) + element = DictAction._parse_iterable(val[:comma_idx]) + values.append(element) + val = val[comma_idx + 1 :] + if is_tuple: + values = tuple(values) + return values + + def __call__(self, parser, namespace, values, option_string=None): + options = {} + for kv in values: + key, val = kv.split("=", maxsplit=1) + options[key] = self._parse_iterable(val) + setattr(namespace, self.dest, options) diff --git a/pointcept/utils/env.py b/pointcept/utils/env.py new file mode 100644 index 0000000000000000000000000000000000000000..653f007dde5c4a7564e732da88dd47e7d37adf97 --- /dev/null +++ b/pointcept/utils/env.py @@ -0,0 +1,36 @@ +""" +Environment Utils + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import random +import numpy as np +import torch +import torch.backends.cudnn as cudnn + +from datetime import datetime + + +def get_random_seed(): + seed = ( + os.getpid() + + int(datetime.now().strftime("%S%f")) + + int.from_bytes(os.urandom(2), "big") + ) + return seed + + +def set_seed(seed=None): + if seed is None: + seed = get_random_seed() + random.seed(seed) + np.random.seed(seed) + torch.manual_seed(seed) + torch.cuda.manual_seed(seed) + torch.cuda.manual_seed_all(seed) + cudnn.benchmark = False + cudnn.deterministic = True + os.environ["PYTHONHASHSEED"] = str(seed) diff --git a/pointcept/utils/events.py b/pointcept/utils/events.py new file mode 100644 index 0000000000000000000000000000000000000000..718ee9191e5cce9343383baa1aae99f22c3d0734 --- /dev/null +++ b/pointcept/utils/events.py @@ -0,0 +1,612 @@ +""" +Events Utils + +Modified from Detectron2 + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import datetime +import json +import logging +import os +import time +import torch +import numpy as np +import traceback +import sys + +from typing import List, Optional, Tuple +from collections import defaultdict +from contextlib import contextmanager + +__all__ = [ + "get_event_storage", + "JSONWriter", + "TensorboardXWriter", + "CommonMetricPrinter", + "EventStorage", + "ExceptionWriter", +] + +_CURRENT_STORAGE_STACK = [] + + +def get_event_storage(): + """ + Returns: + The :class:`EventStorage` object that's currently being used. + Throws an error if no :class:`EventStorage` is currently enabled. + """ + assert len( + _CURRENT_STORAGE_STACK + ), "get_event_storage() has to be called inside a 'with EventStorage(...)' context!" + return _CURRENT_STORAGE_STACK[-1] + + +class EventWriter: + """ + Base class for writers that obtain events from :class:`EventStorage` and process them. + """ + + def write(self): + raise NotImplementedError + + def close(self): + pass + + +class JSONWriter(EventWriter): + """ + Write scalars to a json file. + It saves scalars as one json per line (instead of a big json) for easy parsing. + Examples parsing such a json file: + :: + $ cat metrics.json | jq -s '.[0:2]' + [ + { + "data_time": 0.008433341979980469, + "iteration": 19, + "loss": 1.9228371381759644, + "loss_box_reg": 0.050025828182697296, + "loss_classifier": 0.5316952466964722, + "loss_mask": 0.7236229181289673, + "loss_rpn_box": 0.0856662318110466, + "loss_rpn_cls": 0.48198649287223816, + "lr": 0.007173333333333333, + "time": 0.25401854515075684 + }, + { + "data_time": 0.007216215133666992, + "iteration": 39, + "loss": 1.282649278640747, + "loss_box_reg": 0.06222952902317047, + "loss_classifier": 0.30682939291000366, + "loss_mask": 0.6970193982124329, + "loss_rpn_box": 0.038663312792778015, + "loss_rpn_cls": 0.1471673548221588, + "lr": 0.007706666666666667, + "time": 0.2490077018737793 + } + ] + $ cat metrics.json | jq '.loss_mask' + 0.7126231789588928 + 0.689423680305481 + 0.6776131987571716 + ... + """ + + def __init__(self, json_file, window_size=20): + """ + Args: + json_file (str): path to the json file. New data will be appended if the file exists. + window_size (int): the window size of median smoothing for the scalars whose + `smoothing_hint` are True. + """ + self._file_handle = open(json_file, "a") + self._window_size = window_size + self._last_write = -1 + + def write(self): + storage = get_event_storage() + to_save = defaultdict(dict) + + for k, (v, iter) in storage.latest_with_smoothing_hint( + self._window_size + ).items(): + # keep scalars that have not been written + if iter <= self._last_write: + continue + to_save[iter][k] = v + if len(to_save): + all_iters = sorted(to_save.keys()) + self._last_write = max(all_iters) + + for itr, scalars_per_iter in to_save.items(): + scalars_per_iter["iteration"] = itr + self._file_handle.write(json.dumps(scalars_per_iter, sort_keys=True) + "\n") + self._file_handle.flush() + try: + os.fsync(self._file_handle.fileno()) + except AttributeError: + pass + + def close(self): + self._file_handle.close() + + +class TensorboardXWriter(EventWriter): + """ + Write all scalars to a tensorboard file. + """ + + def __init__(self, log_dir: str, window_size: int = 20, **kwargs): + """ + Args: + log_dir (str): the directory to save the output events + window_size (int): the scalars will be median-smoothed by this window size + kwargs: other arguments passed to `torch.utils.tensorboard.SummaryWriter(...)` + """ + self._window_size = window_size + from torch.utils.tensorboard import SummaryWriter + + self._writer = SummaryWriter(log_dir, **kwargs) + self._last_write = -1 + + def write(self): + storage = get_event_storage() + new_last_write = self._last_write + for k, (v, iter) in storage.latest_with_smoothing_hint( + self._window_size + ).items(): + if iter > self._last_write: + self._writer.add_scalar(k, v, iter) + new_last_write = max(new_last_write, iter) + self._last_write = new_last_write + + # storage.put_{image,histogram} is only meant to be used by + # tensorboard writer. So we access its internal fields directly from here. + if len(storage._vis_data) >= 1: + for img_name, img, step_num in storage._vis_data: + self._writer.add_image(img_name, img, step_num) + # Storage stores all image data and rely on this writer to clear them. + # As a result it assumes only one writer will use its image data. + # An alternative design is to let storage store limited recent + # data (e.g. only the most recent image) that all writers can access. + # In that case a writer may not see all image data if its period is long. + storage.clear_images() + + if len(storage._histograms) >= 1: + for params in storage._histograms: + self._writer.add_histogram_raw(**params) + storage.clear_histograms() + + def close(self): + if hasattr(self, "_writer"): # doesn't exist when the code fails at import + self._writer.close() + + +class CommonMetricPrinter(EventWriter): + """ + Print **common** metrics to the terminal, including + iteration time, ETA, memory, all losses, and the learning rate. + It also applies smoothing using a window of 20 elements. + It's meant to print common metrics in common ways. + To print something in more customized ways, please implement a similar printer by yourself. + """ + + def __init__(self, max_iter: Optional[int] = None, window_size: int = 20): + """ + Args: + max_iter: the maximum number of iterations to train. + Used to compute ETA. If not given, ETA will not be printed. + window_size (int): the losses will be median-smoothed by this window size + """ + self.logger = logging.getLogger(__name__) + self._max_iter = max_iter + self._window_size = window_size + self._last_write = ( + None # (step, time) of last call to write(). Used to compute ETA + ) + + def _get_eta(self, storage) -> Optional[str]: + if self._max_iter is None: + return "" + iteration = storage.iter + try: + eta_seconds = storage.history("time").median(1000) * ( + self._max_iter - iteration - 1 + ) + storage.put_scalar("eta_seconds", eta_seconds, smoothing_hint=False) + return str(datetime.timedelta(seconds=int(eta_seconds))) + except KeyError: + # estimate eta on our own - more noisy + eta_string = None + if self._last_write is not None: + estimate_iter_time = (time.perf_counter() - self._last_write[1]) / ( + iteration - self._last_write[0] + ) + eta_seconds = estimate_iter_time * (self._max_iter - iteration - 1) + eta_string = str(datetime.timedelta(seconds=int(eta_seconds))) + self._last_write = (iteration, time.perf_counter()) + return eta_string + + def write(self): + storage = get_event_storage() + iteration = storage.iter + if iteration == self._max_iter: + # This hook only reports training progress (loss, ETA, etc) but not other data, + # therefore do not write anything after training succeeds, even if this method + # is called. + return + + try: + data_time = storage.history("data_time").avg(20) + except KeyError: + # they may not exist in the first few iterations (due to warmup) + # or when SimpleTrainer is not used + data_time = None + try: + iter_time = storage.history("time").global_avg() + except KeyError: + iter_time = None + try: + lr = "{:.5g}".format(storage.history("lr").latest()) + except KeyError: + lr = "N/A" + + eta_string = self._get_eta(storage) + + if torch.cuda.is_available(): + max_mem_mb = torch.cuda.max_memory_allocated() / 1024.0 / 1024.0 + else: + max_mem_mb = None + + # NOTE: max_mem is parsed by grep in "dev/parse_results.sh" + self.logger.info( + " {eta}iter: {iter} {losses} {time}{data_time}lr: {lr} {memory}".format( + eta=f"eta: {eta_string} " if eta_string else "", + iter=iteration, + losses=" ".join( + [ + "{}: {:.4g}".format(k, v.median(self._window_size)) + for k, v in storage.histories().items() + if "loss" in k + ] + ), + time=( + "time: {:.4f} ".format(iter_time) if iter_time is not None else "" + ), + data_time=( + "data_time: {:.4f} ".format(data_time) + if data_time is not None + else "" + ), + lr=lr, + memory=( + "max_mem: {:.0f}M".format(max_mem_mb) + if max_mem_mb is not None + else "" + ), + ) + ) + + +class EventStorage: + """ + The user-facing class that provides metric storage functionalities. + In the future we may add support for storing / logging other types of data if needed. + """ + + def __init__(self, start_iter=0): + """ + Args: + start_iter (int): the iteration number to start with + """ + self._history = defaultdict(AverageMeter) + self._smoothing_hints = {} + self._latest_scalars = {} + self._iter = start_iter + self._current_prefix = "" + self._vis_data = [] + self._histograms = [] + + # def put_image(self, img_name, img_tensor): + # """ + # Add an `img_tensor` associated with `img_name`, to be shown on + # tensorboard. + # Args: + # img_name (str): The name of the image to put into tensorboard. + # img_tensor (torch.Tensor or numpy.array): An `uint8` or `float` + # Tensor of shape `[channel, height, width]` where `channel` is + # 3. The image format should be RGB. The elements in img_tensor + # can either have values in [0, 1] (float32) or [0, 255] (uint8). + # The `img_tensor` will be visualized in tensorboard. + # """ + # self._vis_data.append((img_name, img_tensor, self._iter)) + + def put_scalar(self, name, value, n=1, smoothing_hint=False): + """ + Add a scalar `value` to the `HistoryBuffer` associated with `name`. + Args: + smoothing_hint (bool): a 'hint' on whether this scalar is noisy and should be + smoothed when logged. The hint will be accessible through + :meth:`EventStorage.smoothing_hints`. A writer may ignore the hint + and apply custom smoothing rule. + It defaults to True because most scalars we save need to be smoothed to + provide any useful signal. + """ + name = self._current_prefix + name + history = self._history[name] + history.update(value, n) + self._latest_scalars[name] = (value, self._iter) + + existing_hint = self._smoothing_hints.get(name) + if existing_hint is not None: + assert ( + existing_hint == smoothing_hint + ), "Scalar {} was put with a different smoothing_hint!".format(name) + else: + self._smoothing_hints[name] = smoothing_hint + + # def put_scalars(self, *, smoothing_hint=True, **kwargs): + # """ + # Put multiple scalars from keyword arguments. + # Examples: + # storage.put_scalars(loss=my_loss, accuracy=my_accuracy, smoothing_hint=True) + # """ + # for k, v in kwargs.items(): + # self.put_scalar(k, v, smoothing_hint=smoothing_hint) + # + # def put_histogram(self, hist_name, hist_tensor, bins=1000): + # """ + # Create a histogram from a tensor. + # Args: + # hist_name (str): The name of the histogram to put into tensorboard. + # hist_tensor (torch.Tensor): A Tensor of arbitrary shape to be converted + # into a histogram. + # bins (int): Number of histogram bins. + # """ + # ht_min, ht_max = hist_tensor.min().item(), hist_tensor.max().item() + # + # # Create a histogram with PyTorch + # hist_counts = torch.histc(hist_tensor, bins=bins) + # hist_edges = torch.linspace(start=ht_min, end=ht_max, steps=bins + 1, dtype=torch.float32) + # + # # Parameter for the add_histogram_raw function of SummaryWriter + # hist_params = dict( + # tag=hist_name, + # min=ht_min, + # max=ht_max, + # num=len(hist_tensor), + # sum=float(hist_tensor.sum()), + # sum_squares=float(torch.sum(hist_tensor**2)), + # bucket_limits=hist_edges[1:].tolist(), + # bucket_counts=hist_counts.tolist(), + # global_step=self._iter, + # ) + # self._histograms.append(hist_params) + + def history(self, name): + """ + Returns: + AverageMeter: the history for name + """ + ret = self._history.get(name, None) + if ret is None: + raise KeyError("No history metric available for {}!".format(name)) + return ret + + def histories(self): + """ + Returns: + dict[name -> HistoryBuffer]: the HistoryBuffer for all scalars + """ + return self._history + + def latest(self): + """ + Returns: + dict[str -> (float, int)]: mapping from the name of each scalar to the most + recent value and the iteration number its added. + """ + return self._latest_scalars + + def latest_with_smoothing_hint(self, window_size=20): + """ + Similar to :meth:`latest`, but the returned values + are either the un-smoothed original latest value, + or a median of the given window_size, + depend on whether the smoothing_hint is True. + This provides a default behavior that other writers can use. + """ + result = {} + for k, (v, itr) in self._latest_scalars.items(): + result[k] = ( + self._history[k].median(window_size) if self._smoothing_hints[k] else v, + itr, + ) + return result + + def smoothing_hints(self): + """ + Returns: + dict[name -> bool]: the user-provided hint on whether the scalar + is noisy and needs smoothing. + """ + return self._smoothing_hints + + def step(self): + """ + User should either: (1) Call this function to increment storage.iter when needed. Or + (2) Set `storage.iter` to the correct iteration number before each iteration. + The storage will then be able to associate the new data with an iteration number. + """ + self._iter += 1 + + @property + def iter(self): + """ + Returns: + int: The current iteration number. When used together with a trainer, + this is ensured to be the same as trainer.iter. + """ + return self._iter + + @iter.setter + def iter(self, val): + self._iter = int(val) + + @property + def iteration(self): + # for backward compatibility + return self._iter + + def __enter__(self): + _CURRENT_STORAGE_STACK.append(self) + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + assert _CURRENT_STORAGE_STACK[-1] == self + _CURRENT_STORAGE_STACK.pop() + + @contextmanager + def name_scope(self, name): + """ + Yields: + A context within which all the events added to this storage + will be prefixed by the name scope. + """ + old_prefix = self._current_prefix + self._current_prefix = name.rstrip("/") + "/" + yield + self._current_prefix = old_prefix + + def clear_images(self): + """ + Delete all the stored images for visualization. This should be called + after images are written to tensorboard. + """ + self._vis_data = [] + + def clear_histograms(self): + """ + Delete all the stored histograms for visualization. + This should be called after histograms are written to tensorboard. + """ + self._histograms = [] + + def reset_history(self, name): + ret = self._history.get(name, None) + if ret is None: + raise KeyError("No history metric available for {}!".format(name)) + ret.reset() + + def reset_histories(self): + for name in self._history.keys(): + self._history[name].reset() + + +class AverageMeter: + """Computes and stores the average and current value""" + + def __init__(self): + self.val = 0 + self.avg = 0 + self.total = 0 + self.count = 0 + + def reset(self): + self.val = 0 + self.avg = 0 + self.total = 0 + self.count = 0 + + def update(self, val, n=1): + self.val = val + self.total += val * n + self.count += n + self.avg = self.total / self.count + + +class HistoryBuffer: + """ + Track a series of scalar values and provide access to smoothed values over a + window or the global average of the series. + """ + + def __init__(self, max_length: int = 1000000) -> None: + """ + Args: + max_length: maximal number of values that can be stored in the + buffer. When the capacity of the buffer is exhausted, old + values will be removed. + """ + self._max_length: int = max_length + self._data: List[Tuple[float, float]] = [] # (value, iteration) pairs + self._count: int = 0 + self._global_avg: float = 0 + + def update(self, value: float, iteration: Optional[float] = None) -> None: + """ + Add a new scalar value produced at certain iteration. If the length + of the buffer exceeds self._max_length, the oldest element will be + removed from the buffer. + """ + if iteration is None: + iteration = self._count + if len(self._data) == self._max_length: + self._data.pop(0) + self._data.append((value, iteration)) + + self._count += 1 + self._global_avg += (value - self._global_avg) / self._count + + def latest(self) -> float: + """ + Return the latest scalar value added to the buffer. + """ + return self._data[-1][0] + + def median(self, window_size: int) -> float: + """ + Return the median of the latest `window_size` values in the buffer. + """ + return np.median([x[0] for x in self._data[-window_size:]]) + + def avg(self, window_size: int) -> float: + """ + Return the mean of the latest `window_size` values in the buffer. + """ + return np.mean([x[0] for x in self._data[-window_size:]]) + + def global_avg(self) -> float: + """ + Return the mean of all the elements in the buffer. Note that this + includes those getting removed due to limited buffer storage. + """ + return self._global_avg + + def values(self) -> List[Tuple[float, float]]: + """ + Returns: + list[(number, iteration)]: content of the current buffer. + """ + return self._data + + +class ExceptionWriter: + + def __init__(self): + self.logger = logging.getLogger(__name__) + + def __enter__(self): + pass + + def __exit__(self, exc_type, exc_val, exc_tb): + if exc_type: + tb = traceback.format_exception(exc_type, exc_val, exc_tb) + formatted_tb_str = "".join(tb) + self.logger.error(formatted_tb_str) + sys.exit(1) # This prevents double logging the error to the console diff --git a/pointcept/utils/logger.py b/pointcept/utils/logger.py new file mode 100644 index 0000000000000000000000000000000000000000..ddaf2c5a765c9f1325737c3cbc73e1169f13cdd4 --- /dev/null +++ b/pointcept/utils/logger.py @@ -0,0 +1,172 @@ +""" +Logger Utils + +Modified from mmcv + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import logging +import torch +import torch.distributed as dist + +from termcolor import colored + +logger_initialized = {} +root_status = 0 + + +class _ColorfulFormatter(logging.Formatter): + def __init__(self, *args, **kwargs): + self._root_name = kwargs.pop("root_name") + "." + super(_ColorfulFormatter, self).__init__(*args, **kwargs) + + def formatMessage(self, record): + log = super(_ColorfulFormatter, self).formatMessage(record) + if record.levelno == logging.WARNING: + prefix = colored("WARNING", "red", attrs=["blink"]) + elif record.levelno == logging.ERROR or record.levelno == logging.CRITICAL: + prefix = colored("ERROR", "red", attrs=["blink", "underline"]) + else: + return log + return prefix + " " + log + + +def get_logger(name, log_file=None, log_level=logging.INFO, file_mode="a", color=False): + """Initialize and get a logger by name. + + If the logger has not been initialized, this method will initialize the + logger by adding one or two handlers, otherwise the initialized logger will + be directly returned. During initialization, a StreamHandler will always be + added. If `log_file` is specified and the process rank is 0, a FileHandler + will also be added. + + Args: + name (str): Logger name. + log_file (str | None): The log filename. If specified, a FileHandler + will be added to the logger. + log_level (int): The logger level. Note that only the process of + rank 0 is affected, and other processes will set the level to + "Error" thus be silent most of the time. + file_mode (str): The file mode used in opening log file. + Defaults to 'a'. + color (bool): Colorful log output. Defaults to True + + Returns: + logging.Logger: The expected logger. + """ + logger = logging.getLogger(name) + + if name in logger_initialized: + return logger + # handle hierarchical names + # e.g., logger "a" is initialized, then logger "a.b" will skip the + # initialization since it is a child of "a". + for logger_name in logger_initialized: + if name.startswith(logger_name): + return logger + + logger.propagate = False + + stream_handler = logging.StreamHandler() + handlers = [stream_handler] + + if dist.is_available() and dist.is_initialized(): + rank = dist.get_rank() + else: + rank = 0 + + # only rank 0 will add a FileHandler + if rank == 0 and log_file is not None: + # Here, the default behaviour of the official logger is 'a'. Thus, we + # provide an interface to change the file mode to the default + # behaviour. + file_handler = logging.FileHandler(log_file, file_mode) + handlers.append(file_handler) + + plain_formatter = logging.Formatter( + "[%(asctime)s %(levelname)s %(filename)s line %(lineno)d %(process)d] %(message)s" + ) + if color: + formatter = _ColorfulFormatter( + colored("[%(asctime)s %(name)s]: ", "green") + "%(message)s", + datefmt="%m/%d %H:%M:%S", + root_name=name, + ) + else: + formatter = plain_formatter + for handler in handlers: + handler.setFormatter(formatter) + handler.setLevel(log_level) + logger.addHandler(handler) + + if rank == 0: + logger.setLevel(log_level) + else: + logger.setLevel(logging.ERROR) + + logger_initialized[name] = True + + return logger + + +def print_log(msg, logger=None, level=logging.INFO): + """Print a log message. + + Args: + msg (str): The message to be logged. + logger (logging.Logger | str | None): The logger to be used. + Some special loggers are: + - "silent": no message will be printed. + - other str: the logger obtained with `get_root_logger(logger)`. + - None: The `print()` method will be used to print log messages. + level (int): Logging level. Only available when `logger` is a Logger + object or "root". + """ + if logger is None: + print(msg) + elif isinstance(logger, logging.Logger): + logger.log(level, msg) + elif logger == "silent": + pass + elif isinstance(logger, str): + _logger = get_logger(logger) + _logger.log(level, msg) + else: + raise TypeError( + "logger should be either a logging.Logger object, str, " + f'"silent" or None, but got {type(logger)}' + ) + + +def get_root_logger(log_file=None, log_level=logging.INFO, file_mode="a"): + """Get the root logger. + + The logger will be initialized if it has not been initialized. By default a + StreamHandler will be added. If `log_file` is specified, a FileHandler will + also be added. The name of the root logger is the top-level package name. + + Args: + log_file (str | None): The log filename. If specified, a FileHandler + will be added to the root logger. + log_level (int): The root logger level. Note that only the process of + rank 0 is affected, while other processes will set the level to + "Error" and be silent most of the time. + file_mode (str): File Mode of logger. (w or a) + + Returns: + logging.Logger: The root logger. + """ + logger = get_logger( + name="pointcept", log_file=log_file, log_level=log_level, file_mode=file_mode + ) + return logger + + +def _log_api_usage(identifier: str): + """ + Internal function used to log the usage of different detectron2 components + inside facebook's infra. + """ + torch._C._log_api_usage_once("pointcept." + identifier) diff --git a/pointcept/utils/misc.py b/pointcept/utils/misc.py new file mode 100644 index 0000000000000000000000000000000000000000..ac44b5dd937157595f37c7b469f43a998d19c87e --- /dev/null +++ b/pointcept/utils/misc.py @@ -0,0 +1,168 @@ +""" +Misc + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os +import warnings +from collections import abc +import numpy as np +import torch +from importlib import import_module + + +class AverageMeter(object): + """Computes and stores the average and current value""" + + def __init__(self): + self.val = 0 + self.avg = 0 + self.sum = 0 + self.count = 0 + + def reset(self): + self.val = 0 + self.avg = 0 + self.sum = 0 + self.count = 0 + + def update(self, val, n=1): + self.val = val + self.sum += val * n + self.count += n + self.avg = self.sum / self.count + + +def intersection_and_union(output, target, K, ignore_index=-1): + # 'K' classes, output and target sizes are N or N * L or N * H * W, each value in range 0 to K - 1. + assert output.ndim in [1, 2, 3] + assert output.shape == target.shape + output = output.reshape(output.size).copy() + target = target.reshape(target.size) + + # Keep only points whose label is not ignore_index. + mask = target != ignore_index + output, target = output[mask], target[mask] + + intersection = output[np.where(output == target)[0]] + area_intersection, _ = np.histogram(intersection, bins=np.arange(K + 1)) + area_output, _ = np.histogram(output, bins=np.arange(K + 1)) + area_target, _ = np.histogram(target, bins=np.arange(K + 1)) + area_union = area_output + area_target - area_intersection + return area_intersection, area_union, area_target + + +def intersection_and_union_gpu(output, target, k, ignore_index=-1): + # 'K' classes, output and target sizes are N or N * L or N * H * W, each value in range 0 to K - 1. + assert output.dim() in [1, 2, 3] + assert output.shape == target.shape + output = output.view(-1) + target = target.view(-1) + output[target == ignore_index] = ignore_index + intersection = output[output == target] + area_intersection = torch.histc(intersection, bins=k, min=0, max=k - 1) + area_output = torch.histc(output, bins=k, min=0, max=k - 1) + area_target = torch.histc(target, bins=k, min=0, max=k - 1) + area_union = area_output + area_target - area_intersection + return area_intersection, area_union, area_target + + +def make_dirs(dir_name): + if not os.path.exists(dir_name): + os.makedirs(dir_name, exist_ok=True) + + +def find_free_port(): + import socket + + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + # Binding to port 0 will cause the OS to find an available port for us + sock.bind(("", 0)) + port = sock.getsockname()[1] + sock.close() + # NOTE: there is still a chance the port could be taken by other processes. + return port + + +def is_seq_of(seq, expected_type, seq_type=None): + """Check whether it is a sequence of some type. + + Args: + seq (Sequence): The sequence to be checked. + expected_type (type): Expected type of sequence items. + seq_type (type, optional): Expected sequence type. + + Returns: + bool: Whether the sequence is valid. + """ + if seq_type is None: + exp_seq_type = abc.Sequence + else: + assert isinstance(seq_type, type) + exp_seq_type = seq_type + if not isinstance(seq, exp_seq_type): + return False + for item in seq: + if not isinstance(item, expected_type): + return False + return True + + +def is_str(x): + """Whether the input is an string instance. + + Note: This method is deprecated since python 2 is no longer supported. + """ + return isinstance(x, str) + + +def import_modules_from_strings(imports, allow_failed_imports=False): + """Import modules from the given list of strings. + + Args: + imports (list | str | None): The given module names to be imported. + allow_failed_imports (bool): If True, the failed imports will return + None. Otherwise, an ImportError is raise. Default: False. + + Returns: + list[module] | module | None: The imported modules. + + Examples: + >>> osp, sys = import_modules_from_strings( + ... ['os.path', 'sys']) + >>> import os.path as osp_ + >>> import sys as sys_ + >>> assert osp == osp_ + >>> assert sys == sys_ + """ + if not imports: + return + single_import = False + if isinstance(imports, str): + single_import = True + imports = [imports] + if not isinstance(imports, list): + raise TypeError(f"custom_imports must be a list but got type {type(imports)}") + imported = [] + for imp in imports: + if not isinstance(imp, str): + raise TypeError(f"{imp} is of type {type(imp)} and cannot be imported.") + try: + imported_tmp = import_module(imp) + except ImportError: + if allow_failed_imports: + warnings.warn(f"{imp} failed to import and is ignored.", UserWarning) + imported_tmp = None + else: + raise ImportError + imported.append(imported_tmp) + if single_import: + imported = imported[0] + return imported + + +class DummyClass: + def __init__(self): + pass diff --git a/pointcept/utils/muon_kimi.py b/pointcept/utils/muon_kimi.py new file mode 100644 index 0000000000000000000000000000000000000000..383c9111abb34f99f5db7b634e35c1f68265dcbe --- /dev/null +++ b/pointcept/utils/muon_kimi.py @@ -0,0 +1,197 @@ +import math +import torch + + +# This code snippet is a modified version adapted from the following GitHub repository: +# https://github.com/KellerJordan/Muon/blob/master/muon.py +@torch.compile +def zeropower_via_newtonschulz5(G, steps): + """ + Newton-Schulz iteration to compute the zeroth power / orthogonalization of G. We opt to use a + quintic iteration whose coefficients are selected to maximize the slope at zero. For the purpose + of minimizing steps, it turns out to be empirically effective to keep increasing the slope at + zero even beyond the point where the iteration no longer converges all the way to one everywhere + on the interval. This iteration therefore does not produce UV^T but rather something like US'V^T + where S' is diagonal with S_{ii}' ~ Uniform(0.5, 1.5), which turns out not to hurt model + performance at all relative to UV^T, where USV^T = G is the SVD. + """ + assert len(G.shape) == 2 + a, b, c = (3.4445, -4.7750, 2.0315) + X = G.bfloat16() + if G.size(0) > G.size(1): + X = X.T + # Ensure spectral norm is at most 1 + X = X / (X.norm() + 1e-7) + # Perform the NS iterations + for _ in range(steps): + A = X @ X.T + B = ( + b * A + c * A @ A + ) # adapted from suggestion by @jxbz, @leloykun, and @YouJiacheng + X = a * X + B @ X + + if G.size(0) > G.size(1): + X = X.T + return X + + +class MuonKIMI(torch.optim.Optimizer): + """ + Muon - MomentUm Orthogonalized by Newton-schulz + + Muon internally runs standard SGD-momentum, and then performs an orthogonalization post- + processing step, in which each 2D parameter's update is replaced with the nearest orthogonal + matrix. To efficiently orthogonalize each update, we use a Newton-Schulz iteration, which has + the advantage that it can be stably run in bfloat16 on the GPU. + + Some warnings: + - We believe this optimizer is unlikely to work well for training with small batch size. + - We believe it may not work well for finetuning pretrained models, but we haven't tested this. + + Arguments: + muon_params: The parameters to be optimized by Muon. + lr: The learning rate. The updates will have spectral norm of `lr`. (0.02 is a good default) + momentum: The momentum used by the internal SGD. (0.95 is a good default) + nesterov: Whether to use Nesterov-style momentum in the internal SGD. (recommended) + ns_steps: The number of Newton-Schulz iterations to run. (6 is probably always enough) + {0, 1}-D or are detected as being the embed or lm_head will be optimized by AdamW as well. + adamw_lr: The learning rate for the internal AdamW. + adamw_betas: The betas for the internal AdamW. + adamw_eps: The epsilon for the internal AdamW. + adamw_wd: The weight decay for the internal AdamW. + """ + + def __init__( + self, + params, + lr=1e-3, + weight_decay=0.01, + momentum=0.95, + nesterov=True, + ns_steps=5, + adamw_betas=(0.9, 0.95), + adamw_eps=1e-8, + ): + wd = weight_decay + + defaults = dict( + lr=lr, + wd=wd, + momentum=momentum, + nesterov=nesterov, + ns_steps=ns_steps, + adamw_betas=adamw_betas, + adamw_eps=adamw_eps, + ) + + super().__init__(params, defaults) + # # Sort parameters into those for which we will use Muon, and those for which we will not + + for group in self.param_groups: + force_adamw = group.get("force_adamw", False) + + for p in group["params"]: + if p.requires_grad: + if force_adamw or p.ndim < 2: + self.state[p]["use_muon"] = False + else: + self.state[p]["use_muon"] = True + else: + self.state[p]["use_muon"] = False + + def adjust_lr_for_muon(self, lr, param_shape): + A, B = param_shape[:2] + # We adjust the learning rate and weight decay based on the size of the parameter matrix + # as described in the paper + adjusted_ratio = 0.2 * math.sqrt(max(A, B)) + adjusted_lr = lr * adjusted_ratio + return adjusted_lr + + def step(self, closure=None): + """Perform a single optimization step. + + Args: + closure (Callable, optional): A closure that reevaluates the model + and returns the loss. + """ + loss = None + if closure is not None: + with torch.enable_grad(): + loss = closure() + + for group in self.param_groups: + + ############################ + # Muon # + ############################ + + params = [p for p in group["params"] if self.state[p]["use_muon"]] + lr = group["lr"] + wd = group["wd"] + momentum = group["momentum"] + + # Distribute parameters + for p in params: + g = p.grad + if g is None: + continue + if g.ndim > 2: + g = g.view(g.size(0), -1) + assert g is not None + + # calc update + state = self.state[p] + if "momentum_buffer" not in state: + state["momentum_buffer"] = torch.zeros_like(g) + buf = state["momentum_buffer"] + buf.mul_(momentum).add_(g) + if group["nesterov"]: + g = g.add(buf, alpha=momentum) + else: + g = buf + u = zeropower_via_newtonschulz5(g, steps=group["ns_steps"]) + + # scale update + adjusted_lr = self.adjust_lr_for_muon(lr, g.shape) + + # apply weight decay + p.data.mul_(1 - lr * wd) + + # apply update + p.data.add_(u.reshape(p.shape), alpha=-adjusted_lr) + + ############################ + # AdamW backup # + ############################ + + params = [p for p in group["params"] if not self.state[p]["use_muon"]] + lr = group["lr"] + beta1, beta2 = group["adamw_betas"] + eps = group["adamw_eps"] + weight_decay = group["wd"] + + for p in params: + g = p.grad + if g is None: + continue + state = self.state[p] + if "step" not in state: + state["step"] = 0 + state["moment1"] = torch.zeros_like(g) + state["moment2"] = torch.zeros_like(g) + state["step"] += 1 + step = state["step"] + buf1 = state["moment1"] + buf2 = state["moment2"] + buf1.lerp_(g, 1 - beta1) + buf2.lerp_(g.square(), 1 - beta2) + + g = buf1 / (eps + buf2.sqrt()) + + bias_correction1 = 1 - beta1**step + bias_correction2 = 1 - beta2**step + scale = bias_correction1 / bias_correction2**0.5 + p.data.mul_(1 - lr * weight_decay) + p.data.add_(g, alpha=-lr / scale) + + return loss diff --git a/pointcept/utils/optimizer.py b/pointcept/utils/optimizer.py new file mode 100644 index 0000000000000000000000000000000000000000..a2cedea937a44d188d311ebf1df34c56dc217052 --- /dev/null +++ b/pointcept/utils/optimizer.py @@ -0,0 +1,63 @@ +""" +Optimizer + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com), Yujia Zhang (yujia.zhang.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import copy +import torch +from pointcept.utils.logger import get_root_logger +from pointcept.utils.registry import Registry +from .muon_kimi import * + +OPTIMIZERS = Registry("optimizers") + + +OPTIMIZERS.register_module(module=torch.optim.SGD, name="SGD") +OPTIMIZERS.register_module(module=torch.optim.Adam, name="Adam") +OPTIMIZERS.register_module(module=torch.optim.AdamW, name="AdamW") +OPTIMIZERS.register_module(module=MuonKIMI, name="Muon_KIMI") + + +def build_optimizer(cfg, model, param_dicts=None): + cfg = copy.deepcopy(cfg) + if param_dicts is None: + cfg.params = model.parameters() + else: + cfg.params = [dict(names=[], params=[], lr=cfg.lr)] + for i in range(len(param_dicts)): + param_group = dict(names=[], params=[]) + if "lr" in param_dicts[i].keys(): + param_group["lr"] = param_dicts[i].lr + if "momentum" in param_dicts[i].keys(): + param_group["momentum"] = param_dicts[i].momentum + if "weight_decay" in param_dicts[i].keys(): + param_group["weight_decay"] = param_dicts[i].weight_decay + if "force_adamw" in param_dicts[i].keys(): + param_group["force_adamw"] = param_dicts[i].force_adamw + cfg.params.append(param_group) + + for n, p in model.named_parameters(): + if not p.requires_grad: + continue + flag = False + for i in range(len(param_dicts)): + if param_dicts[i].keyword in n: + cfg.params[i + 1]["names"].append(n) + cfg.params[i + 1]["params"].append(p) + flag = True + break + if not flag: + cfg.params[0]["names"].append(n) + cfg.params[0]["params"].append(p) + + logger = get_root_logger() + for i in range(len(cfg.params)): + param_names = cfg.params[i].pop("names") + message = "" + for key in cfg.params[i].keys(): + if key != "params": + message += f" {key}: {cfg.params[i][key]};" + logger.info(f"Params Group {i+1} -{message} Params: {param_names}.") + return OPTIMIZERS.build(cfg=cfg) diff --git a/pointcept/utils/path.py b/pointcept/utils/path.py new file mode 100644 index 0000000000000000000000000000000000000000..ce98fa5fd0dfbf6e1d61e833ecc35fea4ab2782b --- /dev/null +++ b/pointcept/utils/path.py @@ -0,0 +1,103 @@ +# Copyright (c) OpenMMLab. All rights reserved. +import os +import os.path as osp +from pathlib import Path + +from .misc import is_str + + +def is_filepath(x): + return is_str(x) or isinstance(x, Path) + + +def fopen(filepath, *args, **kwargs): + if is_str(filepath): + return open(filepath, *args, **kwargs) + elif isinstance(filepath, Path): + return filepath.open(*args, **kwargs) + raise ValueError("`filepath` should be a string or a Path") + + +def check_file_exist(filename, msg_tmpl='file "{}" does not exist'): + if not osp.isfile(filename): + raise FileNotFoundError(msg_tmpl.format(filename)) + + +def mkdir_or_exist(dir_name, mode=0o777): + if dir_name == "": + return + dir_name = osp.expanduser(dir_name) + os.makedirs(dir_name, mode=mode, exist_ok=True) + + +def symlink(src, dst, overwrite=True, **kwargs): + if os.path.lexists(dst) and overwrite: + os.remove(dst) + os.symlink(src, dst, **kwargs) + + +def scandir(dir_path, suffix=None, recursive=False, case_sensitive=True): + """Scan a directory to find the interested files. + + Args: + dir_path (str | obj:`Path`): Path of the directory. + suffix (str | tuple(str), optional): File suffix that we are + interested in. Default: None. + recursive (bool, optional): If set to True, recursively scan the + directory. Default: False. + case_sensitive (bool, optional) : If set to False, ignore the case of + suffix. Default: True. + + Returns: + A generator for all the interested files with relative paths. + """ + if isinstance(dir_path, (str, Path)): + dir_path = str(dir_path) + else: + raise TypeError('"dir_path" must be a string or Path object') + + if (suffix is not None) and not isinstance(suffix, (str, tuple)): + raise TypeError('"suffix" must be a string or tuple of strings') + + if suffix is not None and not case_sensitive: + suffix = ( + suffix.lower() + if isinstance(suffix, str) + else tuple(item.lower() for item in suffix) + ) + + root = dir_path + + def _scandir(dir_path, suffix, recursive, case_sensitive): + for entry in os.scandir(dir_path): + if not entry.name.startswith(".") and entry.is_file(): + rel_path = osp.relpath(entry.path, root) + _rel_path = rel_path if case_sensitive else rel_path.lower() + if suffix is None or _rel_path.endswith(suffix): + yield rel_path + elif recursive and os.path.isdir(entry.path): + # scan recursively if entry.path is a directory + yield from _scandir(entry.path, suffix, recursive, case_sensitive) + + return _scandir(dir_path, suffix, recursive, case_sensitive) + + +def find_vcs_root(path, markers=(".git",)): + """Finds the root directory (including itself) of specified markers. + + Args: + path (str): Path of directory or file. + markers (list[str], optional): List of file or directory names. + + Returns: + The directory contained one of the markers or None if not found. + """ + if osp.isfile(path): + path = osp.dirname(path) + + prev, cur = None, osp.abspath(osp.expanduser(path)) + while cur != prev: + if any(osp.exists(osp.join(cur, marker)) for marker in markers): + return cur + prev, cur = cur, osp.split(cur)[0] + return None diff --git a/pointcept/utils/registry.py b/pointcept/utils/registry.py new file mode 100644 index 0000000000000000000000000000000000000000..7ac308a87d38ff61da14d6b4d5c73b4c68c15a58 --- /dev/null +++ b/pointcept/utils/registry.py @@ -0,0 +1,316 @@ +# Copyright (c) OpenMMLab. All rights reserved. +import inspect +import warnings +from functools import partial + +from .misc import is_seq_of + + +def build_from_cfg(cfg, registry, default_args=None): + """Build a module from configs dict. + + Args: + cfg (dict): Config dict. It should at least contain the key "type". + registry (:obj:`Registry`): The registry to search the type from. + default_args (dict, optional): Default initialization arguments. + + Returns: + object: The constructed object. + """ + if not isinstance(cfg, dict): + raise TypeError(f"cfg must be a dict, but got {type(cfg)}") + if "type" not in cfg: + if default_args is None or "type" not in default_args: + raise KeyError( + '`cfg` or `default_args` must contain the key "type", ' + f"but got {cfg}\n{default_args}" + ) + if not isinstance(registry, Registry): + raise TypeError( + "registry must be an mmcv.Registry object, " f"but got {type(registry)}" + ) + if not (isinstance(default_args, dict) or default_args is None): + raise TypeError( + "default_args must be a dict or None, " f"but got {type(default_args)}" + ) + + args = cfg.copy() + + if default_args is not None: + for name, value in default_args.items(): + args.setdefault(name, value) + + obj_type = args.pop("type") + if isinstance(obj_type, str): + obj_cls = registry.get(obj_type) + if obj_cls is None: + raise KeyError(f"{obj_type} is not in the {registry.name} registry") + elif inspect.isclass(obj_type): + obj_cls = obj_type + else: + raise TypeError(f"type must be a str or valid type, but got {type(obj_type)}") + try: + return obj_cls(**args) + except Exception as e: + # Normal TypeError does not print class name. + raise type(e)(f"{obj_cls.__name__}: {e}") + + +class Registry: + """A registry to map strings to classes. + + Registered object could be built from registry. + Example: + >>> MODELS = Registry('models') + >>> @MODELS.register_module() + >>> class ResNet: + >>> pass + >>> resnet = MODELS.build(dict(type='ResNet')) + + Please refer to + https://mmcv.readthedocs.io/en/latest/understand_mmcv/registry.html for + advanced usage. + + Args: + name (str): Registry name. + build_func(func, optional): Build function to construct instance from + Registry, func:`build_from_cfg` is used if neither ``parent`` or + ``build_func`` is specified. If ``parent`` is specified and + ``build_func`` is not given, ``build_func`` will be inherited + from ``parent``. Default: None. + parent (Registry, optional): Parent registry. The class registered in + children registry could be built from parent. Default: None. + scope (str, optional): The scope of registry. It is the key to search + for children registry. If not specified, scope will be the name of + the package where class is defined, e.g. mmdet, mmcls, mmseg. + Default: None. + """ + + def __init__(self, name, build_func=None, parent=None, scope=None): + self._name = name + self._module_dict = dict() + self._children = dict() + self._scope = self.infer_scope() if scope is None else scope + + # self.build_func will be set with the following priority: + # 1. build_func + # 2. parent.build_func + # 3. build_from_cfg + if build_func is None: + if parent is not None: + self.build_func = parent.build_func + else: + self.build_func = build_from_cfg + else: + self.build_func = build_func + if parent is not None: + assert isinstance(parent, Registry) + parent._add_children(self) + self.parent = parent + else: + self.parent = None + + def __len__(self): + return len(self._module_dict) + + def __contains__(self, key): + return self.get(key) is not None + + def __repr__(self): + format_str = ( + self.__class__.__name__ + f"(name={self._name}, " + f"items={self._module_dict})" + ) + return format_str + + @staticmethod + def infer_scope(): + """Infer the scope of registry. + + The name of the package where registry is defined will be returned. + + Example: + # in mmdet/models/backbone/resnet.py + >>> MODELS = Registry('models') + >>> @MODELS.register_module() + >>> class ResNet: + >>> pass + The scope of ``ResNet`` will be ``mmdet``. + + + Returns: + scope (str): The inferred scope name. + """ + # inspect.stack() trace where this function is called, the index-2 + # indicates the frame where `infer_scope()` is called + filename = inspect.getmodule(inspect.stack()[2][0]).__name__ + split_filename = filename.split(".") + return split_filename[0] + + @staticmethod + def split_scope_key(key): + """Split scope and key. + + The first scope will be split from key. + + Examples: + >>> Registry.split_scope_key('mmdet.ResNet') + 'mmdet', 'ResNet' + >>> Registry.split_scope_key('ResNet') + None, 'ResNet' + + Return: + scope (str, None): The first scope. + key (str): The remaining key. + """ + split_index = key.find(".") + if split_index != -1: + return key[:split_index], key[split_index + 1 :] + else: + return None, key + + @property + def name(self): + return self._name + + @property + def scope(self): + return self._scope + + @property + def module_dict(self): + return self._module_dict + + @property + def children(self): + return self._children + + def get(self, key): + """Get the registry record. + + Args: + key (str): The class name in string format. + + Returns: + class: The corresponding class. + """ + scope, real_key = self.split_scope_key(key) + if scope is None or scope == self._scope: + # get from self + if real_key in self._module_dict: + return self._module_dict[real_key] + else: + # get from self._children + if scope in self._children: + return self._children[scope].get(real_key) + else: + # goto root + parent = self.parent + while parent.parent is not None: + parent = parent.parent + return parent.get(key) + + def build(self, *args, **kwargs): + return self.build_func(*args, **kwargs, registry=self) + + def _add_children(self, registry): + """Add children for a registry. + + The ``registry`` will be added as children based on its scope. + The parent registry could build objects from children registry. + + Example: + >>> models = Registry('models') + >>> mmdet_models = Registry('models', parent=models) + >>> @mmdet_models.register_module() + >>> class ResNet: + >>> pass + >>> resnet = models.build(dict(type='mmdet.ResNet')) + """ + + assert isinstance(registry, Registry) + assert registry.scope is not None + assert ( + registry.scope not in self.children + ), f"scope {registry.scope} exists in {self.name} registry" + self.children[registry.scope] = registry + + def _register_module(self, module_class, module_name=None, force=False): + if not inspect.isclass(module_class): + raise TypeError("module must be a class, " f"but got {type(module_class)}") + + if module_name is None: + module_name = module_class.__name__ + if isinstance(module_name, str): + module_name = [module_name] + for name in module_name: + if not force and name in self._module_dict: + raise KeyError(f"{name} is already registered " f"in {self.name}") + self._module_dict[name] = module_class + + def deprecated_register_module(self, cls=None, force=False): + warnings.warn( + "The old API of register_module(module, force=False) " + "is deprecated and will be removed, please use the new API " + "register_module(name=None, force=False, module=None) instead." + ) + if cls is None: + return partial(self.deprecated_register_module, force=force) + self._register_module(cls, force=force) + return cls + + def register_module(self, name=None, force=False, module=None): + """Register a module. + + A record will be added to `self._module_dict`, whose key is the class + name or the specified name, and value is the class itself. + It can be used as a decorator or a normal function. + + Example: + >>> backbones = Registry('backbone') + >>> @backbones.register_module() + >>> class ResNet: + >>> pass + + >>> backbones = Registry('backbone') + >>> @backbones.register_module(name='mnet') + >>> class MobileNet: + >>> pass + + >>> backbones = Registry('backbone') + >>> class ResNet: + >>> pass + >>> backbones.register_module(ResNet) + + Args: + name (str | None): The module name to be registered. If not + specified, the class name will be used. + force (bool, optional): Whether to override an existing class with + the same name. Default: False. + module (type): Module class to be registered. + """ + if not isinstance(force, bool): + raise TypeError(f"force must be a boolean, but got {type(force)}") + # NOTE: This is a walkaround to be compatible with the old api, + # while it may introduce unexpected bugs. + if isinstance(name, type): + return self.deprecated_register_module(name, force=force) + + # raise the error ahead of time + if not (name is None or isinstance(name, str) or is_seq_of(name, str)): + raise TypeError( + "name must be either of None, an instance of str or a sequence" + f" of str, but got {type(name)}" + ) + + # use it as a normal method: x.register_module(module=SomeClass) + if module is not None: + self._register_module(module_class=module, module_name=name, force=force) + return module + + # use it as a decorator: @x.register_module() + def _register(cls): + self._register_module(module_class=cls, module_name=name, force=force) + return cls + + return _register diff --git a/pointcept/utils/scheduler.py b/pointcept/utils/scheduler.py new file mode 100644 index 0000000000000000000000000000000000000000..68198ba9f4b834486596fa4de9964ec9d992eb8d --- /dev/null +++ b/pointcept/utils/scheduler.py @@ -0,0 +1,206 @@ +""" +Scheduler + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import copy +import numpy as np +import torch.optim.lr_scheduler as lr_scheduler +from .registry import Registry + +SCHEDULERS = Registry("schedulers") + + +@SCHEDULERS.register_module() +class MultiStepLR(lr_scheduler.MultiStepLR): + def __init__( + self, + optimizer, + milestones, + total_steps, + gamma=0.1, + last_epoch=-1, + ): + super().__init__( + optimizer=optimizer, + milestones=[int(rate * total_steps) for rate in milestones], + gamma=gamma, + last_epoch=last_epoch, + ) + + +@SCHEDULERS.register_module() +class MultiStepWithWarmupLR(lr_scheduler.LambdaLR): + def __init__( + self, + optimizer, + milestones, + total_steps, + gamma=0.1, + warmup_rate=0.05, + warmup_scale=1e-6, + last_epoch=-1, + ): + milestones = [rate * total_steps for rate in milestones] + + def multi_step_with_warmup(s): + factor = 1.0 + for i in range(len(milestones)): + if s < milestones[i]: + break + factor *= gamma + + if s <= warmup_rate * total_steps: + warmup_coefficient = 1 - (1 - s / warmup_rate / total_steps) * ( + 1 - warmup_scale + ) + else: + warmup_coefficient = 1.0 + return warmup_coefficient * factor + + super().__init__( + optimizer=optimizer, + lr_lambda=multi_step_with_warmup, + last_epoch=last_epoch, + ) + + +@SCHEDULERS.register_module() +class PolyLR(lr_scheduler.LambdaLR): + def __init__( + self, + optimizer, + total_steps, + power=0.9, + last_epoch=-1, + ): + super().__init__( + optimizer=optimizer, + lr_lambda=lambda s: (1 - s / (total_steps + 1)) ** power, + last_epoch=last_epoch, + ) + + +@SCHEDULERS.register_module() +class ExpLR(lr_scheduler.LambdaLR): + def __init__( + self, + optimizer, + total_steps, + gamma=0.9, + last_epoch=-1, + ): + super().__init__( + optimizer=optimizer, + lr_lambda=lambda s: gamma ** (s / total_steps), + last_epoch=last_epoch, + ) + + +@SCHEDULERS.register_module() +class CosineAnnealingLR(lr_scheduler.CosineAnnealingLR): + def __init__( + self, + optimizer, + total_steps, + eta_min=0, + last_epoch=-1, + ): + super().__init__( + optimizer=optimizer, + T_max=total_steps, + eta_min=eta_min, + last_epoch=last_epoch, + ) + + +@SCHEDULERS.register_module() +class OneCycleLR(lr_scheduler.OneCycleLR): + r""" + torch.optim.lr_scheduler.OneCycleLR, Block total_steps + """ + + def __init__( + self, + optimizer, + max_lr, + total_steps=None, + pct_start=0.3, + anneal_strategy="cos", + cycle_momentum=True, + base_momentum=0.85, + max_momentum=0.95, + div_factor=25.0, + final_div_factor=1e4, + three_phase=False, + last_epoch=-1, + ): + super().__init__( + optimizer=optimizer, + max_lr=max_lr, + total_steps=total_steps, + pct_start=pct_start, + anneal_strategy=anneal_strategy, + cycle_momentum=cycle_momentum, + base_momentum=base_momentum, + max_momentum=max_momentum, + div_factor=div_factor, + final_div_factor=final_div_factor, + three_phase=three_phase, + last_epoch=last_epoch, + ) + + +@SCHEDULERS.register_module() +class CosineScheduler(object): + def __init__( + self, + base_value, + final_value, + total_iters, + start_value=0, + warmup_iters=0, + freeze_value=None, + freeze_iters=0, + ): + self.base_value = base_value + self.final_value = final_value + self.total_iters = total_iters + + warmup_schedule = np.linspace(start_value, base_value, warmup_iters) + + if freeze_value is None: + freeze_value = final_value + freeze_schedule = np.ones(freeze_iters) * freeze_value + + iters = np.arange(total_iters - warmup_iters - freeze_iters) + schedule = final_value + 0.5 * (base_value - final_value) * ( + 1 + np.cos(np.pi * iters / len(iters)) + ) + self.schedule = np.concatenate((warmup_schedule, schedule, freeze_schedule)) + self.iter = 0 + + def get(self, it): + if it >= self.total_iters: + return self.final_value + else: + return self.schedule[it] + + def step(self): + value = self.get(self.iter) + self.iter += 1 + return value + + def reset(self): + self.iter = 0 + + def __getitem__(self, it): + return self.get(it) + + +def build_scheduler(cfg, optimizer): + cfg = copy.deepcopy(cfg) + cfg.optimizer = optimizer + return SCHEDULERS.build(cfg=cfg) diff --git a/pointcept/utils/timer.py b/pointcept/utils/timer.py new file mode 100644 index 0000000000000000000000000000000000000000..3de4a16e33c43fe61ea3088f82216fd62eb6e959 --- /dev/null +++ b/pointcept/utils/timer.py @@ -0,0 +1,70 @@ +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved. +# -*- coding: utf-8 -*- + +from time import perf_counter +from typing import Optional + + +class Timer: + """ + A timer which computes the time elapsed since the start/reset of the timer. + """ + + def __init__(self) -> None: + self.reset() + + def reset(self) -> None: + """ + Reset the timer. + """ + self._start = perf_counter() + self._paused: Optional[float] = None + self._total_paused = 0 + self._count_start = 1 + + def pause(self) -> None: + """ + Pause the timer. + """ + if self._paused is not None: + raise ValueError("Trying to pause a Timer that is already paused!") + self._paused = perf_counter() + + def is_paused(self) -> bool: + """ + Returns: + bool: whether the timer is currently paused + """ + return self._paused is not None + + def resume(self) -> None: + """ + Resume the timer. + """ + if self._paused is None: + raise ValueError("Trying to resume a Timer that is not paused!") + # pyre-fixme[58]: `-` is not supported for operand types `float` and + # `Optional[float]`. + self._total_paused += perf_counter() - self._paused + self._paused = None + self._count_start += 1 + + def seconds(self) -> float: + """ + Returns: + (float): the total number of seconds since the start/reset of the + timer, excluding the time when the timer is paused. + """ + if self._paused is not None: + end_time: float = self._paused # type: ignore + else: + end_time = perf_counter() + return end_time - self._start - self._total_paused + + def avg_seconds(self) -> float: + """ + Returns: + (float): the average number of seconds between every start/reset and + pause. + """ + return self.seconds() / self._count_start diff --git a/pointcept/utils/visualization.py b/pointcept/utils/visualization.py new file mode 100644 index 0000000000000000000000000000000000000000..92ac39b3e87584921aa62db73da65d022f89dc77 --- /dev/null +++ b/pointcept/utils/visualization.py @@ -0,0 +1,128 @@ +""" +Visualization Utils + +Author: Xiaoyang Wu (xiaoyang.wu.cs@gmail.com) +Please cite our work if the code is helpful to you. +""" + +import os + +try: + import open3d as o3d +except ImportError: + o3d = None +import numpy as np +import torch + + +def to_numpy(x): + if isinstance(x, torch.Tensor): + x = x.clone().detach().cpu().numpy() + assert isinstance(x, np.ndarray) + return x + + +def get_point_cloud(coord, color=None, verbose=True): + if not isinstance(coord, list): + coord = [coord] + if color is not None: + color = [color] + + pcd_list = [] + for i in range(len(coord)): + coord_ = to_numpy(coord[i]) + if color is not None: + color_ = to_numpy(color[i]) + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coord_) + pcd.colors = o3d.utility.Vector3dVector( + np.zeros_like(coord_) if color is None else color_ + ) + pcd_list.append(pcd) + if verbose: + o3d.visualization.draw_geometries(pcd_list) + return pcd_list + + +def get_line_set(coord, line, color=(1.0, 0.0, 0.0), verbose=True): + coord = to_numpy(coord) + line = to_numpy(line) + colors = np.array([color for _ in range(len(line))]) + line_set = o3d.geometry.LineSet() + line_set.points = o3d.utility.Vector3dVector(coord) + line_set.lines = o3d.utility.Vector2iVector(line) + line_set.colors = o3d.utility.Vector3dVector(colors) + if verbose: + o3d.visualization.draw_geometries([line_set]) + return line_set + + +def save_point_cloud(coord, color=None, file_path="pc.ply", logger=None): + os.makedirs(os.path.dirname(file_path), exist_ok=True) + coord = to_numpy(coord) + if color is not None: + color = to_numpy(color) + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coord) + pcd.colors = o3d.utility.Vector3dVector( + np.ones_like(coord) if color is None else color + ) + o3d.io.write_point_cloud(file_path, pcd) + if logger is not None: + logger.info(f"Save Point Cloud to: {file_path}") + + +def save_bounding_boxes( + bboxes_corners, color=(1.0, 0.0, 0.0), file_path="bbox.ply", logger=None +): + bboxes_corners = to_numpy(bboxes_corners) + # point list + points = bboxes_corners.reshape(-1, 3) + # line list + box_lines = np.array( + [ + [0, 1], + [1, 2], + [2, 3], + [3, 0], + [4, 5], + [5, 6], + [6, 7], + [7, 0], + [0, 4], + [1, 5], + [2, 6], + [3, 7], + ] + ) + lines = [] + for i, _ in enumerate(bboxes_corners): + lines.append(box_lines + i * 8) + lines = np.concatenate(lines) + # color list + color = np.array([color for _ in range(len(lines))]) + # generate line set + line_set = o3d.geometry.LineSet() + line_set.points = o3d.utility.Vector3dVector(points) + line_set.lines = o3d.utility.Vector2iVector(lines) + line_set.colors = o3d.utility.Vector3dVector(color) + o3d.io.write_line_set(file_path, line_set) + + if logger is not None: + logger.info(f"Save Boxes to: {file_path}") + + +def save_lines( + points, lines, color=(1.0, 0.0, 0.0), file_path="lines.ply", logger=None +): + points = to_numpy(points) + lines = to_numpy(lines) + colors = np.array([color for _ in range(len(lines))]) + line_set = o3d.geometry.LineSet() + line_set.points = o3d.utility.Vector3dVector(points) + line_set.lines = o3d.utility.Vector2iVector(lines) + line_set.colors = o3d.utility.Vector3dVector(colors) + o3d.io.write_line_set(file_path, line_set) + + if logger is not None: + logger.info(f"Save Lines to: {file_path}") diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000000000000000000000000000000000000..4b797c5fd94aceae95c3ed7e6e62cef06faab988 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,47 @@ +[project] +name = "bimstruct3d-segmentation" +version = "0.1.0" +description = "PT-v3m1 point-cloud semantic segmentation model from the BIMStruct3D pipeline: bundled checkpoint + config + segment_scan.py." +requires-python = ">=3.12,<3.13" + +dependencies = [ + "torch==2.7.0", + "torchvision", + "spconv-cu126==2.3.8", + "torch-scatter==2.1.2+pt27cu126", + "torch-cluster==1.6.3+pt27cu126", + "timm", + "addict", + "peft", + "yapf", + "termcolor", + "scipy", + "pillow", + "laspy[laszip]", + "open3d", + "numpy", +] + +# Upstream Pointcept (github.com/Pointcept/Pointcept) ships no setup.py/pyproject.toml +# of its own -- it's only ever used via PYTHONPATH. The pointcept/ directory here is a +# trimmed vendored copy (inference-only backbones/datasets, see comments in its +# __init__.py files) packaged as part of this project so `uv sync` installs it for real +# instead of relying on a cwd-relative import. +[tool.hatch.build.targets.wheel] +packages = ["pointcept"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.uv] +find-links = ["https://data.pyg.org/whl/torch-2.7.0+cu126.html"] + +[tool.uv.sources] +torch = { index = "pytorch-cu126" } +torchvision = { index = "pytorch-cu126" } + +[[tool.uv.index]] +name = "pytorch-cu126" +url = "https://download.pytorch.org/whl/cu126" +explicit = true diff --git a/segment_scan.py b/segment_scan.py new file mode 100644 index 0000000000000000000000000000000000000000..d0c5013363fde4632b71463543d363022b20282c --- /dev/null +++ b/segment_scan.py @@ -0,0 +1,351 @@ +""" +Segment a single raw point-cloud scan (.las/.laz/.ply) with the bundled +PT-v3m1 model, using the pointcept/ package shipped alongside this script. + +Large scans are split into overlapping tiles (points don't fit the model / +GPU memory in one shot), each tile is run through the real Pointcept +inference pipeline (pointcept.datasets.DefaultDataset -> GridSample +fragments -> TTA -> model forward, same machinery pointcept/engines/test.py +uses), and per-point predictions from overlapping tiles are merged back by +majority vote. + +Usage: + python segment_scan.py --input scan.las --output scan_segmented.las + python segment_scan.py --input scan.ply --output scan_segmented.ply + +See README.md for setup / class list. +""" + +import argparse +import shutil +import tempfile +from collections import OrderedDict +from pathlib import Path + +import laspy +import numpy as np +import open3d as o3d +import torch +import torch.nn.functional as F +from addict import Dict as AttrDict + +from pointcept.datasets.defaults import DefaultDataset +from pointcept.datasets.utils import collate_fn +from pointcept.models import build_model +from pointcept.utils.config import Config + +# --------------------------------------------------------------------------- +# Preprocessing recipe the model was evaluated with (extracted from the +# original training config's `data.test` block; kept here rather than in +# configs/model_config.py since it's inference plumbing, not architecture). +# --------------------------------------------------------------------------- + +PREPROCESS_TRANSFORM = [ + dict(type="CenterShift", apply_z=True), + dict(type="NormalizeColor"), +] + +TEST_CFG = dict( + voxelize=dict( + type="GridSample", + grid_size=0.04, + hash_type="fnv", + mode="test", + return_grid_coord=True, + ), + crop=None, + post_transform=[ + dict(type="CenterShift", apply_z=False), + dict(type="ToTensor"), + dict( + type="Collect", + keys=("coord", "grid_coord", "index"), + feat_keys=("color", "normal"), + ), + ], + # test-time augmentation: 5 scales x {no-flip, flip} = 10 forward passes + # per tile, softmax-averaged. Matches how the checkpoint was evaluated. + aug_transform=[ + [dict(type="RandomScale", scale=[s, s])] for s in (0.9, 0.95, 1.0, 1.05, 1.1) + ] + + [ + [dict(type="RandomScale", scale=[s, s]), dict(type="RandomFlip", p=1)] + for s in (0.9, 0.95, 1.0, 1.05, 1.1) + ], +) + + +# --------------------------------------------------------------------------- +# Point-cloud IO +# --------------------------------------------------------------------------- + + +def read_scan(path: Path): + """Read a .las/.laz/.ply file, return (coord[float64,N,3], color[float32,N,3] in 0-255).""" + if path.suffix.lower() in (".las", ".laz"): + return _read_las(path) + elif path.suffix.lower() == ".ply": + return _read_ply(path) + raise ValueError(f"Unsupported input format {path.suffix!r} (expected .las/.laz/.ply)") + + +def _read_las(path: Path): + las = laspy.read(str(path)) + coord = np.vstack((las.x, las.y, las.z)).transpose().astype(np.float64) + rgb = np.vstack((las.red, las.green, las.blue)).transpose().astype(np.float64) + # LAS commonly stores color as 16-bit even when source precision is 8-bit; + # NormalizeColor (below) expects 0-255 range, matching Pointcept convention. + if rgb.max() > 255: + rgb = rgb / 257.0 # 65535 / 255 ~= 257 + color = rgb.astype(np.float32) + return coord, color + + +def _read_ply(path: Path): + pcd = o3d.io.read_point_cloud(str(path)) + coord = np.asarray(pcd.points, dtype=np.float64) + if pcd.has_colors(): + # open3d normalizes PLY colors to [0, 1] on read regardless of the + # source's bit depth; NormalizeColor (below) expects 0-255. + color = (np.asarray(pcd.colors, dtype=np.float64) * 255.0).astype(np.float32) + else: + color = np.zeros_like(coord, dtype=np.float32) + return coord, color + + +def estimate_normals(coord: np.ndarray) -> np.ndarray: + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(coord) + pcd.estimate_normals(fast_normal_computation=False) + return np.asarray(pcd.normals, dtype=np.float32) + + +def write_scan(path: Path, coord: np.ndarray, color: np.ndarray, labels: np.ndarray): + if path.suffix.lower() in (".las", ".laz"): + _write_las(path, coord, color, labels) + elif path.suffix.lower() == ".ply": + _write_ply(path, coord, color, labels) + else: + raise ValueError(f"Unsupported output format {path.suffix!r} (expected .las/.laz/.ply)") + np.save(str(path.with_name(path.stem + "_labels.npy")), labels) + + +def _write_las(path: Path, coord: np.ndarray, color: np.ndarray, labels: np.ndarray): + header = laspy.LasHeader(point_format=3, version="1.2") + header.offsets = np.min(coord, axis=0) + header.scales = np.array([0.001, 0.001, 0.001]) + las = laspy.LasData(header) + las.x, las.y, las.z = coord[:, 0], coord[:, 1], coord[:, 2] + las.red = np.clip(color[:, 0], 0, 255).astype("uint16") + las.green = np.clip(color[:, 1], 0, 255).astype("uint16") + las.blue = np.clip(color[:, 2], 0, 255).astype("uint16") + las.classification = labels.astype("uint8") + las.write(str(path)) + + +def _write_ply(path: Path, coord: np.ndarray, color: np.ndarray, labels: np.ndarray): + # Legacy open3d.geometry.PointCloud has no slot for a custom per-point + # scalar field, so the label is written via the tensor-based t.geometry + # API instead, as a "classification" field (readable in CloudCompare / + # MeshLab alongside "positions"/"colors"). + pcd = o3d.t.geometry.PointCloud() + pcd.point.positions = o3d.core.Tensor(coord.astype(np.float32)) + pcd.point.colors = o3d.core.Tensor(np.clip(color, 0, 255).astype(np.uint8)) + pcd.point.classification = o3d.core.Tensor(labels.astype(np.uint8).reshape(-1, 1)) + o3d.t.io.write_point_cloud(str(path), pcd) + + +# --------------------------------------------------------------------------- +# Tiling: split one large scan into overlapping axis-aligned tiles so each +# tile fits in memory/GPU. Ports the tiling primitives used elsewhere in +# this codebase for large-point-cloud inference, generalized to operate +# directly on a point array instead of a named dataset's preprocessing step. +# --------------------------------------------------------------------------- + + +def make_tile_boxes(coord: np.ndarray, tile_size_m: float, overlap_pct: float): + mins = coord.min(axis=0) + maxs = coord.max(axis=0) + x_range, y_range = maxs[0] - mins[0], maxs[1] - mins[1] + n = max(1, int(max(x_range, y_range) / tile_size_m) + 1) + + box_width = x_range / (n - overlap_pct / 100 * (n - 1)) if n > 1 else x_range + box_height = y_range / (n - overlap_pct / 100 * (n - 1)) if n > 1 else y_range + overlap_w = box_width * overlap_pct / 100 + overlap_h = box_height * overlap_pct / 100 + + boxes = [] + for i in range(n): + for j in range(n): + x0 = mins[0] + i * (box_width - overlap_w) + y0 = mins[1] + j * (box_height - overlap_h) + boxes.append((x0, x0 + box_width, y0, y0 + box_height)) + return boxes + + +def points_in_box(coord: np.ndarray, box) -> np.ndarray: + x0, x1, y0, y1 = box + mask = ( + (coord[:, 0] >= x0) + & (coord[:, 0] <= x1) + & (coord[:, 1] >= y0) + & (coord[:, 1] <= y1) + ) + return np.where(mask)[0] + + +# --------------------------------------------------------------------------- +# Model +# --------------------------------------------------------------------------- + + +def load_model(config_path: Path, weight_path: Path, device: str): + cfg = Config.fromfile(str(config_path)) + model = build_model(cfg.model).to(device) + + # weights_only=False: this checkpoint predates PyTorch 2.6's stricter + # default and needs the full unpickler. Only do this for checkpoints + # from a source you trust (as this one is -- it's our own training run). + checkpoint = torch.load(str(weight_path), map_location=device, weights_only=False) + weight = OrderedDict() + for key, value in checkpoint["state_dict"].items(): + weight[key[7:] if key.startswith("module.") else key] = value + model.load_state_dict(weight, strict=True) + model.eval() + return model, cfg + + +@torch.no_grad() +def infer_tile(model, dataset: DefaultDataset, idx: int, num_classes: int, device: str): + """Run one tiled DefaultDataset item (with TTA fragments) through the model. + + Mirrors pointcept.engines.test.SemSegTester's per-fragment loop. + """ + item = dataset[idx] + fragment_list = item["fragment_list"] + n_points = item["segment"].shape[0] + pred = torch.zeros((n_points, num_classes), device=device) + + for fragment in fragment_list: + batch = collate_fn([fragment]) + for key in batch: + if isinstance(batch[key], torch.Tensor): + batch[key] = batch[key].to(device, non_blocking=True) + logits = model(batch)["seg_logits"] + probs = F.softmax(logits, dim=-1) + pred[batch["index"]] += probs + + return pred.argmax(dim=1).cpu().numpy() + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--input", required=True, type=Path, help="Input .las/.laz/.ply scan") + parser.add_argument( + "--output", required=True, type=Path, help="Output labeled .las/.laz/.ply" + ) + parser.add_argument( + "--config", type=Path, default=Path(__file__).parent / "configs/model_config.py" + ) + parser.add_argument( + "--weight", type=Path, default=Path(__file__).parent / "weights/model_best.pth" + ) + parser.add_argument("--tile-size", type=float, default=20.0, help="Tile size in meters") + parser.add_argument("--overlap", type=float, default=25.0, help="Tile overlap, percent") + parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu") + parser.add_argument( + "--work-dir", type=Path, default=None, help="Temp dir for tiles (default: auto, deleted after run)" + ) + parser.add_argument( + "--keep-work-dir", action="store_true", help="Don't delete the tile working directory" + ) + args = parser.parse_args() + + print(f"[1/5] Loading model from {args.weight} (device={args.device}) ...") + model, cfg = load_model(args.config, args.weight, args.device) + num_classes = cfg.data.num_classes + class_names = cfg.data.names + + print(f"[2/5] Reading {args.input} ...") + coord, color = read_scan(args.input) + n_points = coord.shape[0] + print(f" {n_points:,} points") + + print(" estimating normals ...") + normal = estimate_normals(coord) + + print(f"[3/5] Tiling ({args.tile_size} m tiles, {args.overlap}% overlap) ...") + boxes = make_tile_boxes(coord, args.tile_size, args.overlap) + + work_dir = args.work_dir or Path(tempfile.mkdtemp(prefix="segment_scan_")) + tiles_dir = work_dir / "tiles" + tiles_dir.mkdir(parents=True, exist_ok=True) + + ids_by_tile_name = {} + n_tiles = 0 + for box in boxes: + idx = points_in_box(coord, box) + if idx.size == 0: + continue + tile_name = f"tile_{n_tiles:05d}" + tile_dir = tiles_dir / tile_name + tile_dir.mkdir(parents=True, exist_ok=True) + np.save(tile_dir / "coord.npy", coord[idx].astype(np.float32)) + np.save(tile_dir / "color.npy", color[idx]) + np.save(tile_dir / "normal.npy", normal[idx]) + ids_by_tile_name[tile_name] = idx + n_tiles += 1 + print(f" {n_tiles} non-empty tiles (out of {len(boxes)} grid cells)") + + print("[4/5] Running inference per tile ...") + dataset = DefaultDataset( + split="", + data_root=str(tiles_dir), + transform=PREPROCESS_TRANSFORM, + test_mode=True, + test_cfg=AttrDict(TEST_CFG), + ) + + votes = torch.zeros((n_points, num_classes), dtype=torch.int16) + for i in range(len(dataset)): + print(f" tile {i + 1}/{len(dataset)}", end="\r") + # DefaultDataset.get_data_list() uses glob, whose order isn't + # guaranteed to match tile-creation order -- look ids up by name. + tile_name = Path(dataset.data_list[i]).name + tile_pred = infer_tile(model, dataset, i, num_classes, args.device) + one_hot = F.one_hot(torch.from_numpy(tile_pred), num_classes=num_classes) + votes[ids_by_tile_name[tile_name]] += one_hot.to(torch.int16) + print() + + if not args.keep_work_dir and args.work_dir is None: + shutil.rmtree(work_dir, ignore_errors=True) + + print("[5/5] Merging and writing output ...") + covered = votes.sum(dim=1) > 0 + labels_full = votes.argmax(dim=1).numpy() + labels_full[~covered.numpy()] = 255 # sentinel for "never covered by a tile" + + out_coord = coord[covered.numpy()] + out_color = color[covered.numpy()] + out_labels = labels_full[covered.numpy()] + args.output.parent.mkdir(parents=True, exist_ok=True) + write_scan(args.output, out_coord, out_color, out_labels) + + dropped = n_points - out_coord.shape[0] + print(f" wrote {out_coord.shape[0]:,} labeled points to {args.output}") + if dropped: + print(f" ({dropped:,} points were not covered by any tile and were dropped)") + + counts = np.bincount(out_labels, minlength=num_classes) + print("\nLabel histogram:") + for name, count in zip(class_names, counts): + print(f" {name:>10s}: {count:,}") + + +if __name__ == "__main__": + main() diff --git a/uv.lock b/uv.lock new file mode 100644 index 0000000000000000000000000000000000000000..a39371c64aeaa0c92f5c2f412072a976c1129066 --- /dev/null +++ b/uv.lock @@ -0,0 +1,1648 @@ +version = 1 +revision = 3 +requires-python = "==3.12.*" +resolution-markers = [ + "sys_platform == 'win32'", + "sys_platform == 'emscripten'", + "(platform_machine != 'aarch64' and sys_platform == 'linux') or (platform_python_implementation != 'CPython' and sys_platform == 'linux') or (sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32')", + "platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux'", +] + +[[package]] +name = "accelerate" +version = "1.14.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "psutil" }, + { name = "pyyaml" }, + { name = "safetensors" }, + { name = "torch" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8d/75/94cd5d389649578aca399e5aa822637eec18319a1dadc400ffe2f9a7493f/accelerate-1.14.0.tar.gz", hash = "sha256:41b9c4377a54e0b460a959b0defa1b736e4ca0a2373252d9a539964c2afe3c8d", size = 412167, upload-time = "2026-06-11T13:45:52.326Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a8/db/253133d7e7cb40d3af384bb2f5c0b4a2b7fdcffbc95c688cc67a20a3c103/accelerate-1.14.0-py3-none-any.whl", hash = "sha256:e94390c2863b873be18f623f9df48a0d8fe5eff13ea7f1a00092b0a7904888c6", size = 389246, upload-time = "2026-06-11T13:45:50.477Z" }, +] + +[[package]] +name = "addict" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/85/ef/fd7649da8af11d93979831e8f1f8097e85e82d5bfeabc8c68b39175d8e75/addict-2.4.0.tar.gz", hash = "sha256:b3b2210e0e067a281f5646c8c5db92e99b7231ea8b0eb5f74dbdf9e259d4e494", size = 9186, upload-time = "2020-11-21T16:21:31.416Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/00/b08f23b7d7e1e14ce01419a467b583edbb93c6cdb8654e54a9cc579cd61f/addict-2.4.0-py3-none-any.whl", hash = "sha256:249bb56bbfd3cdc2a004ea0ff4c2b6ddc84d53bc2194761636eb314d5cfa5dfc", size = 3832, upload-time = "2020-11-21T16:21:29.588Z" }, +] + +[[package]] +name = "annotated-doc" +version = "0.0.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/8e/38aa427ed5402449e226975b649c5dc73ccadfefeb95e6aecb8f8ea4b6b6/annotated_doc-0.0.5.tar.gz", hash = "sha256:c7e58ce09192557605d8bbd92836d7e1d520ac9580096042c0bfd197efacf1bb", size = 10758, upload-time = "2026-07-28T13:50:58.129Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/30/e900b21425a860e195f32e37657aa1f7c7f2b1bfb26f03ca209b90933c06/annotated_doc-0.0.5-py3-none-any.whl", hash = "sha256:117bac03a25ede5df5440e855b32d556049ca169ead221505badf432fed4b101", size = 5302, upload-time = "2026-07-28T13:50:57.239Z" }, +] + +[[package]] +name = "annotated-types" +version = "0.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5f/56/a8120250d128bed162cd73c76d45f6ef9991f3e068f62a8ee060afa3104a/annotated_types-0.8.0.tar.gz", hash = "sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7", size = 15893, upload-time = "2026-07-23T20:16:13.995Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/99/91/8acff4f5e50511b911bbccb72b8628a49c68ce14148cd9f6431094859a90/annotated_types-0.8.0-py3-none-any.whl", hash = "sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0", size = 13427, upload-time = "2026-07-23T20:16:12.938Z" }, +] + +[[package]] +name = "anyio" +version = "4.14.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, +] + +[[package]] +name = "attrs" +version = "26.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" }, +] + +[[package]] +name = "bimstruct3d-segmentation" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "addict" }, + { name = "laspy", extra = ["laszip"] }, + { name = "numpy" }, + { name = "open3d" }, + { name = "peft" }, + { name = "pillow" }, + { name = "scipy" }, + { name = "spconv-cu126" }, + { name = "termcolor" }, + { name = "timm" }, + { name = "torch" }, + { name = "torch-cluster" }, + { name = "torch-scatter" }, + { name = "torchvision" }, + { name = "yapf" }, +] + +[package.metadata] +requires-dist = [ + { name = "addict" }, + { name = "laspy", extras = ["laszip"] }, + { name = "numpy" }, + { name = "open3d" }, + { name = "peft" }, + { name = "pillow" }, + { name = "scipy" }, + { name = "spconv-cu126", specifier = "==2.3.8" }, + { name = "termcolor" }, + { name = "timm" }, + { name = "torch", specifier = "==2.7.0", index = "https://download.pytorch.org/whl/cu126" }, + { name = "torch-cluster", specifier = "==1.6.3+pt27cu126" }, + { name = "torch-scatter", specifier = "==2.1.2+pt27cu126" }, + { name = "torchvision", index = "https://download.pytorch.org/whl/cu126" }, + { name = "yapf" }, +] + +[[package]] +name = "blinker" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/21/28/9b3f50ce0e048515135495f198351908d99540d69bfdc8c1d15b73dc55ce/blinker-1.9.0.tar.gz", hash = "sha256:b4ce2265a7abece45e7cc896e98dbebe6cead56bcf805a3d23136d145f5445bf", size = 22460, upload-time = "2024-11-08T17:25:47.436Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl", hash = "sha256:ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc", size = 8458, upload-time = "2024-11-08T17:25:46.184Z" }, +] + +[[package]] +name = "ccimport" +version = "0.4.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "ninja" }, + { name = "pybind11" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/95/08/69b69ae7f5410ca78cd14ca66d0acd1dc280aae9c5f4334a083219a5077f/ccimport-0.4.4.tar.gz", hash = "sha256:278c8416bf9fd8c445d0f8b17080e4e16f7abc7b0a68d4861857c3c4da755ce1", size = 27220, upload-time = "2024-10-15T15:06:10.338Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a3/fa/c069b7befe2f02825186ea1cd1dceb9c5356a80928271aa539b909daeaf4/ccimport-0.4.4-py3-none-any.whl", hash = "sha256:7c78f1258d372de787b7ebe351fb56890f2141b8643c3a45b1842b23828d5245", size = 27287, upload-time = "2024-10-15T15:06:08.932Z" }, +] + +[[package]] +name = "certifi" +version = "2026.7.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/c2/24167ea9858356b47a87a50d39908bfdb72ceeefe0041586e704e5376b3a/certifi-2026.7.22.tar.gz", hash = "sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55", size = 138112, upload-time = "2026-07-22T03:35:12.644Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl", hash = "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775", size = 136983, upload-time = "2026-07-22T03:35:11.276Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.4.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bd/2a/23f34ec9d04624958e137efdc394888716353190e75f25dd22c7a2c7a8aa/charset_normalizer-3.4.9.tar.gz", hash = "sha256:673611bbd43f0810bec0b0f028ddeaaa501190339cac411f347ac76917c3ae7b", size = 152439, upload-time = "2026-07-07T14:34:58.454Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/70/4a/ecbd131485c07fcdfad54e28946d513e3da22ef3b4bd854dcafae54ec739/charset_normalizer-3.4.9-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:45b0cc4e3556cd875e09102988d1ab8356c998b596c9fced84547c8138b487a0", size = 319300, upload-time = "2026-07-07T14:33:15.666Z" }, + { url = "https://files.pythonhosted.org/packages/ec/96/5d9364e3342d69f3a045e1777bc47c85c383e6e9466d561b33fdb419d1f9/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9b2aff1c7b3884512b9512c3eaadd9bab39fb45042ffaaa1dd08ff2b9f8109d9", size = 215802, upload-time = "2026-07-07T14:33:17.031Z" }, + { url = "https://files.pythonhosted.org/packages/4b/4c/5361f9aa7f2cb58d94f2ab831b3d493f69efb1d239654b4744e3c09527cb/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9104ed0bd76a429d46f9ec0dbc9b08ad1d2dcdf2b00a5a0daa1c145329b35b44", size = 237171, upload-time = "2026-07-07T14:33:18.576Z" }, + { url = "https://files.pythonhosted.org/packages/50/78/ce342ca4ff30b2eb49fe6d9578df85974f90c67d294113e94efdd9664cbd/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7b86a2b16095d250c6f58b3d9b2eee6f4147754344f3dab0922f7c9bf7d226c9", size = 233075, upload-time = "2026-07-07T14:33:20.084Z" }, + { url = "https://files.pythonhosted.org/packages/01/c4/4fa4c8b3097a11f3c5f09a35b72ed6855fb1d332469504962ab7bafcc702/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5e226f6218febc71f6c1fc2fafb91c226f75bdc1d8fb12d66823716e891608fd", size = 224256, upload-time = "2026-07-07T14:33:21.747Z" }, + { url = "https://files.pythonhosted.org/packages/87/3a/ad914516df7e358a81aae018caa5e0470ba827fa6d763b1d2e87d920a5f6/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:90c44bc373b7687f6948b693cceaea1348ae0975d7474746559494468e3c1d84", size = 208784, upload-time = "2026-07-07T14:33:23.313Z" }, + { url = "https://files.pythonhosted.org/packages/d7/74/3c12f9755717dfe5c5c87da63f35d765fa0c00382ec26bf23f7fae34f2ba/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9cdef90ae47919cae358d8ab15797a800ed41da7aba5d72419fb510729e2ed4b", size = 219928, upload-time = "2026-07-07T14:33:24.814Z" }, + { url = "https://files.pythonhosted.org/packages/33/9a/895095b83e7907abd6d3d99aad3a38ad0d9686cc186cb0c94c24320fe63e/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:60f44ade2cf573dad7a277e6f8ca9a51a21dda572b13bd7d8539bb3cd5dbedde", size = 218489, upload-time = "2026-07-07T14:33:26.42Z" }, + { url = "https://files.pythonhosted.org/packages/a1/34/ef5c05f412f42520d7709b7d3784d19640839eb7366ded1755511585429f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:a1786910334ed46ab1dd73222f2cd1e05c2c3bb39f6dddb4f8b36fc382058a39", size = 210267, upload-time = "2026-07-07T14:33:27.952Z" }, + { url = "https://files.pythonhosted.org/packages/83/dc/9b29fa4412b318bf3bfea985c35d67eb55e04b59a7c3f2237168b0e0be6f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:03d07803992c6c7bbc976327f34b18b6160327fc81cb82c9d504720ac0be3b62", size = 226030, upload-time = "2026-07-07T14:33:29.397Z" }, + { url = "https://files.pythonhosted.org/packages/0e/42/6dbc00b8cd16011691203e33570fa42ed5746599a2e878112d16eab403a3/charset_normalizer-3.4.9-cp312-cp312-win32.whl", hash = "sha256:78841cccf1af7b40f6f716338d50c0902dbe88d9f800b3c973b7a9a0a693a642", size = 151185, upload-time = "2026-07-07T14:33:30.781Z" }, + { url = "https://files.pythonhosted.org/packages/80/cc/f920afd1a23c58ccd53c1d36085a71893a4737ff5e66e0371efab6809850/charset_normalizer-3.4.9-cp312-cp312-win_amd64.whl", hash = "sha256:4b3dac63058cc36820b0dd072f89898604e2d39686fe05321729d00d8ac185a0", size = 162557, upload-time = "2026-07-07T14:33:32.176Z" }, + { url = "https://files.pythonhosted.org/packages/f0/e6/0386d43a261ff4e4b30c5857af7df877254b46bec7b9d1b74b6bf969a90b/charset_normalizer-3.4.9-cp312-cp312-win_arm64.whl", hash = "sha256:78fa18e436a1a0e58dbd7e02fc4473f3f32cceb12df9dfca542d075961c307d2", size = 152665, upload-time = "2026-07-07T14:33:33.711Z" }, + { url = "https://files.pythonhosted.org/packages/98/2b/f97f1c193fb855c345d678f5077d6926034db0722df74c8f057020e05a25/charset_normalizer-3.4.9-py3-none-any.whl", hash = "sha256:68e5f26a1ad57ded6d1cfb85331d1c1a195314756471d97758c48498bb4dcdf5", size = 64538, upload-time = "2026-07-07T14:34:56.993Z" }, +] + +[[package]] +name = "click" +version = "8.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/76/d4/81420972a676e8ffea40450d8c8c92943e7218a78fe9b64359836cc9876b/click-8.4.2.tar.gz", hash = "sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6", size = 338000, upload-time = "2026-06-24T17:45:15.148Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/e2/79c688af8b210d232694e31e59da9f6ec747bae31c3f5946e4e9b98860d5/click-8.4.2-py3-none-any.whl", hash = "sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76", size = 119243, upload-time = "2026-06-24T17:45:13.73Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "comm" +version = "0.2.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4c/13/7d740c5849255756bc17888787313b61fd38a0a8304fc4f073dfc46122aa/comm-0.2.3.tar.gz", hash = "sha256:2dc8048c10962d55d7ad693be1e7045d891b7ce8d999c97963a5e3e99c055971", size = 6319, upload-time = "2025-07-25T14:02:04.452Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/60/97/891a0971e1e4a8c5d2b20bbe0e524dc04548d2307fee33cdeba148fd4fc7/comm-0.2.3-py3-none-any.whl", hash = "sha256:c615d91d75f7f04f095b30d1c1711babd43bdc6419c1be9886a85f2f4e489417", size = 7294, upload-time = "2025-07-25T14:02:02.896Z" }, +] + +[[package]] +name = "configargparse" +version = "1.7.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3f/0b/30328302903c55218ffc5199646d0e9d28348ff26c02ba77b2ffc58d294a/configargparse-1.7.5.tar.gz", hash = "sha256:e3f9a7bb6be34d66b2e3c4a2f58e3045f8dfae47b0dc039f87bcfaa0f193fb0f", size = 53548, upload-time = "2026-03-11T02:19:38.144Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fe/19/3ba5e1b0bcc7b91aeab6c258afd70e4907d220fed3972febe38feb40db30/configargparse-1.7.5-py3-none-any.whl", hash = "sha256:1e63fdffedf94da9cd435fc13a1cd24777e76879dd2343912c1f871d4ac8c592", size = 27692, upload-time = "2026-03-11T02:19:36.442Z" }, +] + +[[package]] +name = "contourpy" +version = "1.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/58/01/1253e6698a07380cd31a736d248a3f2a50a7c88779a1813da27503cadc2a/contourpy-1.3.3.tar.gz", hash = "sha256:083e12155b210502d0bca491432bb04d56dc3432f95a979b429f2848c3dbe880", size = 13466174, upload-time = "2025-07-26T12:03:12.549Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/be/45/adfee365d9ea3d853550b2e735f9d66366701c65db7855cd07621732ccfc/contourpy-1.3.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b08a32ea2f8e42cf1d4be3169a98dd4be32bafe4f22b6c4cb4ba810fa9e5d2cb", size = 293419, upload-time = "2025-07-26T12:01:21.16Z" }, + { url = "https://files.pythonhosted.org/packages/53/3e/405b59cfa13021a56bba395a6b3aca8cec012b45bf177b0eaf7a202cde2c/contourpy-1.3.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:556dba8fb6f5d8742f2923fe9457dbdd51e1049c4a43fd3986a0b14a1d815fc6", size = 273979, upload-time = "2025-07-26T12:01:22.448Z" }, + { url = "https://files.pythonhosted.org/packages/d4/1c/a12359b9b2ca3a845e8f7f9ac08bdf776114eb931392fcad91743e2ea17b/contourpy-1.3.3-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:92d9abc807cf7d0e047b95ca5d957cf4792fcd04e920ca70d48add15c1a90ea7", size = 332653, upload-time = "2025-07-26T12:01:24.155Z" }, + { url = "https://files.pythonhosted.org/packages/63/12/897aeebfb475b7748ea67b61e045accdfcf0d971f8a588b67108ed7f5512/contourpy-1.3.3-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b2e8faa0ed68cb29af51edd8e24798bb661eac3bd9f65420c1887b6ca89987c8", size = 379536, upload-time = "2025-07-26T12:01:25.91Z" }, + { url = "https://files.pythonhosted.org/packages/43/8a/a8c584b82deb248930ce069e71576fc09bd7174bbd35183b7943fb1064fd/contourpy-1.3.3-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:626d60935cf668e70a5ce6ff184fd713e9683fb458898e4249b63be9e28286ea", size = 384397, upload-time = "2025-07-26T12:01:27.152Z" }, + { url = "https://files.pythonhosted.org/packages/cc/8f/ec6289987824b29529d0dfda0d74a07cec60e54b9c92f3c9da4c0ac732de/contourpy-1.3.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4d00e655fcef08aba35ec9610536bfe90267d7ab5ba944f7032549c55a146da1", size = 362601, upload-time = "2025-07-26T12:01:28.808Z" }, + { url = "https://files.pythonhosted.org/packages/05/0a/a3fe3be3ee2dceb3e615ebb4df97ae6f3828aa915d3e10549ce016302bd1/contourpy-1.3.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:451e71b5a7d597379ef572de31eeb909a87246974d960049a9848c3bc6c41bf7", size = 1331288, upload-time = "2025-07-26T12:01:31.198Z" }, + { url = "https://files.pythonhosted.org/packages/33/1d/acad9bd4e97f13f3e2b18a3977fe1b4a37ecf3d38d815333980c6c72e963/contourpy-1.3.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:459c1f020cd59fcfe6650180678a9993932d80d44ccde1fa1868977438f0b411", size = 1403386, upload-time = "2025-07-26T12:01:33.947Z" }, + { url = "https://files.pythonhosted.org/packages/cf/8f/5847f44a7fddf859704217a99a23a4f6417b10e5ab1256a179264561540e/contourpy-1.3.3-cp312-cp312-win32.whl", hash = "sha256:023b44101dfe49d7d53932be418477dba359649246075c996866106da069af69", size = 185018, upload-time = "2025-07-26T12:01:35.64Z" }, + { url = "https://files.pythonhosted.org/packages/19/e8/6026ed58a64563186a9ee3f29f41261fd1828f527dd93d33b60feca63352/contourpy-1.3.3-cp312-cp312-win_amd64.whl", hash = "sha256:8153b8bfc11e1e4d75bcb0bff1db232f9e10b274e0929de9d608027e0d34ff8b", size = 226567, upload-time = "2025-07-26T12:01:36.804Z" }, + { url = "https://files.pythonhosted.org/packages/d1/e2/f05240d2c39a1ed228d8328a78b6f44cd695f7ef47beb3e684cf93604f86/contourpy-1.3.3-cp312-cp312-win_arm64.whl", hash = "sha256:07ce5ed73ecdc4a03ffe3e1b3e3c1166db35ae7584be76f65dbbe28a7791b0cc", size = 193655, upload-time = "2025-07-26T12:01:37.999Z" }, +] + +[[package]] +name = "cumm-cu126" +version = "0.7.11" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "fire" }, + { name = "numpy" }, + { name = "pccm" }, + { name = "pybind11" }, + { name = "sympy" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/5a/d87f69d6dafa63d192cfdd08339962d126c7954f216c74d05dc380e51761/cumm_cu126-0.7.11-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:569fadb2773acb1fdd49f5406352e4415407a1dedebe33e6fd7603414dee9775", size = 26587013, upload-time = "2024-12-15T13:58:28.95Z" }, + { url = "https://files.pythonhosted.org/packages/a5/f7/1818da409d682b435a6f61d5a55e09fc02153f9ac38c422a28337abe6410/cumm_cu126-0.7.11-cp312-cp312-win_amd64.whl", hash = "sha256:4bf211b216906165847e00fa691578fdce0e233472e3e1cf8ebcd5dbd1f818bf", size = 1221855, upload-time = "2024-12-15T13:38:27.616Z" }, +] + +[[package]] +name = "cycler" +version = "0.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a9/95/a3dbbb5028f35eafb79008e7522a75244477d2838f38cbb722248dabc2a8/cycler-0.12.1.tar.gz", hash = "sha256:88bb128f02ba341da8ef447245a9e138fae777f6a23943da4540077d3601eb1c", size = 7615, upload-time = "2023-10-07T05:32:18.335Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321, upload-time = "2023-10-07T05:32:16.783Z" }, +] + +[[package]] +name = "dash" +version = "4.4.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "comm" }, + { name = "flask" }, + { name = "importlib-metadata" }, + { name = "janus" }, + { name = "nest-asyncio" }, + { name = "plotly" }, + { name = "pydantic" }, + { name = "requests" }, + { name = "retrying" }, + { name = "setuptools" }, + { name = "typing-extensions" }, + { name = "werkzeug" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5a/67/d5cea6dedfbfde4165436fb58154480ea6358a9e71a739d3366575b5b724/dash-4.4.1.tar.gz", hash = "sha256:9356ca7856bc496c12c5b6de5978aaf783e090d07450ee395ade1c0c2cbdf816", size = 8536482, upload-time = "2026-07-21T18:58:19.72Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9a/29/498125538103a11eab9cfbbde41bacf93e14ab3ac92ae9fb4ac0ac4dd61b/dash-4.4.1-py3-none-any.whl", hash = "sha256:72120a91b10ee4d73f9446efd5d6a4ec218086feed7b1b479d2259844d1f658f", size = 8945027, upload-time = "2026-07-21T18:58:11.405Z" }, +] + +[[package]] +name = "fastjsonschema" +version = "2.22.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e4/98/474719c58eddaf77fa443b063693e76d49db32bbe851bcbaf58d2700119f/fastjsonschema-2.22.1.tar.gz", hash = "sha256:0b83d1ce8d7845b959dcb20e1a5c3c8883b6541d9c52ab02cce5166b75ec805f", size = 382291, upload-time = "2026-07-27T13:31:08.515Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/17/e1/62cc96341f01bdff2ba967441939178fcd1900d11ce7e6554d9954a5d7ec/fastjsonschema-2.22.1-py3-none-any.whl", hash = "sha256:cf377ff5c9a6f4f3125fb35f75a2c5767bd824ffbcf62c209a93cd48d1453999", size = 26239, upload-time = "2026-07-27T13:31:03.251Z" }, +] + +[[package]] +name = "filelock" +version = "3.32.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/57/3ba6e6cb097f85b855b00163d169f35365f44277df044dcf96d55b8f62a3/filelock-3.32.2.tar.gz", hash = "sha256:c33351e1f49cae33414acbc6d56784e6ecee82514ec90795da1161fc4836b5b8", size = 217172, upload-time = "2026-07-29T22:46:04.895Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/e8/72f8cef9fdfeffe06213fe8508039396ee48daa0e3259457ed766173bfd6/filelock-3.32.2-py3-none-any.whl", hash = "sha256:87dd94cf281e586d135fa51132b8e3d9a598b316e90377a288663c9321036c82", size = 98830, upload-time = "2026-07-29T22:46:03.52Z" }, +] + +[[package]] +name = "fire" +version = "0.7.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "termcolor" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c0/00/f8d10588d2019d6d6452653def1ee807353b21983db48550318424b5ff18/fire-0.7.1.tar.gz", hash = "sha256:3b208f05c736de98fb343310d090dcc4d8c78b2a89ea4f32b837c586270a9cbf", size = 88720, upload-time = "2025-08-16T20:20:24.175Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/4c/93d0f85318da65923e4b91c1c2ff03d8a458cbefebe3bc612a6693c7906d/fire-0.7.1-py3-none-any.whl", hash = "sha256:e43fd8a5033a9001e7e2973bab96070694b9f12f2e0ecf96d4683971b5ab1882", size = 115945, upload-time = "2025-08-16T20:20:22.87Z" }, +] + +[[package]] +name = "flask" +version = "3.1.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "blinker" }, + { name = "click" }, + { name = "itsdangerous" }, + { name = "jinja2" }, + { name = "markupsafe" }, + { name = "werkzeug" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/26/00/35d85dcce6c57fdc871f3867d465d780f302a175ea360f62533f12b27e2b/flask-3.1.3.tar.gz", hash = "sha256:0ef0e52b8a9cd932855379197dd8f94047b359ca0a78695144304cb45f87c9eb", size = 759004, upload-time = "2026-02-19T05:00:57.678Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/9c/34f6962f9b9e9c71f6e5ed806e0d0ff03c9d1b0b2340088a0cf4bce09b18/flask-3.1.3-py3-none-any.whl", hash = "sha256:f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c", size = 103424, upload-time = "2026-02-19T05:00:56.027Z" }, +] + +[[package]] +name = "fonttools" +version = "4.63.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/84/69/c97f2c18e0db87d2c7b15da1974dace76ae938f1cfa22e2727a648b7ed43/fonttools-4.63.0.tar.gz", hash = "sha256:caeb583deeb5168e694b65cda8b4ee62abedfa66cf88488734466f2366b9c4e0", size = 3597189, upload-time = "2026-05-14T12:04:30.958Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/08/ef/b3c6b9b5be2f82416d73fe2ed2e96e2793cd80e7510bd6a17ca79cdd88ec/fonttools-4.63.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:37dd23e621e3b0aef1baa70a303b80aaf38449632cfc8fd2a55fb285bbccfc02", size = 2881131, upload-time = "2026-05-14T12:03:13.386Z" }, + { url = "https://files.pythonhosted.org/packages/44/a0/c815bea63117fa63e4e1c01f8a1110d2112fa003f838e6467094ec2432ce/fonttools-4.63.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a9faff9e0c1f76f9fd55899d2ce785832efebab37eb8ae13995853aef178bef0", size = 2426704, upload-time = "2026-05-14T12:03:15.801Z" }, + { url = "https://files.pythonhosted.org/packages/44/04/0b91d8e916e92ad1fac9e4624760baf0fd5ff2ead614c2f68fb21373f03f/fonttools-4.63.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ef3048ef05dbb552b89817713d9cac912e00d0fde4a3105c00d29e52e10c89af", size = 5044298, upload-time = "2026-05-14T12:03:18.085Z" }, + { url = "https://files.pythonhosted.org/packages/77/c7/2342da9830e3e9d4870305ca5d2091d2a83284f2953079b7bdd3b5e029d8/fonttools-4.63.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:58dc6bb86a78d782f00f9190ca02c119cf5bbe2807536e361e18d42019f877d8", size = 4999800, upload-time = "2026-05-14T12:03:20.161Z" }, + { url = "https://files.pythonhosted.org/packages/e6/6d/67fe16c48d7ce050979b33f47e0d28a318f02da030602e944c34f7a16ef3/fonttools-4.63.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ee08ebfa58f6e1aeff5697ab9582105bb620008c1caafb681e4c557e7483027b", size = 4982666, upload-time = "2026-05-14T12:03:22.87Z" }, + { url = "https://files.pythonhosted.org/packages/f2/00/3bbab338c07c71fa56269953845e92c951a61457bbbb0f1022551ea266d9/fonttools-4.63.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:27fdc65af8da6f88b9c6121c47a464cbe359fcfff7ff6fc2d37a1f395d755b78", size = 5133598, upload-time = "2026-05-14T12:03:25.168Z" }, + { url = "https://files.pythonhosted.org/packages/62/f2/aa27c7f98db5b064883dadcc5283947e81e034de42e22a33675878d98b54/fonttools-4.63.0-cp312-cp312-win32.whl", hash = "sha256:af2fd1664d00a397d75f806985ddb36282091c2131a73a6485c23b4a34722263", size = 2292575, upload-time = "2026-05-14T12:03:27.496Z" }, + { url = "https://files.pythonhosted.org/packages/87/36/cccb9bc2a6ab63d1b2980374f0dca72ce95ae267c9b4cfe77455bb70d0d4/fonttools-4.63.0-cp312-cp312-win_amd64.whl", hash = "sha256:59ac449f8cca9b4ffa08d2e7bbadad87ce710d69d1eda5c3c1ce579baa987272", size = 2343211, upload-time = "2026-05-14T12:03:30.057Z" }, + { url = "https://files.pythonhosted.org/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" }, +] + +[[package]] +name = "fsspec" +version = "2026.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/00/78/f34251dadb8f3921264a1d9b8946f5e542014ee2614b285261b4e40e6775/fsspec-2026.7.0.tar.gz", hash = "sha256:c803c40f4cf860b49dea58ee3e1c33cb9c790520e233537e1340049f89b82a88", size = 317040, upload-time = "2026-07-28T16:34:51.052Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fd/3c/6a2bf344106328fd04963664a60b9bb6496fc25df8e962fcdc1367285fb9/fsspec-2026.7.0-py3-none-any.whl", hash = "sha256:b57ddbafedfaef7018c1ecab32aa200a9d7ca26b77965f64e48b70061249d279", size = 206583, upload-time = "2026-07-28T16:34:49.538Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "hf-xet" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1b/ab/522a2ab67f27971a9d48ca666d4fca85ef7d5282d142e31fd087e27b1bbe/hf_xet-1.6.0.tar.gz", hash = "sha256:2e58454a340b3556dfa4972d5451aff4fba8dd42a236600ba1a1d2b1514f0fef", size = 920527, upload-time = "2026-08-03T22:33:13.243Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/50/7afa2c9c787405864fc47a0d1bbc02c62e9101947ed43c1f43899fc7d91d/hf_xet-1.6.0-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:633dc0cd71d32da58ab8c03ad38e2fac452c15c2b0a2866ebf6ededfe0a5061d", size = 4071729, upload-time = "2026-08-03T22:33:00.721Z" }, + { url = "https://files.pythonhosted.org/packages/4b/69/55b8dcf636142ae660fec1869fcac14c4da2e8412e14d6eee1523be77e9f/hf_xet-1.6.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:f0906082d9932ae0c0057fa194041c22b4e2cdb46b2592ef3b91f020d62a081a", size = 3876287, upload-time = "2026-08-03T22:33:02.251Z" }, + { url = "https://files.pythonhosted.org/packages/67/4e/a28359bf1c1ecf11eba22123168c138698f7cb576ac678f5a2e16cd5da08/hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d62671bb130879cef0ee4c9ebe47a14af6c66ec53e6d84dc15936e5ffdfac82f", size = 4464663, upload-time = "2026-08-03T22:33:03.802Z" }, + { url = "https://files.pythonhosted.org/packages/9a/69/1f0cbc2fb22ae6082d094f743d1b8945a3f36f6089cb95f42b7ee348cda7/hf_xet-1.6.0-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:0e6e21fa3cdfcdcd76748564bf593870a5e013f47d97cf10aed63aa222cff5b7", size = 4262538, upload-time = "2026-08-03T22:33:05.287Z" }, + { url = "https://files.pythonhosted.org/packages/d1/3a/4f4f2301ade26e404462d3336fa11f7958d914cabbabdd6e03c3c5d5658c/hf_xet-1.6.0-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:4fc74352a17015bd0ee90038bc9efe38db894cde45f268b6712b04fce8cd0acb", size = 4460520, upload-time = "2026-08-03T22:33:06.81Z" }, + { url = "https://files.pythonhosted.org/packages/ab/5f/311725e2a905534dfee2dcb5b08414f249147f1f12252bfc2bd24caa075c/hf_xet-1.6.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8fb4f71cba6129110c3374a33f919001ff130488fc23553698e34cc1c2a1198c", size = 4675937, upload-time = "2026-08-03T22:33:08.616Z" }, + { url = "https://files.pythonhosted.org/packages/98/b7/8c59a66d15205024662f1d66968136f13893f96df1ddc5087e2e281fc95f/hf_xet-1.6.0-cp38-abi3-win_amd64.whl", hash = "sha256:fb4fadde1b2b70bf4c0c14a6dccbe7194b1c28947fefd5bbe3fed9d940676c3b", size = 4033128, upload-time = "2026-08-03T22:33:10.171Z" }, + { url = "https://files.pythonhosted.org/packages/73/63/ca511b6f802f28cf3489b280fe77475bcca8de85e81a6299d7916b5b5555/hf_xet-1.6.0-cp38-abi3-win_arm64.whl", hash = "sha256:3dc3e35441ba395006af5aaacc40ef2e603c51ef46c3530b9156185f00935ea3", size = 3859359, upload-time = "2026-08-03T22:33:11.725Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "huggingface-hub" +version = "1.27.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "filelock" }, + { name = "fsspec" }, + { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" }, + { name = "httpx" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "tqdm" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3e/9b/ddf3d02a8681f1b9ce52fda03d755dad6b74c4f8172304c4c8d2975450f9/huggingface_hub-1.27.0.tar.gz", hash = "sha256:c1fed40ea82a6b41b477f5243546549b792ae0a93abcea608cff66089bf8f8df", size = 942668, upload-time = "2026-08-07T12:48:05.161Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/de/d8/95b735e183957c1f26d94c52977f09d466d55119cbbc1558ea4975e4c216/huggingface_hub-1.27.0-py3-none-any.whl", hash = "sha256:7df6827c2f956c60fbaa64646e979e566db76f619dd0a9729dfb8c5a3eb4f68d", size = 784926, upload-time = "2026-08-07T12:48:02.905Z" }, +] + +[[package]] +name = "idna" +version = "3.18" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/63/9496c57188a2ee585e0f1db071d75089a11e98aa86eb99d9d7618fc1edce/idna-3.18.tar.gz", hash = "sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848", size = 196711, upload-time = "2026-06-02T14:34:07.794Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/5e/d4e9f1a599fb8e573b7b87160658329fbf28d19eac2718f51fc3def3aa5a/idna-3.18-py3-none-any.whl", hash = "sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2", size = 65455, upload-time = "2026-06-02T14:34:06.319Z" }, +] + +[[package]] +name = "importlib-metadata" +version = "9.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "zipp" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a9/01/15bb152d77b21318514a96f43af312635eb2500c96b55398d020c93d86ea/importlib_metadata-9.0.0.tar.gz", hash = "sha256:a4f57ab599e6a2e3016d7595cfd72eb4661a5106e787a95bcc90c7105b831efc", size = 56405, upload-time = "2026-03-20T06:42:56.999Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/38/3d/2d244233ac4f76e38533cfcb2991c9eb4c7bf688ae0a036d30725b8faafe/importlib_metadata-9.0.0-py3-none-any.whl", hash = "sha256:2d21d1cc5a017bd0559e36150c21c830ab1dc304dedd1b7ea85d20f45ef3edd7", size = 27789, upload-time = "2026-03-20T06:42:55.665Z" }, +] + +[[package]] +name = "itsdangerous" +version = "2.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9c/cb/8ac0172223afbccb63986cc25049b154ecfb5e85932587206f42317be31d/itsdangerous-2.2.0.tar.gz", hash = "sha256:e0050c0b7da1eea53ffaf149c0cfbb5c6e2e2b69c4bef22c81fa6eb73e5f6173", size = 54410, upload-time = "2024-04-16T21:28:15.614Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl", hash = "sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef", size = 16234, upload-time = "2024-04-16T21:28:14.499Z" }, +] + +[[package]] +name = "janus" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/7f/69884b6618be4baf6ebcacc716ee8680a842428a19f403db6d1c0bb990aa/janus-2.0.0.tar.gz", hash = "sha256:0970f38e0e725400496c834a368a67ee551dc3b5ad0a257e132f5b46f2e77770", size = 22910, upload-time = "2024-12-13T12:59:08.622Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/68/34/65604740edcb20e1bda6a890348ed7d282e7dd23aa00401cbe36fd0edbd9/janus-2.0.0-py3-none-any.whl", hash = "sha256:7e6449d34eab04cd016befbd7d8c0d8acaaaab67cb59e076a69149f9031745f9", size = 12161, upload-time = "2024-12-13T12:59:06.106Z" }, +] + +[[package]] +name = "jinja2" +version = "3.1.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, +] + +[[package]] +name = "joblib" +version = "1.5.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/41/f2/d34e8b3a08a9cc79a50b2208a93dce981fe615b64d5a4d4abee421d898df/joblib-1.5.3.tar.gz", hash = "sha256:8561a3269e6801106863fd0d6d84bb737be9e7631e33aaed3fb9ce5953688da3", size = 331603, upload-time = "2025-12-15T08:41:46.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl", hash = "sha256:5fc3c5039fc5ca8c0276333a188bbd59d6b7ab37fe6632daa76bc7f9ec18e713", size = 309071, upload-time = "2025-12-15T08:41:44.973Z" }, +] + +[[package]] +name = "jsonschema" +version = "4.26.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "jsonschema-specifications" }, + { name = "referencing" }, + { name = "rpds-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b3/fc/e067678238fa451312d4c62bf6e6cf5ec56375422aee02f9cb5f909b3047/jsonschema-4.26.0.tar.gz", hash = "sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326", size = 366583, upload-time = "2026-01-07T13:41:07.246Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/69/90/f63fb5873511e014207a475e2bb4e8b2e570d655b00ac19a9a0ca0a385ee/jsonschema-4.26.0-py3-none-any.whl", hash = "sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce", size = 90630, upload-time = "2026-01-07T13:41:05.306Z" }, +] + +[[package]] +name = "jsonschema-specifications" +version = "2025.9.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "referencing" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/19/74/a633ee74eb36c44aa6d1095e7cc5569bebf04342ee146178e2d36600708b/jsonschema_specifications-2025.9.1.tar.gz", hash = "sha256:b540987f239e745613c7a9176f3edb72b832a4ac465cf02712288397832b5e8d", size = 32855, upload-time = "2025-09-08T01:34:59.186Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" }, +] + +[[package]] +name = "jupyter-core" +version = "5.9.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "platformdirs" }, + { name = "traitlets" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/02/49/9d1284d0dc65e2c757b74c6687b6d319b02f822ad039e5c512df9194d9dd/jupyter_core-5.9.1.tar.gz", hash = "sha256:4d09aaff303b9566c3ce657f580bd089ff5c91f5f89cf7d8846c3cdf465b5508", size = 89814, upload-time = "2025-10-16T19:19:18.444Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/e7/80988e32bf6f73919a113473a604f5a8f09094de312b9d52b79c2df7612b/jupyter_core-5.9.1-py3-none-any.whl", hash = "sha256:ebf87fdc6073d142e114c72c9e29a9d7ca03fad818c5d300ce2adc1fb0743407", size = 29032, upload-time = "2025-10-16T19:19:16.783Z" }, +] + +[[package]] +name = "kiwisolver" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d0/67/9c61eccb13f0bdca9307614e782fec49ffdde0f7a2314935d489fa93cd9c/kiwisolver-1.5.0.tar.gz", hash = "sha256:d4193f3d9dc3f6f79aaed0e5637f45d98850ebf01f7ca20e69457f3e8946b66a", size = 103482, upload-time = "2026-03-09T13:15:53.382Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4d/b2/818b74ebea34dabe6d0c51cb1c572e046730e64844da6ed646d5298c40ce/kiwisolver-1.5.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:4e9750bc21b886308024f8a54ccb9a2cc38ac9fa813bf4348434e3d54f337ff9", size = 123158, upload-time = "2026-03-09T13:13:23.127Z" }, + { url = "https://files.pythonhosted.org/packages/bf/d9/405320f8077e8e1c5c4bd6adc45e1e6edf6d727b6da7f2e2533cf58bff71/kiwisolver-1.5.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:72ec46b7eba5b395e0a7b63025490d3214c11013f4aacb4f5e8d6c3041829588", size = 66388, upload-time = "2026-03-09T13:13:24.765Z" }, + { url = "https://files.pythonhosted.org/packages/99/9f/795fedf35634f746151ca8839d05681ceb6287fbed6cc1c9bf235f7887c2/kiwisolver-1.5.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ed3a984b31da7481b103f68776f7128a89ef26ed40f4dc41a2223cda7fb24819", size = 64068, upload-time = "2026-03-09T13:13:25.878Z" }, + { url = "https://files.pythonhosted.org/packages/c4/13/680c54afe3e65767bed7ec1a15571e1a2f1257128733851ade24abcefbcc/kiwisolver-1.5.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bb5136fb5352d3f422df33f0c879a1b0c204004324150cc3b5e3c4f310c9049f", size = 1477934, upload-time = "2026-03-09T13:13:27.166Z" }, + { url = "https://files.pythonhosted.org/packages/c8/2f/cebfcdb60fd6a9b0f6b47a9337198bcbad6fbe15e68189b7011fd914911f/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b2af221f268f5af85e776a73d62b0845fc8baf8ef0abfae79d29c77d0e776aaf", size = 1278537, upload-time = "2026-03-09T13:13:28.707Z" }, + { url = "https://files.pythonhosted.org/packages/f2/0d/9b782923aada3fafb1d6b84e13121954515c669b18af0c26e7d21f579855/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b0f172dc8ffaccb8522d7c5d899de00133f2f1ca7b0a49b7da98e901de87bf2d", size = 1296685, upload-time = "2026-03-09T13:13:30.528Z" }, + { url = "https://files.pythonhosted.org/packages/27/70/83241b6634b04fe44e892688d5208332bde130f38e610c0418f9ede47ded/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6ab8ba9152203feec73758dad83af9a0bbe05001eb4639e547207c40cfb52083", size = 1346024, upload-time = "2026-03-09T13:13:32.818Z" }, + { url = "https://files.pythonhosted.org/packages/e4/db/30ed226fb271ae1a6431fc0fe0edffb2efe23cadb01e798caeb9f2ceae8f/kiwisolver-1.5.0-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:cdee07c4d7f6d72008d3f73b9bf027f4e11550224c7c50d8df1ae4a37c1402a6", size = 987241, upload-time = "2026-03-09T13:13:34.435Z" }, + { url = "https://files.pythonhosted.org/packages/ec/bd/c314595208e4c9587652d50959ead9e461995389664e490f4dce7ff0f782/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7c60d3c9b06fb23bd9c6139281ccbdc384297579ae037f08ae90c69f6845c0b1", size = 2227742, upload-time = "2026-03-09T13:13:36.4Z" }, + { url = "https://files.pythonhosted.org/packages/c1/43/0499cec932d935229b5543d073c2b87c9c22846aab48881e9d8d6e742a2d/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:e315e5ec90d88e140f57696ff85b484ff68bb311e36f2c414aa4286293e6dee0", size = 2323966, upload-time = "2026-03-09T13:13:38.204Z" }, + { url = "https://files.pythonhosted.org/packages/3d/6f/79b0d760907965acfd9d61826a3d41f8f093c538f55cd2633d3f0db269f6/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:1465387ac63576c3e125e5337a6892b9e99e0627d52317f3ca79e6930d889d15", size = 1977417, upload-time = "2026-03-09T13:13:39.966Z" }, + { url = "https://files.pythonhosted.org/packages/ab/31/01d0537c41cb75a551a438c3c7a80d0c60d60b81f694dac83dd436aec0d0/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:530a3fd64c87cffa844d4b6b9768774763d9caa299e9b75d8eca6a4423b31314", size = 2491238, upload-time = "2026-03-09T13:13:41.698Z" }, + { url = "https://files.pythonhosted.org/packages/e4/34/8aefdd0be9cfd00a44509251ba864f5caf2991e36772e61c408007e7f417/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:1d9daea4ea6b9be74fe2f01f7fbade8d6ffab263e781274cffca0dba9be9eec9", size = 2294947, upload-time = "2026-03-09T13:13:43.343Z" }, + { url = "https://files.pythonhosted.org/packages/ad/cf/0348374369ca588f8fe9c338fae49fa4e16eeb10ffb3d012f23a54578a9e/kiwisolver-1.5.0-cp312-cp312-win_amd64.whl", hash = "sha256:f18c2d9782259a6dc132fdc7a63c168cbc74b35284b6d75c673958982a378384", size = 73569, upload-time = "2026-03-09T13:13:45.792Z" }, + { url = "https://files.pythonhosted.org/packages/28/26/192b26196e2316e2bd29deef67e37cdf9870d9af8e085e521afff0fed526/kiwisolver-1.5.0-cp312-cp312-win_arm64.whl", hash = "sha256:f7c7553b13f69c1b29a5bde08ddc6d9d0c8bfb84f9ed01c30db25944aeb852a7", size = 64997, upload-time = "2026-03-09T13:13:46.878Z" }, + { url = "https://files.pythonhosted.org/packages/1c/fa/2910df836372d8761bb6eff7d8bdcb1613b5c2e03f260efe7abe34d388a7/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-macosx_10_13_x86_64.whl", hash = "sha256:5ae8e62c147495b01a0f4765c878e9bfdf843412446a247e28df59936e99e797", size = 130262, upload-time = "2026-03-09T13:15:35.629Z" }, + { url = "https://files.pythonhosted.org/packages/0f/41/c5f71f9f00aabcc71fee8b7475e3f64747282580c2fe748961ba29b18385/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:f6764a4ccab3078db14a632420930f6186058750df066b8ea2a7106df91d3203", size = 138036, upload-time = "2026-03-09T13:15:36.894Z" }, + { url = "https://files.pythonhosted.org/packages/fa/06/7399a607f434119c6e1fdc8ec89a8d51ccccadf3341dee4ead6bd14caaf5/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c31c13da98624f957b0fb1b5bae5383b2333c2c3f6793d9825dd5ce79b525cb7", size = 194295, upload-time = "2026-03-09T13:15:38.22Z" }, + { url = "https://files.pythonhosted.org/packages/b5/91/53255615acd2a1eaca307ede3c90eb550bae9c94581f8c00081b6b1c8f44/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-win_amd64.whl", hash = "sha256:1f1489f769582498610e015a8ef2d36f28f505ab3096d0e16b4858a9ec214f57", size = 75987, upload-time = "2026-03-09T13:15:39.65Z" }, +] + +[[package]] +name = "lark" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/da/34/28fff3ab31ccff1fd4f6c7c7b0ceb2b6968d8ea4950663eadcb5720591a0/lark-1.3.1.tar.gz", hash = "sha256:b426a7a6d6d53189d318f2b6236ab5d6429eaf09259f1ca33eb716eed10d2905", size = 382732, upload-time = "2025-10-27T18:25:56.653Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/3d/14ce75ef66813643812f3093ab17e46d3a206942ce7376d31ec2d36229e7/lark-1.3.1-py3-none-any.whl", hash = "sha256:c629b661023a014c37da873b4ff58a817398d12635d3bbb2c5a03be7fe5d1e12", size = 113151, upload-time = "2025-10-27T18:25:54.882Z" }, +] + +[[package]] +name = "laspy" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9c/5d/5a540ac25dcc57fa77691363bb753363b192a3759aefef4b40e1250ddb17/laspy-2.7.0.tar.gz", hash = "sha256:f56feb5445e75d6ff12ee814aab6a35339290e75264ff277c6bf553f3025a3f5", size = 1929624, upload-time = "2026-01-14T22:18:42.856Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ed/fb/ae27ebca327117e35cc717a3a87afdcbbf10d5aac52e405d16ebe777357e/laspy-2.7.0-py3-none-any.whl", hash = "sha256:15f5344c62a1023461996bdf5d1ba5fdd813e96694a524fee712931134f3792f", size = 86056, upload-time = "2026-01-14T22:18:40.779Z" }, +] + +[package.optional-dependencies] +laszip = [ + { name = "laszip" }, +] + +[[package]] +name = "laszip" +version = "0.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3b/ee/947e8e869dca0a496aabe78a6f6780782449674b16d72eb92b95c0429faa/laszip-0.3.0.tar.gz", hash = "sha256:9ddf91649b519db308f63d772a19e7647458f5d5512e3d927629f0196b9c6216", size = 297312, upload-time = "2025-12-31T11:01:01.756Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fa/2f/ee0290af0a18ab6cd30f37fdedf94e01f34b35e77ae456c638e2433e7572/laszip-0.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:be69ed8513bb8d1ef83523ce6842cbf1f2c219038da6e23aca8e84b03a0d84b0", size = 265129, upload-time = "2025-12-31T10:21:11.144Z" }, + { url = "https://files.pythonhosted.org/packages/f1/8a/e9620466f6b0b4aea044c291720f177a23ae6a3db314c10b90f6362d8961/laszip-0.3.0-cp312-cp312-macosx_11_0_x86_64.whl", hash = "sha256:be3bf0f1f0e778d02959c961898d77e9581a5bf6593b9a5a6117017fd9e44150", size = 289525, upload-time = "2025-12-31T10:21:14.166Z" }, + { url = "https://files.pythonhosted.org/packages/c9/c2/f4f1460dc3f4101014eae47c86579a9c9fb00689119a34cf75e0eb288c63/laszip-0.3.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bae9fc6e4d1f35f66320b1c87335709171c679f05ccb182472c05e7c8848c79f", size = 410043, upload-time = "2025-12-31T10:21:16.079Z" }, + { url = "https://files.pythonhosted.org/packages/84/a7/7e6f146fb3da6a59d5c83d6d448f2d1c4378cab392f6ce4bd63c9bbdd99e/laszip-0.3.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dd9dde83b24d6f51ff07d6c966d2e8eec5c99087aad5fddb50d9d759b0ae75d9", size = 449738, upload-time = "2025-12-31T10:21:17.813Z" }, + { url = "https://files.pythonhosted.org/packages/c2/84/9fad7af734811fc84585417d8c48f57930f63ca021676f4472d864401c10/laszip-0.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:293e58b72fd797eba2d1101c35eb1f057d2741fb7740547d4c194795dbe8733e", size = 1332362, upload-time = "2025-12-31T10:21:20.478Z" }, + { url = "https://files.pythonhosted.org/packages/19/69/cc568b9353e1d93c4b73720afffeaf945f3a8b8a5c1e3f6d82bbae1ba00a/laszip-0.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bb278ea80a24ada6723f0830085a51551705751f74148de190cf9e74250ed9ef", size = 1414323, upload-time = "2025-12-31T10:21:23.286Z" }, + { url = "https://files.pythonhosted.org/packages/29/3d/de07b8095535523ad2b3dadfe014699292be852ab12c6f2e4947c4dd0b9b/laszip-0.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:21c53f1505d1833acd547630fdef274a9b599c39e1f9cd9d7bc800679d80e47b", size = 245800, upload-time = "2025-12-31T10:21:25.213Z" }, +] + +[[package]] +name = "markdown-it-py" +version = "4.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" }, +] + +[[package]] +name = "markupsafe" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/72/147da192e38635ada20e0a2e1a51cf8823d2119ce8883f7053879c2199b5/markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e", size = 11615, upload-time = "2025-09-27T18:36:30.854Z" }, + { url = "https://files.pythonhosted.org/packages/9a/81/7e4e08678a1f98521201c3079f77db69fb552acd56067661f8c2f534a718/markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce", size = 12020, upload-time = "2025-09-27T18:36:31.971Z" }, + { url = "https://files.pythonhosted.org/packages/1e/2c/799f4742efc39633a1b54a92eec4082e4f815314869865d876824c257c1e/markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d", size = 24332, upload-time = "2025-09-27T18:36:32.813Z" }, + { url = "https://files.pythonhosted.org/packages/3c/2e/8d0c2ab90a8c1d9a24f0399058ab8519a3279d1bd4289511d74e909f060e/markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d", size = 22947, upload-time = "2025-09-27T18:36:33.86Z" }, + { url = "https://files.pythonhosted.org/packages/2c/54/887f3092a85238093a0b2154bd629c89444f395618842e8b0c41783898ea/markupsafe-3.0.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a", size = 21962, upload-time = "2025-09-27T18:36:35.099Z" }, + { url = "https://files.pythonhosted.org/packages/c9/2f/336b8c7b6f4a4d95e91119dc8521402461b74a485558d8f238a68312f11c/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b", size = 23760, upload-time = "2025-09-27T18:36:36.001Z" }, + { url = "https://files.pythonhosted.org/packages/32/43/67935f2b7e4982ffb50a4d169b724d74b62a3964bc1a9a527f5ac4f1ee2b/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f", size = 21529, upload-time = "2025-09-27T18:36:36.906Z" }, + { url = "https://files.pythonhosted.org/packages/89/e0/4486f11e51bbba8b0c041098859e869e304d1c261e59244baa3d295d47b7/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b", size = 23015, upload-time = "2025-09-27T18:36:37.868Z" }, + { url = "https://files.pythonhosted.org/packages/2f/e1/78ee7a023dac597a5825441ebd17170785a9dab23de95d2c7508ade94e0e/markupsafe-3.0.3-cp312-cp312-win32.whl", hash = "sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d", size = 14540, upload-time = "2025-09-27T18:36:38.761Z" }, + { url = "https://files.pythonhosted.org/packages/aa/5b/bec5aa9bbbb2c946ca2733ef9c4ca91c91b6a24580193e891b5f7dbe8e1e/markupsafe-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c", size = 15105, upload-time = "2025-09-27T18:36:39.701Z" }, + { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" }, +] + +[[package]] +name = "matplotlib" +version = "3.11.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "contourpy" }, + { name = "cycler" }, + { name = "fonttools" }, + { name = "kiwisolver" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pillow" }, + { name = "pyparsing" }, + { name = "python-dateutil" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/49/64/f9a391af28f518b11ad45a8a712353c94a0aefce09d3703200e5c54b610a/matplotlib-3.11.1.tar.gz", hash = "sha256:69647db5746941c793d6e445a4cd349323ffb87d9cc958c2ad84a659b4832d30", size = 32612045, upload-time = "2026-07-18T03:39:46.63Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f2/6c/7ef7ebcb2bd9739b2b66b18b076e077f44bb46fdbe28ca0506edb3c62c79/matplotlib-3.11.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:e15ef41507f3d525f46154ac9e3ae785dacde9f20e593a25de8986267892ef74", size = 9453849, upload-time = "2026-07-18T03:38:19.593Z" }, + { url = "https://files.pythonhosted.org/packages/eb/f8/6d0c312c8d9738e7d9677f09fe5c986b3239e651a7b73a2deb38b65e4a71/matplotlib-3.11.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:21a67b961a6d597bca54fae826cd20695ba4a6e4d05424a08da6e13e3176fd6b", size = 9283113, upload-time = "2026-07-18T03:38:21.95Z" }, + { url = "https://files.pythonhosted.org/packages/c9/cf/b4ad2cc81b6672ea29ea04e64e350a9f9b493b0908ccd884c67eeff8f7b2/matplotlib-3.11.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ba8f811b8ddfac493734d6af0b2dff96919d0c28ca0d641858dab4262777c6ea", size = 10035615, upload-time = "2026-07-18T03:38:24.315Z" }, + { url = "https://files.pythonhosted.org/packages/88/90/4e10e033d9b66589d8ed98b84c95cdbb57033d57c1f41339d7393dbd2f2e/matplotlib-3.11.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c52f7ad20ef476806ed212380b1d54d20310c8b86bdc2c9a68b51f0024a44472", size = 10842559, upload-time = "2026-07-18T03:38:26.285Z" }, + { url = "https://files.pythonhosted.org/packages/88/eb/799612d0f8cd3e816a10fec59329fca52cd2353264df80378dfc541ae855/matplotlib-3.11.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:8b14eb22961fe865efb0e4ff167e333e428908b00115a8d800ccb65ee108e481", size = 10927532, upload-time = "2026-07-18T03:38:28.532Z" }, + { url = "https://files.pythonhosted.org/packages/88/89/56649bbaa2fd12e20f3be03dbcc135b0c8676d88bac17977599e3eb442a0/matplotlib-3.11.1-cp312-cp312-win_amd64.whl", hash = "sha256:88a2a27dd9691ae448dfae4b26f59036be90c3c28757edd3553a29559d00859f", size = 9333886, upload-time = "2026-07-18T03:38:30.477Z" }, + { url = "https://files.pythonhosted.org/packages/c1/11/4d124efbbad677b7b7552f6f85a3bd432d4232f95400cea98fcd2ae36ef3/matplotlib-3.11.1-cp312-cp312-win_arm64.whl", hash = "sha256:480194afceca4df2f137c2721227d3cba67121fbf4397b69cee7f83714b0a58a", size = 9007545, upload-time = "2026-07-18T03:38:32.833Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "mpmath" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e0/47/dd32fa426cc72114383ac549964eecb20ecfd886d1e5ccf5340b55b02f57/mpmath-1.3.0.tar.gz", hash = "sha256:7a28eb2a9774d00c7bc92411c19a89209d5da7c4c9a9e227be8330a23a25b91f", size = 508106, upload-time = "2023-03-07T16:47:11.061Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198, upload-time = "2023-03-07T16:47:09.197Z" }, +] + +[[package]] +name = "narwhals" +version = "2.24.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2b/1d/58946e5aab18393e793bd4add6985b95d0e01c3a2d832f38f54468b10dcd/narwhals-2.24.0.tar.gz", hash = "sha256:b5c0f684ccd9d7475b564111e319a4964abcf2baf79d3cf6b1003d06ac9b828d", size = 661143, upload-time = "2026-07-13T10:49:19.086Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/85/a5bfaebfd305ac18b57b0854d74e37e586809061a91fda62f0bd50c8518e/narwhals-2.24.0-py3-none-any.whl", hash = "sha256:42fdedf44e5b2ca7505630d45b4ac3058f38d8485cba9fe1652ca23152df7489", size = 461030, upload-time = "2026-07-13T10:49:17.571Z" }, +] + +[[package]] +name = "nbformat" +version = "5.11.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "fastjsonschema" }, + { name = "jsonschema" }, + { name = "jupyter-core" }, + { name = "traitlets" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b2/fd/80f407a9525bc5bd9865e5c37db3b78867fa43217f8aac5eab22b5f028b3/nbformat-5.11.0.tar.gz", hash = "sha256:7dbaed4a69cae28c2b4d44ab7430a6af4544fb89455023f6f21550be757b60c8", size = 151822, upload-time = "2026-08-06T12:29:55.597Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/be/4a/0eece9dad5e73230ca972f7bc29456ce7d74772f123d401fcf67379008f7/nbformat-5.11.0-py3-none-any.whl", hash = "sha256:f70a17f591a9ccd1c601d5e61a4b20972703926df0ba42458ce14bf575766bb6", size = 79820, upload-time = "2026-08-06T12:29:54.178Z" }, +] + +[[package]] +name = "nest-asyncio" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/83/f8/51569ac65d696c8ecbee95938f89d4abf00f47d58d48f6fbabfe8f0baefe/nest_asyncio-1.6.0.tar.gz", hash = "sha256:6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe", size = 7418, upload-time = "2024-01-21T14:25:19.227Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/c4/c2971a3ba4c6103a3d10c4b0f24f461ddc027f0f09763220cf35ca1401b3/nest_asyncio-1.6.0-py3-none-any.whl", hash = "sha256:87af6efd6b5e897c81050477ef65c62e2b2f35d51703cae01aff2905b1852e1c", size = 5195, upload-time = "2024-01-21T14:25:17.223Z" }, +] + +[[package]] +name = "networkx" +version = "3.6.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, +] + +[[package]] +name = "ninja" +version = "1.13.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/43/73/79a0b22fc731989c708068427579e840a6cf4e937fe7ae5c5d0b7356ac22/ninja-1.13.0.tar.gz", hash = "sha256:4a40ce995ded54d9dc24f8ea37ff3bf62ad192b547f6c7126e7e25045e76f978", size = 242558, upload-time = "2025-08-11T15:10:19.421Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3c/74/d02409ed2aa865e051b7edda22ad416a39d81a84980f544f8de717cab133/ninja-1.13.0-py3-none-macosx_10_9_universal2.whl", hash = "sha256:fa2a8bfc62e31b08f83127d1613d10821775a0eb334197154c4d6067b7068ff1", size = 310125, upload-time = "2025-08-11T15:09:50.971Z" }, + { url = "https://files.pythonhosted.org/packages/8e/de/6e1cd6b84b412ac1ef327b76f0641aeb5dcc01e9d3f9eee0286d0c34fd93/ninja-1.13.0-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3d00c692fb717fd511abeb44b8c5d00340c36938c12d6538ba989fe764e79630", size = 177467, upload-time = "2025-08-11T15:09:52.767Z" }, + { url = "https://files.pythonhosted.org/packages/c8/83/49320fb6e58ae3c079381e333575fdbcf1cca3506ee160a2dcce775046fa/ninja-1.13.0-py3-none-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:be7f478ff9f96a128b599a964fc60a6a87b9fa332ee1bd44fa243ac88d50291c", size = 187834, upload-time = "2025-08-11T15:09:54.115Z" }, + { url = "https://files.pythonhosted.org/packages/56/c7/ba22748fb59f7f896b609cd3e568d28a0a367a6d953c24c461fe04fc4433/ninja-1.13.0-py3-none-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:60056592cf495e9a6a4bea3cd178903056ecb0943e4de45a2ea825edb6dc8d3e", size = 202736, upload-time = "2025-08-11T15:09:55.745Z" }, + { url = "https://files.pythonhosted.org/packages/79/22/d1de07632b78ac8e6b785f41fa9aad7a978ec8c0a1bf15772def36d77aac/ninja-1.13.0-py3-none-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:1c97223cdda0417f414bf864cfb73b72d8777e57ebb279c5f6de368de0062988", size = 179034, upload-time = "2025-08-11T15:09:57.394Z" }, + { url = "https://files.pythonhosted.org/packages/ed/de/0e6edf44d6a04dabd0318a519125ed0415ce437ad5a1ec9b9be03d9048cf/ninja-1.13.0-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fb46acf6b93b8dd0322adc3a4945452a4e774b75b91293bafcc7b7f8e6517dfa", size = 180716, upload-time = "2025-08-11T15:09:58.696Z" }, + { url = "https://files.pythonhosted.org/packages/54/28/938b562f9057aaa4d6bfbeaa05e81899a47aebb3ba6751e36c027a7f5ff7/ninja-1.13.0-py3-none-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4be9c1b082d244b1ad7ef41eb8ab088aae8c109a9f3f0b3e56a252d3e00f42c1", size = 146843, upload-time = "2025-08-11T15:10:00.046Z" }, + { url = "https://files.pythonhosted.org/packages/2a/fb/d06a3838de4f8ab866e44ee52a797b5491df823901c54943b2adb0389fbb/ninja-1.13.0-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:6739d3352073341ad284246f81339a384eec091d9851a886dfa5b00a6d48b3e2", size = 154402, upload-time = "2025-08-11T15:10:01.657Z" }, + { url = "https://files.pythonhosted.org/packages/31/bf/0d7808af695ceddc763cf251b84a9892cd7f51622dc8b4c89d5012779f06/ninja-1.13.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:11be2d22027bde06f14c343f01d31446747dbb51e72d00decca2eb99be911e2f", size = 552388, upload-time = "2025-08-11T15:10:03.349Z" }, + { url = "https://files.pythonhosted.org/packages/9d/70/c99d0c2c809f992752453cce312848abb3b1607e56d4cd1b6cded317351a/ninja-1.13.0-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:aa45b4037b313c2f698bc13306239b8b93b4680eb47e287773156ac9e9304714", size = 472501, upload-time = "2025-08-11T15:10:04.735Z" }, + { url = "https://files.pythonhosted.org/packages/9f/43/c217b1153f0e499652f5e0766da8523ce3480f0a951039c7af115e224d55/ninja-1.13.0-py3-none-musllinux_1_2_i686.whl", hash = "sha256:5f8e1e8a1a30835eeb51db05cf5a67151ad37542f5a4af2a438e9490915e5b72", size = 638280, upload-time = "2025-08-11T15:10:06.512Z" }, + { url = "https://files.pythonhosted.org/packages/8c/45/9151bba2c8d0ae2b6260f71696330590de5850e5574b7b5694dce6023e20/ninja-1.13.0-py3-none-musllinux_1_2_ppc64le.whl", hash = "sha256:3d7d7779d12cb20c6d054c61b702139fd23a7a964ec8f2c823f1ab1b084150db", size = 642420, upload-time = "2025-08-11T15:10:08.35Z" }, + { url = "https://files.pythonhosted.org/packages/3c/fb/95752eb635bb8ad27d101d71bef15bc63049de23f299e312878fc21cb2da/ninja-1.13.0-py3-none-musllinux_1_2_riscv64.whl", hash = "sha256:d741a5e6754e0bda767e3274a0f0deeef4807f1fec6c0d7921a0244018926ae5", size = 585106, upload-time = "2025-08-11T15:10:09.818Z" }, + { url = "https://files.pythonhosted.org/packages/c1/31/aa56a1a286703800c0cbe39fb4e82811c277772dc8cd084f442dd8e2938a/ninja-1.13.0-py3-none-musllinux_1_2_s390x.whl", hash = "sha256:e8bad11f8a00b64137e9b315b137d8bb6cbf3086fbdc43bf1f90fd33324d2e96", size = 707138, upload-time = "2025-08-11T15:10:11.366Z" }, + { url = "https://files.pythonhosted.org/packages/34/6f/5f5a54a1041af945130abdb2b8529cbef0cdcbbf9bcf3f4195378319d29a/ninja-1.13.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:b4f2a072db3c0f944c32793e91532d8948d20d9ab83da9c0c7c15b5768072200", size = 581758, upload-time = "2025-08-11T15:10:13.295Z" }, + { url = "https://files.pythonhosted.org/packages/95/97/51359c77527d45943fe7a94d00a3843b81162e6c4244b3579fe8fc54cb9c/ninja-1.13.0-py3-none-win32.whl", hash = "sha256:8cfbb80b4a53456ae8a39f90ae3d7a2129f45ea164f43fadfa15dc38c4aef1c9", size = 267201, upload-time = "2025-08-11T15:10:15.158Z" }, + { url = "https://files.pythonhosted.org/packages/29/45/c0adfbfb0b5895aa18cec400c535b4f7ff3e52536e0403602fc1a23f7de9/ninja-1.13.0-py3-none-win_amd64.whl", hash = "sha256:fb8ee8719f8af47fed145cced4a85f0755dd55d45b2bddaf7431fa89803c5f3e", size = 309975, upload-time = "2025-08-11T15:10:16.697Z" }, + { url = "https://files.pythonhosted.org/packages/df/93/a7b983643d1253bb223234b5b226e69de6cda02b76cdca7770f684b795f5/ninja-1.13.0-py3-none-win_arm64.whl", hash = "sha256:3c0b40b1f0bba764644385319028650087b4c1b18cdfa6f45cb39a3669b81aa9", size = 290806, upload-time = "2025-08-11T15:10:18.018Z" }, +] + +[[package]] +name = "numpy" +version = "2.5.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/80/db0b4559e57ec36362bedbb05530a87fafbcb6067708c946967a41d449e7/numpy-2.5.2.tar.gz", hash = "sha256:d482d171c406ae88c5b19cad3b6a1c4c5209f886ab74bc44c2c865c23f52d860", size = 20773161, upload-time = "2026-08-09T13:48:27.962Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/69/72/dccb0aaf40972777283303919f613964227266d0c13adebb79ac124f1c3e/numpy-2.5.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:14e373cfc6387177e8409dac3c7159be8eb05cd77096cd7c950268b86f62831c", size = 16891693, upload-time = "2026-08-09T13:44:51.702Z" }, + { url = "https://files.pythonhosted.org/packages/60/2e/b5aee50a1f74ac815cf8331812cb8251e29024025de462e0c047641c614c/numpy-2.5.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4bbd96c833ecc8cc069ce518078fc8c60cb9cbfb0fea5b7a803ad65035596d03", size = 11903109, upload-time = "2026-08-09T13:44:55.501Z" }, + { url = "https://files.pythonhosted.org/packages/f3/f4/29e78102a80601cf034d4e9767022cffeca2c3b4c926e1754572ca95593d/numpy-2.5.2-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:6e8172ddfcf5cf74b811d372b570b83c60bd2de87a6fbfbebdadb4a9bd9c6cbb", size = 5350202, upload-time = "2026-08-09T13:44:58.401Z" }, + { url = "https://files.pythonhosted.org/packages/11/4b/dcd3b7eadaf4035d2c7a4289d232523a6964f602598ef7674e4bd7291f93/numpy-2.5.2-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:65f188481f1669e26f62b701e8205d19e460fa4a9b52a1414ba382330e4a3414", size = 6687736, upload-time = "2026-08-09T13:45:00.813Z" }, + { url = "https://files.pythonhosted.org/packages/e5/21/4947e0e9d6c9fc2e2ff15b8949049ee44f63adb9cacc729ab8793f97e712/numpy-2.5.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8ee9c4eeb8454b3660a8b53493563c3e121c2fc94fbd72b848ef814ed7b676a9", size = 15612696, upload-time = "2026-08-09T13:45:04.151Z" }, + { url = "https://files.pythonhosted.org/packages/3a/5f/62d28cf019460c7f1394105b4d49d9911a9c444cb77ab0bd95a204c5a6de/numpy-2.5.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3cdec01fa790a186d430433fdd4d4ffb70eed6f0eeb4bf05c8dbe2dce0a9bcb8", size = 16722264, upload-time = "2026-08-09T13:45:07.714Z" }, + { url = "https://files.pythonhosted.org/packages/14/25/3f0be4c1b9fdf5dd5e708a6806978564d7c46a055c000496309ff2a2f8af/numpy-2.5.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7999d4ddb0c4025018373fd787510d46e04c769467af22869707b3c1cfd459ab", size = 16974396, upload-time = "2026-08-09T13:45:11.316Z" }, + { url = "https://files.pythonhosted.org/packages/22/72/6262cbdeeb45da9d971e40715f579d791603ba8ec0b5e2db1ac55454421d/numpy-2.5.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:c1f017dc0875c9209d219f97feceb7d54c2661bb243deb4114478e1295808af7", size = 18476044, upload-time = "2026-08-09T13:45:14.869Z" }, + { url = "https://files.pythonhosted.org/packages/36/33/29208b8b075bde62d26a81d14b358c42b0f69b6cabd98d4ff97f37f22b05/numpy-2.5.2-cp312-cp312-win32.whl", hash = "sha256:d6a48072864e3324e194a8fbb3c657bcc5b5c869dbc64c9537b1d5c862572c0a", size = 6072817, upload-time = "2026-08-09T13:45:17.867Z" }, + { url = "https://files.pythonhosted.org/packages/7f/b9/87fea2769fe1c47c1b5b01d8310772c9d1a85d485de7cf386ef7a3332b02/numpy-2.5.2-cp312-cp312-win_amd64.whl", hash = "sha256:28ac63476ec7651484215ee7fa15a1f78b57c14621f01e392afe17b9a1390ce4", size = 12464674, upload-time = "2026-08-09T13:45:20.734Z" }, + { url = "https://files.pythonhosted.org/packages/14/52/032b97e00461ab0809bbe4c588b035620e5a14b8cdee47ecddefc7b17d33/numpy-2.5.2-cp312-cp312-win_arm64.whl", hash = "sha256:27650bb0e7140fa3d37b9923b4803645e0b125d190f326eecfd3f4dad8e8ade1", size = 10397131, upload-time = "2026-08-09T13:45:23.73Z" }, +] + +[[package]] +name = "nvidia-cublas-cu12" +version = "12.6.4.1" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/af/eb/ff4b8c503fa1f1796679dce648854d58751982426e4e4b37d6fce49d259c/nvidia_cublas_cu12-12.6.4.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:08ed2686e9875d01b58e3cb379c6896df8e76c75e0d4a7f7dace3d7b6d9ef8eb", size = 393138322, upload-time = "2024-11-20T17:40:25.65Z" }, +] + +[[package]] +name = "nvidia-cuda-cupti-cu12" +version = "12.6.80" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/60/7b6497946d74bcf1de852a21824d63baad12cd417db4195fc1bfe59db953/nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6768bad6cab4f19e8292125e5f1ac8aa7d1718704012a0e3272a6f61c4bce132", size = 8917980, upload-time = "2024-11-20T17:36:04.019Z" }, + { url = "https://files.pythonhosted.org/packages/a5/24/120ee57b218d9952c379d1e026c4479c9ece9997a4fb46303611ee48f038/nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_x86_64.whl", hash = "sha256:a3eff6cdfcc6a4c35db968a06fcadb061cbc7d6dde548609a941ff8701b98b73", size = 8917972, upload-time = "2024-10-01T16:58:06.036Z" }, +] + +[[package]] +name = "nvidia-cuda-nvrtc-cu12" +version = "12.6.77" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/75/2e/46030320b5a80661e88039f59060d1790298b4718944a65a7f2aeda3d9e9/nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl", hash = "sha256:35b0cc6ee3a9636d5409133e79273ce1f3fd087abb0532d2d2e8fff1fe9efc53", size = 23650380, upload-time = "2024-10-01T17:00:14.643Z" }, +] + +[[package]] +name = "nvidia-cuda-runtime-cu12" +version = "12.6.77" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e1/23/e717c5ac26d26cf39a27fbc076240fad2e3b817e5889d671b67f4f9f49c5/nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ba3b56a4f896141e25e19ab287cd71e52a6a0f4b29d0d31609f60e3b4d5219b7", size = 897690, upload-time = "2024-11-20T17:35:30.697Z" }, + { url = "https://files.pythonhosted.org/packages/f0/62/65c05e161eeddbafeca24dc461f47de550d9fa8a7e04eb213e32b55cfd99/nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl", hash = "sha256:a84d15d5e1da416dd4774cb42edf5e954a3e60cc945698dc1d5be02321c44dc8", size = 897678, upload-time = "2024-10-01T16:57:33.821Z" }, +] + +[[package]] +name = "nvidia-cudnn-cu12" +version = "9.5.1.17" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cublas-cu12" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/78/4535c9c7f859a64781e43c969a3a7e84c54634e319a996d43ef32ce46f83/nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:30ac3869f6db17d170e0e556dd6cc5eee02647abc31ca856634d5a40f82c15b2", size = 570988386, upload-time = "2024-10-25T19:54:26.39Z" }, +] + +[[package]] +name = "nvidia-cufft-cu12" +version = "11.3.0.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-nvjitlink-cu12" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/8f/16/73727675941ab8e6ffd86ca3a4b7b47065edcca7a997920b831f8147c99d/nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ccba62eb9cef5559abd5e0d54ceed2d9934030f51163df018532142a8ec533e5", size = 200221632, upload-time = "2024-11-20T17:41:32.357Z" }, + { url = "https://files.pythonhosted.org/packages/60/de/99ec247a07ea40c969d904fc14f3a356b3e2a704121675b75c366b694ee1/nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.whl", hash = "sha256:768160ac89f6f7b459bee747e8d175dbf53619cfe74b2a5636264163138013ca", size = 200221622, upload-time = "2024-10-01T17:03:58.79Z" }, +] + +[[package]] +name = "nvidia-cufile-cu12" +version = "1.11.1.6" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b2/66/cc9876340ac68ae71b15c743ddb13f8b30d5244af344ec8322b449e35426/nvidia_cufile_cu12-1.11.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cc23469d1c7e52ce6c1d55253273d32c565dd22068647f3aa59b3c6b005bf159", size = 1142103, upload-time = "2024-11-20T17:42:11.83Z" }, +] + +[[package]] +name = "nvidia-curand-cu12" +version = "10.3.7.77" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/73/1b/44a01c4e70933637c93e6e1a8063d1e998b50213a6b65ac5a9169c47e98e/nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a42cd1344297f70b9e39a1e4f467a4e1c10f1da54ff7a85c12197f6c652c8bdf", size = 56279010, upload-time = "2024-11-20T17:42:50.958Z" }, + { url = "https://files.pythonhosted.org/packages/4a/aa/2c7ff0b5ee02eaef890c0ce7d4f74bc30901871c5e45dee1ae6d0083cd80/nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_x86_64.whl", hash = "sha256:99f1a32f1ac2bd134897fc7a203f779303261268a65762a623bf30cc9fe79117", size = 56279000, upload-time = "2024-10-01T17:04:45.274Z" }, +] + +[[package]] +name = "nvidia-cusolver-cu12" +version = "11.7.1.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cublas-cu12" }, + { name = "nvidia-cusparse-cu12" }, + { name = "nvidia-nvjitlink-cu12" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/f0/6e/c2cf12c9ff8b872e92b4a5740701e51ff17689c4d726fca91875b07f655d/nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e9e49843a7707e42022babb9bcfa33c29857a93b88020c4e4434656a655b698c", size = 158229790, upload-time = "2024-11-20T17:43:43.211Z" }, + { url = "https://files.pythonhosted.org/packages/9f/81/baba53585da791d043c10084cf9553e074548408e04ae884cfe9193bd484/nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.whl", hash = "sha256:6cf28f17f64107a0c4d7802be5ff5537b2130bfc112f25d5a30df227058ca0e6", size = 158229780, upload-time = "2024-10-01T17:05:39.875Z" }, +] + +[[package]] +name = "nvidia-cusparse-cu12" +version = "12.5.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-nvjitlink-cu12" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/06/1e/b8b7c2f4099a37b96af5c9bb158632ea9e5d9d27d7391d7eb8fc45236674/nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7556d9eca156e18184b94947ade0fba5bb47d69cec46bf8660fd2c71a4b48b73", size = 216561367, upload-time = "2024-11-20T17:44:54.824Z" }, + { url = "https://files.pythonhosted.org/packages/43/ac/64c4316ba163e8217a99680c7605f779accffc6a4bcd0c778c12948d3707/nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.whl", hash = "sha256:23749a6571191a215cb74d1cdbff4a86e7b19f1200c071b3fcf844a5bea23a2f", size = 216561357, upload-time = "2024-10-01T17:06:29.861Z" }, +] + +[[package]] +name = "nvidia-cusparselt-cu12" +version = "0.6.3" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3b/9a/72ef35b399b0e183bc2e8f6f558036922d453c4d8237dab26c666a04244b/nvidia_cusparselt_cu12-0.6.3-py3-none-manylinux2014_x86_64.whl", hash = "sha256:e5c8a26c36445dd2e6812f1177978a24e2d37cacce7e090f297a688d1ec44f46", size = 156785796, upload-time = "2024-10-15T21:29:17.709Z" }, +] + +[[package]] +name = "nvidia-nccl-cu12" +version = "2.26.2" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/67/ca/f42388aed0fddd64ade7493dbba36e1f534d4e6fdbdd355c6a90030ae028/nvidia_nccl_cu12-2.26.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:694cf3879a206553cc9d7dbda76b13efaf610fdb70a50cba303de1b0d1530ac6", size = 201319755, upload-time = "2025-03-13T00:29:55.296Z" }, +] + +[[package]] +name = "nvidia-nvjitlink-cu12" +version = "12.6.85" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9d/d7/c5383e47c7e9bf1c99d5bd2a8c935af2b6d705ad831a7ec5c97db4d82f4f/nvidia_nvjitlink_cu12-12.6.85-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:eedc36df9e88b682efe4309aa16b5b4e78c2407eac59e8c10a6a47535164369a", size = 19744971, upload-time = "2024-11-20T17:46:53.366Z" }, +] + +[[package]] +name = "nvidia-nvtx-cu12" +version = "12.6.77" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/9a/fff8376f8e3d084cd1530e1ef7b879bb7d6d265620c95c1b322725c694f4/nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b90bed3df379fa79afbd21be8e04a0314336b8ae16768b58f2d34cb1d04cd7d2", size = 89276, upload-time = "2024-11-20T17:38:27.621Z" }, + { url = "https://files.pythonhosted.org/packages/9e/4e/0d0c945463719429b7bd21dece907ad0bde437a2ff12b9b12fee94722ab0/nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl", hash = "sha256:6574241a3ec5fdc9334353ab8c479fe75841dbe8f4532a8fc97ce63503330ba1", size = 89265, upload-time = "2024-10-01T17:00:38.172Z" }, +] + +[[package]] +name = "open3d" +version = "0.19.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "addict" }, + { name = "configargparse" }, + { name = "dash" }, + { name = "flask" }, + { name = "matplotlib" }, + { name = "nbformat" }, + { name = "numpy" }, + { name = "pandas" }, + { name = "pillow" }, + { name = "pyquaternion" }, + { name = "pyyaml" }, + { name = "scikit-learn" }, + { name = "tqdm" }, + { name = "werkzeug" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/37/c5/286c605e087e72ad83eab130451ce13b768caa4374d926dc735edc20da5a/open3d-0.19.0-cp312-cp312-macosx_10_15_universal2.whl", hash = "sha256:9e4a8d29443ba4c83010d199d56c96bf553dd970d3351692ab271759cbe2d7ac", size = 103202754, upload-time = "2025-01-08T07:26:27.169Z" }, + { url = "https://files.pythonhosted.org/packages/2b/95/3723e5ade77c234a1650db11cbe59fe25c4f5af6c224f8ea22ff088bb36a/open3d-0.19.0-cp312-cp312-manylinux_2_31_x86_64.whl", hash = "sha256:01e4590dc2209040292ebe509542fbf2bf869ea60bcd9be7a3fe77b65bad3192", size = 447665185, upload-time = "2025-01-08T07:23:39.769Z" }, + { url = "https://files.pythonhosted.org/packages/9f/c4/35a6e0a35aa72420e75dc28d54b24beaff79bcad150423e47c67d2ad8773/open3d-0.19.0-cp312-cp312-win_amd64.whl", hash = "sha256:665839837e1d3a62524804c31031462c3b548a2b6ed55214e6deb91522844f97", size = 69169961, upload-time = "2025-01-08T07:27:35.392Z" }, +] + +[[package]] +name = "packaging" +version = "26.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412, upload-time = "2026-08-04T18:15:28.737Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" }, +] + +[[package]] +name = "pandas" +version = "3.0.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "python-dateutil" }, + { name = "tzdata", marker = "sys_platform == 'emscripten' or sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/be/4f/5f3422a2afec5ffc46308b79e53291365a93748b498ac2e58bead0197916/pandas-3.0.5.tar.gz", hash = "sha256:dca3734d6ab7c906e6730f0788b0a1dbb9f2467731f9711f77995c8e9d62d712", size = 4658219, upload-time = "2026-07-22T22:19:28.819Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1c/54/1dc810ea558d1320b597aa140a514f2fdf1d2ea09c38cf556f13ea712ec9/pandas-3.0.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fa290c16964d4963fbfbc358928239cf3bd755b20e988ce944877def2f44471d", size = 10411717, upload-time = "2026-07-22T22:18:08.307Z" }, + { url = "https://files.pythonhosted.org/packages/68/56/fbe81c09195924d8b7b8d4461a20458fe80a6a5ed6b24f0314da684277e1/pandas-3.0.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c2e26bb46934b8a2ca0c3de1d3d606fc5f6746584791b2db264d58cf370e08dc", size = 9957095, upload-time = "2026-07-22T22:18:10.6Z" }, + { url = "https://files.pythonhosted.org/packages/e0/51/fac252f4a913ed5eabf3c11b880a9e8d5a6c10f0b2129d0462212d238b4d/pandas-3.0.5-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:73fa87b08a7ef706f8aafda39ddaccf2a99047bea62d8c88a0361bcafb2237bc", size = 10485458, upload-time = "2026-07-22T22:18:12.834Z" }, + { url = "https://files.pythonhosted.org/packages/12/98/e976540c1addf70442be7842a18cf70884a964abbf69442504f4d2939989/pandas-3.0.5-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d373ce03ffd84010ed9839fa73672a9c8256990532e158440c0085db7d914b34", size = 10998091, upload-time = "2026-07-22T22:18:15.209Z" }, + { url = "https://files.pythonhosted.org/packages/a4/8c/1f29b5be8d3fc47dd7567eb167fabba2085879b31e0287ce7cba6d3d2ff4/pandas-3.0.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2a29c53d85ea98c5e792c59ef82ee9fbe6ca902c0d0adb6b23f45ef894cd7bf6", size = 11499501, upload-time = "2026-07-22T22:18:17.689Z" }, + { url = "https://files.pythonhosted.org/packages/9d/e2/bd9c98ad2df7b38bde002adde4cdf353519da51881634323b126c55997f9/pandas-3.0.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a5ad3b02ed6bc7d7ae9b70804b2c6aa31827489d150f8e623ce82491b82085d7", size = 12060559, upload-time = "2026-07-22T22:18:20.147Z" }, + { url = "https://files.pythonhosted.org/packages/f3/9a/ffbd852d58bd74a617fe2f8ee6a58a96982271ce41cf981eab22190b4a4b/pandas-3.0.5-cp312-cp312-pyemscripten_2024_0_wasm32.whl", hash = "sha256:b2acb4650527eec6822c3dadb2b771277b65e7dae7a267d4bccf65fd1bb3fbce", size = 7197652, upload-time = "2026-07-22T22:18:22.502Z" }, + { url = "https://files.pythonhosted.org/packages/70/b5/d2d3e9ae73362ba4229651b0ee1455cf78073a1ce585f6ff693782ce263e/pandas-3.0.5-cp312-cp312-win_amd64.whl", hash = "sha256:80a611068e8a3ac23f7398c6c14eb46dc974e5cc9997f653e2dcfd1da74edd41", size = 9831691, upload-time = "2026-07-22T22:18:24.534Z" }, + { url = "https://files.pythonhosted.org/packages/52/51/dea1e89d6a6796b9c43f85a09b484ee03edb8a4c4842e73e200a8c11301c/pandas-3.0.5-cp312-cp312-win_arm64.whl", hash = "sha256:25ff585b972a18ef1fe9ffa3ac6544d9950508aa76832e5147640b6022821e49", size = 9105796, upload-time = "2026-07-22T22:18:27.064Z" }, +] + +[[package]] +name = "pccm" +version = "0.4.16" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "ccimport" }, + { name = "fire" }, + { name = "lark" }, + { name = "portalocker" }, + { name = "pybind11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a7/6f/bdd551b14c333207300bf8cf6b327e68fcfc8650d46af187a73967042f70/pccm-0.4.16.tar.gz", hash = "sha256:a9987c2fc6b663eb5ec1c75430617bb7f3ab3d0724da049918689d5d7c5c8190", size = 67186, upload-time = "2024-09-06T03:34:11.09Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8a/53/39e6cd6b19e1be302c22bc6db8683968fd52f1df5ed6e38921a16d013406/pccm-0.4.16-py3-none-any.whl", hash = "sha256:97e97c341c50c40965bb8f9a8a1ad201868d64e8baa06f9581bedeb3bc223eec", size = 73731, upload-time = "2024-09-06T03:34:09.646Z" }, +] + +[[package]] +name = "peft" +version = "0.20.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "accelerate" }, + { name = "huggingface-hub" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "psutil" }, + { name = "pyyaml" }, + { name = "safetensors" }, + { name = "torch" }, + { name = "tqdm" }, + { name = "transformers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b4/08/02541a2c29be7c78698f73d438bc0e733b214f3f35ec5db79fca8da8fc61/peft-0.20.0.tar.gz", hash = "sha256:4769c8093a4ca145fd6fb3fd4dd50449675f5fe46434ad1e98b285a132d4b1d0", size = 880503, upload-time = "2026-07-28T13:46:01.85Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/28/79/13bcabb8048126422d5c4b880575d40886c726f354db88cfeed4325525bb/peft-0.20.0-py3-none-any.whl", hash = "sha256:0fbba16ffebfad3de96e06f2da6860fd860292324b85b6141909fa1e26ea9233", size = 775777, upload-time = "2026-07-28T13:45:59.809Z" }, +] + +[[package]] +name = "pillow" +version = "12.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1c/3d/bb7fca845737cf9d7dbde16ed1843984665ff2e0a518f5db43e77ec540b9/pillow-12.3.0.tar.gz", hash = "sha256:3b8182a766685eaa002637e28b4ec8d6b18819a0c71f579bf0dbaa5830297cce", size = 47025035, upload-time = "2026-07-01T11:56:38.965Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/37/bf/fb3ebff8ddcb76aac5a01389251bbbb9519922a9b520d8247c1ca864a25d/pillow-12.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ba09209fbe443b4acccebe845d8a138b89a8f4fbaeedd44953490b5315d5e965", size = 5345969, upload-time = "2026-07-01T11:54:06.397Z" }, + { url = "https://files.pythonhosted.org/packages/d8/66/9a386a92561f402389a4fc70c18838bf6d35eb5eb5c6850b4b2dc64f5048/pillow-12.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ffd0c5368496f41b0944be820fcb7a838aa6e623d250b01acf2643939c3f99d7", size = 4780323, upload-time = "2026-07-01T11:54:09.351Z" }, + { url = "https://files.pythonhosted.org/packages/25/27/ac8f99618ffd3dde21db0f4d4b1d2ab00c0880595bfd17df103f7f39fd0c/pillow-12.3.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d9c7f76c0673154f044e9d78c8655fb4213f6ca31a836df48b40fe5d187717b9", size = 6266838, upload-time = "2026-07-01T11:54:11.71Z" }, + { url = "https://files.pythonhosted.org/packages/84/21/a35af28dcc61f37ed850a2d64c65c701321dfbf25085e469d5559360cbbf/pillow-12.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:78cb2c6865a35ab8ff8b75fd122f6033b92a62c82801110e48ddd6c936a45d91", size = 6940830, upload-time = "2026-07-01T11:54:13.732Z" }, + { url = "https://files.pythonhosted.org/packages/eb/51/8b08617af3ad95e33ce6d7dd2c99ed6c8298f7fb131636303956be022e25/pillow-12.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e491916b378fba47242221bb9ead245211b70d504f495d105d17b14a24b4907c", size = 6344383, upload-time = "2026-07-01T11:54:15.756Z" }, + { url = "https://files.pythonhosted.org/packages/1d/72/cf78ac9780bb93c28328f408973845a309d4d145041665f734572ced1b52/pillow-12.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0dd2064cbc55aaec028ef5fbb60fa47bb6c3e7918e07ff17935284b227a9d2df", size = 7052934, upload-time = "2026-07-01T11:54:17.721Z" }, + { url = "https://files.pythonhosted.org/packages/20/20/25e0f4dc178a6bc0696793720055519a0de89e7661dae886992decbd2f81/pillow-12.3.0-cp312-cp312-win32.whl", hash = "sha256:dbce0b29841537a2fa4a214c2bbf14de3587c9680caa9b4e217568472490b28f", size = 6472684, upload-time = "2026-07-01T11:54:19.839Z" }, + { url = "https://files.pythonhosted.org/packages/45/89/da2f7971a317f83d807fdd4065c0af40208e59e692cc43d315a71a0e96d1/pillow-12.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:a2b55dd6b2a4c4b7d87ffa56bdb33fdc5fdb9a462173861a7bc097f17d91cb09", size = 7227137, upload-time = "2026-07-01T11:54:22.025Z" }, + { url = "https://files.pythonhosted.org/packages/de/47/4845a0a6c0dbf1db8456bd9fc791f13c5ced7ced20606d08a0aacfd25b49/pillow-12.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:331b624368d4f1d069149002f25f44bc61c8919ce8ddb3c45bdad8f6e2d89510", size = 2568267, upload-time = "2026-07-01T11:54:24.051Z" }, +] + +[[package]] +name = "platformdirs" +version = "4.11.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/36/0a/062135c9a98dac804265073cc3afdbec5ae1aa37980bb354f461bafe81b4/platformdirs-4.11.1.tar.gz", hash = "sha256:bb1af68078f25e2f3e111e2d43b8d536df41b73c8a684b40bb018223b66fae27", size = 32396, upload-time = "2026-08-07T23:06:48.516Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4c/85/9b31b44296cfa3bb56cddb35e6a0f6578bab0b490c0806c0245e32c6110c/platformdirs-4.11.1-py3-none-any.whl", hash = "sha256:2efd27d363e8dd2e661639ffb398865a5e0a46442a11d266bf375a0e0c10e386", size = 23261, upload-time = "2026-08-07T23:06:47.219Z" }, +] + +[[package]] +name = "plotly" +version = "6.9.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "narwhals" }, + { name = "packaging" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/96/07/795c79dbce40c39bece88e69d049babbd23ffa95b5d117f248db8ea03abb/plotly-6.9.0.tar.gz", hash = "sha256:967ad33e8c704fed051800d11d985eb206a9c795c14206b30a6f463ed9c67d0d", size = 6919903, upload-time = "2026-07-09T14:55:59.982Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/18/d8544811ab076f876c4892b3714f5b0dad335e1dc33aef826df431b8325d/plotly-6.9.0-py3-none-any.whl", hash = "sha256:36bebe2f1bb13884774fe61689c329071446f6ce4a8927fb1f0d6fb24f581236", size = 9909646, upload-time = "2026-07-09T14:55:55.421Z" }, +] + +[[package]] +name = "portalocker" +version = "4.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/45/f7/24b2821a077e0d85197a8c4093a4f71f78713d0a6f216c3f7cc90f7a5cc4/portalocker-4.1.0.tar.gz", hash = "sha256:91d0ff02d6f5f9a2dbf1ef1367ddb05b83136b2676707a1b2c8746f1f0dcd992", size = 97751, upload-time = "2026-08-02T15:05:25.159Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/61/91/288c883303be067c1648f1e63ea38e5f8eb5ab7123fd3a9a7366148e58b7/portalocker-4.1.0-py3-none-any.whl", hash = "sha256:d985a430d265adf31adf12bc0bf3501aea59efc495e9104c057e5dfb7394c226", size = 65914, upload-time = "2026-08-02T15:05:23.525Z" }, +] + +[[package]] +name = "psutil" +version = "7.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" }, + { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" }, + { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" }, + { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" }, + { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" }, + { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" }, + { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" }, + { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" }, +] + +[[package]] +name = "pybind11" +version = "3.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/76/f3/95b0f40b31df41dbfe6bb0857419c9442c15839cbac4796f1c26ae0b6081/pybind11-3.1.0.tar.gz", hash = "sha256:a1cc06b524ab3edca51f8ad3895f9c4fa20b8b19283173dff4ae781449dc9639", size = 603746, upload-time = "2026-08-06T23:33:00.675Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/33/fd/8762f7ee3e4e4be6d1d846cffb4916dd9bb02b2f800d3603718a0efe494c/pybind11-3.1.0-py3-none-any.whl", hash = "sha256:b8488090f8acffbcb6b5d6a85571a6827a0a2981ffb75e5a0b27b87c4a6b7dd0", size = 319402, upload-time = "2026-08-06T23:32:59.047Z" }, +] + +[[package]] +name = "pydantic" +version = "2.13.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/18/a5/b60d21ac674192f8ab0ba4e9fd860690f9b4a6e51ca5df118733b487d8d6/pydantic-2.13.4.tar.gz", hash = "sha256:c40756b57adaa8b1efeeced5c196f3f3b7c435f90e84ea7f443901bec8099ef6", size = 844775, upload-time = "2026-05-06T13:43:05.343Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl", hash = "sha256:45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba", size = 472262, upload-time = "2026-05-06T13:43:02.641Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.46.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9d/56/921726b776ace8d8f5db44c4ef961006580d91dc52b803c489fafd1aa249/pydantic_core-2.46.4.tar.gz", hash = "sha256:62f875393d7f270851f20523dd2e29f082bcc82292d66db2b64ea71f64b6e1c1", size = 471464, upload-time = "2026-05-06T13:37:06.98Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ce/8c/af022f0af448d7747c5154288d46b5f2bc5f17366eaa0e23e9aa04d59f3b/pydantic_core-2.46.4-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:3245406455a5d98187ec35530fd772b1d799b26667980872c8d4614991e2c4a2", size = 2106158, upload-time = "2026-05-06T13:38:57.215Z" }, + { url = "https://files.pythonhosted.org/packages/19/95/6195171e385007300f0f5574592e467c568becce2d937a0b6804f218bc49/pydantic_core-2.46.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:962ccbab7b642487b1d8b7df90ef677e03134cf1fd8880bf698649b22a69371f", size = 1951724, upload-time = "2026-05-06T13:37:02.697Z" }, + { url = "https://files.pythonhosted.org/packages/8e/bc/f47d1ff9cbb1620e1b5b697eef06010035735f07820180e74178226b27b3/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8233f2947cf85404441fd7e0085f53b10c93e0ee78611099b5c7237e36aacbf7", size = 1975742, upload-time = "2026-05-06T13:37:09.448Z" }, + { url = "https://files.pythonhosted.org/packages/5b/11/9b9a5b0306345664a2da6410877af6e8082481b5884b3ddd78d47c6013ce/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3a233125ac121aa3ffba9a2b59edfc4a985a76092dc8279586ab4b71390875e7", size = 2052418, upload-time = "2026-05-06T13:37:38.234Z" }, + { url = "https://files.pythonhosted.org/packages/f1/b7/a65fec226f5d78fc39f4a13c4cc0c768c22b113438f60c14adc9d2865038/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5b712b53160b79a5850310b912a5ef8e57e56947c8ad690c227f5c9d7e561712", size = 2232274, upload-time = "2026-05-06T13:38:27.753Z" }, + { url = "https://files.pythonhosted.org/packages/68/f0/92039db98b907ef49269a8271f67db9cb78ae2fc68062ef7e4e77adb5f61/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9401557acd873c3a7f3eb9383edef8ac4968f9510e340f4808d427e75667e7b4", size = 2309940, upload-time = "2026-05-06T13:38:05.353Z" }, + { url = "https://files.pythonhosted.org/packages/5f/97/2aab507d3d00ca626e8e57c1eac6a79e4e5fbcc63eb99733ff55d1717f65/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:926c9541b14b12b1681dca8a0b75feb510b06c6341b70a8e500c2fdcff837cce", size = 2094516, upload-time = "2026-05-06T13:39:10.577Z" }, + { url = "https://files.pythonhosted.org/packages/22/37/a8aca44d40d737dde2bc05b3c6c07dff0de07ce6f82e9f3167aeaf4d5dea/pydantic_core-2.46.4-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:56cb4851bcaf3d117eddcef4fe66afd750a50274b0da8e22be256d10e5611987", size = 2136854, upload-time = "2026-05-06T13:40:22.59Z" }, + { url = "https://files.pythonhosted.org/packages/24/99/fcef1b79238c06a8cbec70819ac722ba76e02bc8ada9b0fd66eba40da01b/pydantic_core-2.46.4-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c68fcd102d71ea85c5b2dfac3f4f8476eff42a9e078fd5faefff6d145063536b", size = 2180306, upload-time = "2026-05-06T13:40:10.666Z" }, + { url = "https://files.pythonhosted.org/packages/ae/6c/fc44000918855b42779d007ae63b0532794739027b2f417321cddbc44f6a/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:b2f69dec1725e79a012d920df1707de5caf7ed5e08f3be4435e25803efc47458", size = 2190044, upload-time = "2026-05-06T13:40:43.231Z" }, + { url = "https://files.pythonhosted.org/packages/6b/65/d9cadc9f1920d7a127ad2edba16c1db7916e59719285cd6c94600b0080ba/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:8d0820e8192167f80d88d64038e609c31452eeca865b4e1d9950a27a4609b00b", size = 2329133, upload-time = "2026-05-06T13:39:57.365Z" }, + { url = "https://files.pythonhosted.org/packages/d0/cf/c873d91679f3a30bcf5e7ac280ce5573483e72295307685120d0d5ad3416/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:fbdb89b3e1c94a30cc5edfce477c6e6a5dc4d8f84665b455c27582f211a1c72c", size = 2374464, upload-time = "2026-05-06T13:38:06.976Z" }, + { url = "https://files.pythonhosted.org/packages/47/bd/6f2fc8188f31bf10590f1e98e7b306336161fac930a8c514cd7bd828c7dc/pydantic_core-2.46.4-cp312-cp312-win32.whl", hash = "sha256:9aa768456404a8bf48a4406685ac2bec8e72b62c69313734fa3b73cf33b3a894", size = 1974823, upload-time = "2026-05-06T13:40:47.985Z" }, + { url = "https://files.pythonhosted.org/packages/40/8c/985c1d41ea1107c2534abd9870e4ed5c8e7669b5c308297835c001e7a1c4/pydantic_core-2.46.4-cp312-cp312-win_amd64.whl", hash = "sha256:e9c26f834c65f5752f3f06cb08cb86a913ceb7274d0db6e267808a708b46bc89", size = 2072919, upload-time = "2026-05-06T13:39:21.153Z" }, + { url = "https://files.pythonhosted.org/packages/c4/ba/f463d006e0c47373ca7ec5e1a261c59dc01ef4d62b2657af925fb0deee3a/pydantic_core-2.46.4-cp312-cp312-win_arm64.whl", hash = "sha256:4fc73cb559bdb54b1134a706a2802a4cddd27a0633f5abb7e53056268751ac6a", size = 2027604, upload-time = "2026-05-06T13:39:03.753Z" }, + { url = "https://files.pythonhosted.org/packages/9d/1d/8987ad40f65ae1432753072f214fb5c74fe47ffbd0698bb9cbbb585664f8/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:1d8ba486450b14f3b1d63bc521d410ec7565e52f887b9fb671791886436a42f7", size = 2095527, upload-time = "2026-05-06T13:39:52.283Z" }, + { url = "https://files.pythonhosted.org/packages/64/d3/84c282a7eee1d3ac4c0377546ef5a1ea436ce26840d9ac3b7ed54a377507/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:3009f12e4e90b7f88b4f9adb1b0c4a3d58fe7820f3238c190047209d148026df", size = 1936024, upload-time = "2026-05-06T13:40:15.671Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ca/eac61596cdeb4d7e174d3dc0bd8a6238f14f75f97a24e7b7db4c7e7340a0/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ad785e92e6dc634c21555edc8bd6b64957ab844541bcb96a1366c202951ae526", size = 1990696, upload-time = "2026-05-06T13:38:34.717Z" }, + { url = "https://files.pythonhosted.org/packages/fa/c3/7c8b240552251faf6b3a957db200fcfbbcec36763c050428b601e0c9b83b/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0", size = 2147590, upload-time = "2026-05-06T13:39:29.883Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pyparsing" +version = "3.3.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f3/91/9c6ee907786a473bf81c5f53cf703ba0957b23ab84c264080fb5a450416f/pyparsing-3.3.2.tar.gz", hash = "sha256:c777f4d763f140633dcb6d8a3eda953bf7a214dc4eff598413c070bcdc117cbc", size = 6851574, upload-time = "2026-01-21T03:57:59.36Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl", hash = "sha256:850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d", size = 122781, upload-time = "2026-01-21T03:57:55.912Z" }, +] + +[[package]] +name = "pyquaternion" +version = "0.9.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7d/0d/3d092aa20efaedacb89c3221a92c6491be5b28f618a2c36b52b53e7446c2/pyquaternion-0.9.9.tar.gz", hash = "sha256:b1f61af219cb2fe966b5fb79a192124f2e63a3f7a777ac3cadf2957b1a81bea8", size = 15530, upload-time = "2020-10-05T01:31:30.327Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/b3/d8482e8cacc8ea15a356efea13d22ce1c5914a9ee36622ba250523240bf2/pyquaternion-0.9.9-py3-none-any.whl", hash = "sha256:e65f6e3f7b1fdf1a9e23f82434334a1ae84f14223eee835190cd2e841f8172ec", size = 14361, upload-time = "2020-10-05T01:31:37.575Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" }, + { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" }, + { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" }, + { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" }, + { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" }, + { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" }, + { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" }, + { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, +] + +[[package]] +name = "referencing" +version = "0.37.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "rpds-py" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/22/f5/df4e9027acead3ecc63e50fe1e36aca1523e1719559c499951bb4b53188f/referencing-0.37.0.tar.gz", hash = "sha256:44aefc3142c5b842538163acb373e24cce6632bd54bdb01b21ad5863489f50d8", size = 78036, upload-time = "2025-10-13T15:30:48.871Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" }, +] + +[[package]] +name = "regex" +version = "2026.7.19" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/20/98/04b13f1ddfb63158025291c02e03eb42fbb7acb51d091d541050eb4e35e8/regex-2026.7.19.tar.gz", hash = "sha256:7e77b324909c1617cbb4c668677e2c6ae13f44d7c1de0d4f15f2e3c10f3315b5", size = 416440, upload-time = "2026-07-19T00:19:48.923Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3b/b9/d11d7e501ac8fd7d617684423ebb9561e0b998481c1e4cbc0cb212c5d74a/regex-2026.7.19-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:2cc3460cedf7579948486eab03bc9ad7089df4d7281c0f47f4afe03e8d13f02d", size = 496778, upload-time = "2026-07-19T00:17:05.677Z" }, + { url = "https://files.pythonhosted.org/packages/3f/a9/a5ab6f312f24318019170dc485d5421fe4f89e43a98640da50d95a8a7041/regex-2026.7.19-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0e9554c8785eac5cffe6300f69a91f58ba72bc88a5f8d661235ad7c6aa5b8ccd", size = 297122, upload-time = "2026-07-19T00:17:07.59Z" }, + { url = "https://files.pythonhosted.org/packages/b3/63/4cab4d7f2d384a144d420b763d97674cb70619c878ea6fcd7640d0e62143/regex-2026.7.19-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d7da47a0f248977f08e2cb659ff3c17ddc13a4d39b3a7baa0a81bf5b415430f6", size = 292009, upload-time = "2026-07-19T00:17:09.648Z" }, + { url = "https://files.pythonhosted.org/packages/22/85/102a81b218298957d4ea7d2f084fae537a71add9d6ff93c8e67284c5f45e/regex-2026.7.19-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:93db40c8de0815baab96a06e08a984bac71f989d13bab789e382158c5d426797", size = 796708, upload-time = "2026-07-19T00:17:11.542Z" }, + { url = "https://files.pythonhosted.org/packages/78/b5/dc136af5629938a037cd2b304c12240e132ec92f38be8ff9cc89af2a1f2d/regex-2026.7.19-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:66bd62c59a5427746e8c44becae1d9b99d22fb13f30f492083dfb9ad7c45cc18", size = 865651, upload-time = "2026-07-19T00:17:13.312Z" }, + { url = "https://files.pythonhosted.org/packages/e0/75/67402ae3cd9c8c988a4c805d15ee3eef015e7ca4cb112cf3e640fc1f4153/regex-2026.7.19-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1649eb39fcc9ea80c4d2f110fde2b8ab2aef3877b98f02ab9b14e961f418c511", size = 911756, upload-time = "2026-07-19T00:17:15.015Z" }, + { url = "https://files.pythonhosted.org/packages/2a/8e/096d00c7c480ef2ff4265349b14e2261d4ab787ba1f74e2e80d1c58079c3/regex-2026.7.19-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9dce8ec9695f531a1b8a6f314fd4b393adcccf2ea861db480cdf97a301d01a68", size = 801798, upload-time = "2026-07-19T00:17:17.208Z" }, + { url = "https://files.pythonhosted.org/packages/f0/41/e7ecac6edb5722417f85cc67eaf386322fbe8acf6918ec2fdc37c20dd9d0/regex-2026.7.19-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3080a7fd38ef049bd489e01c970c97dd84ff446a885b0f1f6b26d9b1ad13ce11", size = 776933, upload-time = "2026-07-19T00:17:19.347Z" }, + { url = "https://files.pythonhosted.org/packages/6f/69/03c9b3f058d66403e0ca2c938696e81d51cd4c6d47ec5265f02f96948d9a/regex-2026.7.19-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1d793a7988e04fcb1e2e135567443d82173225d657419ec09414a9b5a145b986", size = 784338, upload-time = "2026-07-19T00:17:21.057Z" }, + { url = "https://files.pythonhosted.org/packages/f6/f7/b38ab3d43f284afbb618fcd15d0e77eb786ae461ce1f6bc7494619ddc0f2/regex-2026.7.19-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:e8b0abe7d870f53ca5143895fef7d1041a0c831a140d3dc2c760dd7ba25d4a8b", size = 860452, upload-time = "2026-07-19T00:17:23.119Z" }, + { url = "https://files.pythonhosted.org/packages/15/5c/ff60ef0571121714f3cf9920bc183071e384a10b556d042e0fdb06cc07a5/regex-2026.7.19-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:4e5413bd5f13d3a4e3539ca98f70f75e7fca92518dd7f117f030ebedd10b60cb", size = 765958, upload-time = "2026-07-19T00:17:24.81Z" }, + { url = "https://files.pythonhosted.org/packages/aa/0f/bd34021162c0ab47f9a315bd56cd5642e920c8e5668a75ef6c6a6fca590d/regex-2026.7.19-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:73b133a9e6fb512858e7f065e96f1180aa46646bc74a83aea62f1d314f3dd035", size = 851765, upload-time = "2026-07-19T00:17:26.993Z" }, + { url = "https://files.pythonhosted.org/packages/2a/20/a2ca43edade0595cccfdc98636739f536d9e26898e7dbddc2b9e98898953/regex-2026.7.19-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:dbe6493fbd27321b1d1f2dd4f5c7e5bd4d8b1d7cab7f32fd67db3d0b2ed8248a", size = 789714, upload-time = "2026-07-19T00:17:28.699Z" }, + { url = "https://files.pythonhosted.org/packages/5d/47/e02db4015d424fc83c00ea0ac8c5e5ec14397943de9abf909d5ce3a25931/regex-2026.7.19-cp312-cp312-win32.whl", hash = "sha256:ddd67571c10869f65a5d7dde536d1e066e306cc90de57d7de4d5f34802428bb5", size = 267157, upload-time = "2026-07-19T00:17:31.051Z" }, + { url = "https://files.pythonhosted.org/packages/08/8e/c780c131f79b42ed22d1bd7da4096c2c35f813e835acd02ef0f018bd892c/regex-2026.7.19-cp312-cp312-win_amd64.whl", hash = "sha256:e30d40268a28d54ce0437031750497004c22602b8e3ab891f759b795a003b312", size = 277777, upload-time = "2026-07-19T00:17:32.848Z" }, + { url = "https://files.pythonhosted.org/packages/3e/4c/e4d7e086449bdf379d89774bf1f89dc4a41943f3c5a6125a03905b34b5fb/regex-2026.7.19-cp312-cp312-win_arm64.whl", hash = "sha256:de9208bb427130c82a5dbfd104f92c8876fc9559278c880b3002755bbbe9c83d", size = 277136, upload-time = "2026-07-19T00:17:34.803Z" }, +] + +[[package]] +name = "requests" +version = "2.34.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856, upload-time = "2026-05-14T19:25:27.735Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, +] + +[[package]] +name = "retrying" +version = "1.4.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c8/5a/b17e1e257d3e6f2e7758930e1256832c9ddd576f8631781e6a072914befa/retrying-1.4.2.tar.gz", hash = "sha256:d102e75d53d8d30b88562d45361d6c6c934da06fab31bd81c0420acb97a8ba39", size = 11411, upload-time = "2025-08-03T03:35:25.189Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/67/f3/6cd296376653270ac1b423bb30bd70942d9916b6978c6f40472d6ac038e7/retrying-1.4.2-py3-none-any.whl", hash = "sha256:bbc004aeb542a74f3569aeddf42a2516efefcdaff90df0eb38fbfbf19f179f59", size = 10859, upload-time = "2025-08-03T03:35:23.829Z" }, +] + +[[package]] +name = "rich" +version = "15.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, +] + +[[package]] +name = "rpds-py" +version = "2026.6.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/2a/9618a122aeb2a169a28b03889a2995fe297588964333d4a7d67bdf46e147/rpds_py-2026.6.3.tar.gz", hash = "sha256:1cebd1337c242e4ec2293e541f712b2da849b29f48f0c293684b71c0632625d4", size = 64051, upload-time = "2026-06-30T07:17:53.009Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5c/be/2e8974163072e7bab7df1a5acd54c4498e75e35d6d18b864d3a9d5dadc92/rpds_py-2026.6.3-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a0811d33247c3d6128a3001d763f2aa056bb3425204335400ac54f89eec3a0d0", size = 343691, upload-time = "2026-06-30T07:15:14.96Z" }, + { url = "https://files.pythonhosted.org/packages/a4/73/319dfa745dd668efe89309141ded489126461fcecd2b8f3a3cda185129b6/rpds_py-2026.6.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:538949e262e46caa31ac01bdb3c1e8f642622922cacbabbae6a8445d9dc33eaf", size = 338542, upload-time = "2026-06-30T07:15:16.267Z" }, + { url = "https://files.pythonhosted.org/packages/21/63/4239893be1c4d09b709b1a8f6be4188f0870084ff547f46606b8a75f1b03/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:55927d532399c2c646100ff7feb48eaa940ad70f42cd68e1328f3ded9f81ca24", size = 368180, upload-time = "2026-06-30T07:15:17.62Z" }, + { url = "https://files.pythonhosted.org/packages/1c/ca/9c5de382225234ceb37b1844ebdb140db12b2a278bb9efe2fcd19f6c82ce/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f56f1695bc5c0871cbc33dc0130fcf503aab0c57dcc5a6700a4f49eba4f2652e", size = 375067, upload-time = "2026-06-30T07:15:18.952Z" }, + { url = "https://files.pythonhosted.org/packages/87/dc/863f69d1bf04ade34b7fe0d59b9fdf6f0135fe2d7cbca74f1d665589559d/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:270b293dae9058fc9fcedab50f13cebf46fb8ed1d1d54e0521a9da5d6b211975", size = 490509, upload-time = "2026-06-30T07:15:20.434Z" }, + { url = "https://files.pythonhosted.org/packages/ce/ef/eac16a12048b45ec7c7fa94f2be3438a5f26bf9cc8580b18a1cfd609b7f6/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:127565fead0a10943b282957bd5447804ff3160ad79f2ad2635e6d249e380680", size = 382754, upload-time = "2026-06-30T07:15:21.831Z" }, + { url = "https://files.pythonhosted.org/packages/04/8f/d2f3f532616be4d06c316ef119683e832bd3d41e112bf3a88f4151c95b17/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ecabd69db66de867690f9797f2f8fa27ba501bbc24540cbdbdc649cd15888ba6", size = 366189, upload-time = "2026-06-30T07:15:23.371Z" }, + { url = "https://files.pythonhosted.org/packages/e3/29/41a7b0e98a4b44cd676ab7598419623373eb43b20be68c084935c1a8cf88/rpds_py-2026.6.3-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:58eadac9cd119677b60e1cf8ac4052f35949d71b8a9e5556efccbe82533cf22a", size = 377750, upload-time = "2026-06-30T07:15:24.659Z" }, + { url = "https://files.pythonhosted.org/packages/2e/05/ecda0bec46f9a1565090bcdc941d023f6a25aff85fda28f89f8d19878152/rpds_py-2026.6.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7491ee23305ac3eb59e492b6945881f5cd77a6f731061a3f25b77fd40f9e99a4", size = 395576, upload-time = "2026-06-30T07:15:25.987Z" }, + { url = "https://files.pythonhosted.org/packages/68/a8/6ed52f03ee6cb854ce78785cc9a9a672eb880e83fd7224d471f667d151f1/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2c99f7e8ccb3dd6e3e4bfeac657a7b208c9bac8075f4b078c02d7404c34107fa", size = 543807, upload-time = "2026-06-30T07:15:27.356Z" }, + { url = "https://files.pythonhosted.org/packages/8f/d6/156c0d3eea27ba09b92562ba2364ba124c0a061b199e17eac637cd25a5e2/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:62698275682bf121181861295c9181e789030a2d516071f5b8f3c23c170cd0fc", size = 611187, upload-time = "2026-06-30T07:15:28.931Z" }, + { url = "https://files.pythonhosted.org/packages/f1/31/774212ed989c62f7f310220089f9b0a3fb8f40f5443d1727abd5d9f52bc9/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a214c993455f99a89aaeadc9b21241900037adc9d97203e374d75513c5911822", size = 573030, upload-time = "2026-06-30T07:15:30.553Z" }, + { url = "https://files.pythonhosted.org/packages/c9/50/22f73127a41f1ce4f87fe39aadfb9a126345801c274aa93ae88456249327/rpds_py-2026.6.3-cp312-cp312-win32.whl", hash = "sha256:501f9f04a588d6a09179368c57071301445191767c64e4b52a6aa9871f1ef5ed", size = 202185, upload-time = "2026-06-30T07:15:32.027Z" }, + { url = "https://files.pythonhosted.org/packages/04/3a/f0ee4d4dde9d3b69dedf1b5f74e7a40017046d55052d173e418c6a94f960/rpds_py-2026.6.3-cp312-cp312-win_amd64.whl", hash = "sha256:2c958bf94822e9290a40aaf2a822d4bc5c88099093e3948ad6c571eca9272e5f", size = 220394, upload-time = "2026-06-30T07:15:33.359Z" }, + { url = "https://files.pythonhosted.org/packages/f3/83/3382fe37f809b59f02aac04dbc4e765b480b46ee0227ed516e3bdc4d3dfc/rpds_py-2026.6.3-cp312-cp312-win_arm64.whl", hash = "sha256:22bffe6042b9bcb0822bcd1955ec00e245daf17b4344e4ed8e9551b976b63e96", size = 215753, upload-time = "2026-06-30T07:15:34.778Z" }, +] + +[[package]] +name = "safetensors" +version = "0.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/45/06/f955dbbb1859e3bd23c8ac6141af5106e7ad5fedec4a3a6e3d60f94b7001/safetensors-0.8.0.tar.gz", hash = "sha256:fabaf3e0f18a6618d9b36560682562157f77c2b71fcffc7b432be2baed9d753d", size = 325846, upload-time = "2026-06-09T07:52:25.563Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/39/a0/f718cda65b05407d228f97602cf60dca269c979867aa5beb25410de26cd3/safetensors-0.8.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:c554f85858e05226d3c2828e32395e677434685d6d94594a41643361c5e837f0", size = 473568, upload-time = "2026-06-09T07:52:18.829Z" }, + { url = "https://files.pythonhosted.org/packages/f5/b1/fa7c600e7dceae12e9606c7578cbc9ff1e1ed55844883ee5c92205e86226/safetensors-0.8.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:c80201d22cbf405b80647a60ada77bba06c8fba2da2743ba1e89cdcc39a81f25", size = 484562, upload-time = "2026-06-09T07:52:17.518Z" }, + { url = "https://files.pythonhosted.org/packages/09/7d/65a7de0af421317bb36a067241e4235fff194eed60b961ed6d3f59a3fc60/safetensors-0.8.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7a46e5ff292c356d6991e60942ba7f79817682d3a2cef0702136448cb9c4d235", size = 502844, upload-time = "2026-06-09T07:52:07.624Z" }, + { url = "https://files.pythonhosted.org/packages/91/4f/3175c9d75634e0e0dda0082794193521035edd7c70a6f212bf33ca06ddf4/safetensors-0.8.0-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4124502b78f03534117c848f87a39b8f31e577b15eff423bf8bfb95f2a8c30d0", size = 511823, upload-time = "2026-06-09T07:52:09.565Z" }, + { url = "https://files.pythonhosted.org/packages/20/87/846c289e7aa2299eff406335717cf43ce8777194ece8aad75772e0411615/safetensors-0.8.0-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7bc0a787ba8a35be368ee3574edfa2b1ad389eebd0a72e482ae275490e3f6c98", size = 633461, upload-time = "2026-06-09T07:52:11.128Z" }, + { url = "https://files.pythonhosted.org/packages/76/22/8d64d9df2c45d5ded401df889d0ad90882804ca172d79ec4f0df8f727fe0/safetensors-0.8.0-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:040070828e36dc8e122178bbbd5830ff9e97920affb84cbe0f46442497bed358", size = 545148, upload-time = "2026-06-09T07:52:13.603Z" }, + { url = "https://files.pythonhosted.org/packages/28/50/f203ff3a3ddfe19308efc83c5a3a29ed02bf786732ec35e68bf9162f3365/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd6f3f93c9a0a7cc2788ee63fb763353d4bd2e89b0751bc78fcf7dda00bea774", size = 516040, upload-time = "2026-06-09T07:52:16.29Z" }, + { url = "https://files.pythonhosted.org/packages/46/fb/cdaed17ceb2948784fd9c36b6fd3e951b608547cea81a48e8ee6f8cfdfcb/safetensors-0.8.0-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:fcdd41ec4628fee5799f807c73c353629130fbd942aa23d83c623dd6c9d52d78", size = 513832, upload-time = "2026-06-09T07:52:12.37Z" }, + { url = "https://files.pythonhosted.org/packages/0d/49/1e15de264dcc3b77943d2d0c56a95809956883b1c2d6d585c792523f180b/safetensors-0.8.0-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:8e9f537aa183a38ace122d27303dcd986b26bd2a7591f9181d7f0c396f4677ca", size = 559930, upload-time = "2026-06-09T07:52:14.743Z" }, + { url = "https://files.pythonhosted.org/packages/2a/43/bf38443278eab4b1be1fce2931e2b012ad9cb7df52ada751d0aab8f7659a/safetensors-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:87eec7ffed2b809f05a398a8becb7d013f19f7837cd15d9748580d6cf30dbaf4", size = 678670, upload-time = "2026-06-09T07:52:20.032Z" }, + { url = "https://files.pythonhosted.org/packages/72/e3/68cd3fa5b48488e84add63e04cb12f3bc28ae4638c06d4508c6e88823d0e/safetensors-0.8.0-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:4a95ae2b05d7726d751da4ebf626a2ca782b706e101bd894c95bc2450b1cffcc", size = 786679, upload-time = "2026-06-09T07:52:21.322Z" }, + { url = "https://files.pythonhosted.org/packages/29/4b/1c19c509d56e01f4fbb3d0a2e597450f6cc04d1d56cf52defb0a62dfd715/safetensors-0.8.0-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:3ae091f16662658bdc019a4ff6cb4c085bb7d725eb5978b183ffd265863b6d2d", size = 765683, upload-time = "2026-06-09T07:52:22.594Z" }, + { url = "https://files.pythonhosted.org/packages/27/43/41c1621732edd934d868a00d1b891584c892a7b62a9aab82ea5a0a5623ee/safetensors-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8e080062fcde23be189565e1c3305d16751a218ecf9412c8601e64204eb6f846", size = 722361, upload-time = "2026-06-09T07:52:23.924Z" }, + { url = "https://files.pythonhosted.org/packages/8e/3f/73ccf82579412b4a71c4ca673f10b5f1f888d7cf5af7fe24f27d30307be4/safetensors-0.8.0-cp310-abi3-win32.whl", hash = "sha256:2ddf52eac562eda224f99acfa7889d02968c1fd59a5b011ae7d8137c37e9c02d", size = 342401, upload-time = "2026-06-09T07:52:28.895Z" }, + { url = "https://files.pythonhosted.org/packages/1b/6d/3fba214c1e5e0f69991677ec3bc17023f0421776975e1de0c682dca475e2/safetensors-0.8.0-cp310-abi3-win_amd64.whl", hash = "sha256:096ec1a98435df7beb08853bb5aa9081a84f23d0adc67ed1a0a10550f608373f", size = 355540, upload-time = "2026-06-09T07:52:27.832Z" }, + { url = "https://files.pythonhosted.org/packages/8d/fc/7eedc3510d97878876e32774eebbeb61c43f148a96e915c84229a3e967aa/safetensors-0.8.0-cp310-abi3-win_arm64.whl", hash = "sha256:f7838e5135a406ad3e02efdcb8cf2e5397d368b0154537c4fec682dbc544d452", size = 340500, upload-time = "2026-06-09T07:52:26.745Z" }, +] + +[[package]] +name = "scikit-learn" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "joblib" }, + { name = "narwhals" }, + { name = "numpy" }, + { name = "scipy" }, + { name = "threadpoolctl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/fa/6f/37092bdb25f712817231799fc5674d8e704066a8a70c1d2d40517e18b4ab/scikit_learn-1.9.0.tar.gz", hash = "sha256:8833266989d3a5110178a9fae30783675460724d0e1efb13b14901d2c660c557", size = 7750767, upload-time = "2026-06-02T11:54:32.706Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ac/20/75f915ff375d6249e6550ac740fdbbd66159a068fd3af1400ff62036b07a/scikit_learn-1.9.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2bd41b0d201bc81575531b96b713d3eb5e5f50fb0b82101ff0f92294fdc236ac", size = 8741122, upload-time = "2026-06-02T11:53:24.08Z" }, + { url = "https://files.pythonhosted.org/packages/cc/d5/2b5148f2279196775e1db2aeb85d14b70ac80e7e32b3b28e7ebeafb0901d/scikit_learn-1.9.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:5be45aa4a42a68a533913a6ed736cf309de2226411c79ef8d609a5456f1939b1", size = 8261512, upload-time = "2026-06-02T11:53:27.183Z" }, + { url = "https://files.pythonhosted.org/packages/a0/ee/5adbc77656b71f9456a2f5a7a9fdb4bcf9207a6b962889f1c2f9323afa4e/scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5e50ed4da51974e86e940690e9a3d82e729b62b5a49f7c9bac534d515d39d86f", size = 8837603, upload-time = "2026-06-02T11:53:30.328Z" }, + { url = "https://files.pythonhosted.org/packages/6c/c2/63fdda36c56437eeb44aaf9493c8bcd62ce230ab1598924fc626ffbfa943/scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:056c92bb67ad4c28463c2f2653d9701449201e7e7a9e94e321be0f71c4fef2b8", size = 9132097, upload-time = "2026-06-02T11:53:33.456Z" }, + { url = "https://files.pythonhosted.org/packages/83/a4/c8e67227c680e2259c8864ae72ff48b06e16a6f51253a22167aa02a8aa4e/scikit_learn-1.9.0-cp312-cp312-win_amd64.whl", hash = "sha256:4306775fad04cc4b472a1b15af1ae9cede1540fbfcc17fbce3767cd8dc7ae283", size = 8211173, upload-time = "2026-06-02T11:53:36.602Z" }, + { url = "https://files.pythonhosted.org/packages/cf/fd/3c0863792e98e67e9184aa4029288a175935eb65443afcd30d4f143450cf/scikit_learn-1.9.0-cp312-cp312-win_arm64.whl", hash = "sha256:26e22435f63bcdcf396b574273f29f13dd531f5ea035801f5be10ba1540a4e60", size = 7867451, upload-time = "2026-06-02T11:53:39.075Z" }, +] + +[[package]] +name = "scipy" +version = "1.18.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a7/25/c2700dfaf6442b4effaa91af24ebce5dc9d31bb4a69706313aae70d72cd0/scipy-1.18.0.tar.gz", hash = "sha256:67b2ad2ad54c72ca6d04975a9b2df8c3638c34ddd5b28738e94fc2b57929d378", size = 30774447, upload-time = "2026-06-19T15:01:43.456Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/19/ca10ead60b0acc80b2b833c2c4a4f2ff753d0f58b811f70d911c7e94a25c/scipy-1.18.0-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:7bd21faaf5a1a3b2eff922d02db5f191b99a6518db9078a8fb23169f6d22259a", size = 31056519, upload-time = "2026-06-19T14:59:45.203Z" }, + { url = "https://files.pythonhosted.org/packages/96/72/1e6442a00cd2924d361aa1b642ab6373ec35c6fabf311a760be9f76e0f13/scipy-1.18.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:265915e79107de9f946b855e50d7470d5893ec3f54b342e1aa6201cbdcd8bb6b", size = 28681889, upload-time = "2026-06-19T14:59:48.103Z" }, + { url = "https://files.pythonhosted.org/packages/9b/2d/11dd93d21e147a73ba22bd75c0b9208d3a2e0ec76d53170ce7d9029b1015/scipy-1.18.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:9ab7b758be6940954a713ee466e2043e9f6e2ed965c1fce5c91039f4be3d90a9", size = 20423580, upload-time = "2026-06-19T14:59:50.665Z" }, + { url = "https://files.pythonhosted.org/packages/9c/01/93552f75e0d2a7dd115a45e59209c51e8d514daff02fc887d2623be06fe1/scipy-1.18.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:97b6cddaaee0a779ef6b5ca83c9604b27cc16b2b8fc22c142652df8793319fb8", size = 23054441, upload-time = "2026-06-19T14:59:53.564Z" }, + { url = "https://files.pythonhosted.org/packages/3c/23/21f5e703643d66f21faa6b4c73195bfcad70c55efcb4f1ab327cd7c4101a/scipy-1.18.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:52a96e21517c7292375c0e27dd796a811f03fcea5fd4d108fdfea8145dcf17ab", size = 33968720, upload-time = "2026-06-19T14:59:56.415Z" }, + { url = "https://files.pythonhosted.org/packages/dd/aa/1b939f6c67ed68635bb538e6752d3dacc02f66535182e939a89581a44e9c/scipy-1.18.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f55797419e16e7f30cf88ffb3113ce0467f00cfe3f70d5c281730b21769bfc2", size = 35287115, upload-time = "2026-06-19T14:59:59.411Z" }, + { url = "https://files.pythonhosted.org/packages/b6/ff/eec46be7e9234208f801062b53e1983085eddebd693f6c9bfb03b459830d/scipy-1.18.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ad033410e2e0672ffdc1042110cef20e1c46f8fd0616cee1d44d8d58fad8fc11", size = 35577989, upload-time = "2026-06-19T15:00:02.235Z" }, + { url = "https://files.pythonhosted.org/packages/84/ca/210d4759c7210bb7d269437421959b39a33434e2776b60c5cb8a763bb30a/scipy-1.18.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:4a55985d54c769c872e64b7f4c8a81cc30ef700cc04296abbbf3705439c126de", size = 37421717, upload-time = "2026-06-19T15:00:05.102Z" }, + { url = "https://files.pythonhosted.org/packages/2b/54/9a9edb45345bd6744da5ddfb6628e5d5185920494c6a67ec45b6381004cb/scipy-1.18.0-cp312-cp312-win_amd64.whl", hash = "sha256:71ccc8faa2dd16ac310233203474a8b5cb67f10dedd54a3116d34943f4b19132", size = 36597428, upload-time = "2026-06-19T15:00:08.112Z" }, + { url = "https://files.pythonhosted.org/packages/99/0e/33f32a2a58987e26aec0f7df252cbbad1e90ae77bdbc76f40dd4ed0cf0ea/scipy-1.18.0-cp312-cp312-win_arm64.whl", hash = "sha256:d88363fd9d8fbd3511bd273f1a49efb2a540773ddf92a91d57498ce7dd7f3e76", size = 24351481, upload-time = "2026-06-19T15:00:11.103Z" }, +] + +[[package]] +name = "setuptools" +version = "84.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6d/44/f5da03a8ef95d369145c5bb53050e7877c9f3d312e128605fd9504829143/setuptools-84.0.0.tar.gz", hash = "sha256:f4695c21257f0d9b537ec2692c941d02ee143b7cc1276941349a546573b2ef73", size = 1168449, upload-time = "2026-08-08T18:27:58.365Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/95/9c/c510029fc6ef33a6275cd2c5d3cecd6613dfd6aa401d57c54f1c18852ccf/setuptools-84.0.0-py3-none-any.whl", hash = "sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670", size = 818216, upload-time = "2026-08-08T18:27:56.719Z" }, +] + +[[package]] +name = "shellingham" +version = "1.5.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/58/15/8b3609fd3830ef7b27b655beb4b4e9c62313a4e8da8c676e142cc210d58e/shellingham-1.5.4.tar.gz", hash = "sha256:8dbca0739d487e5bd35ab3ca4b36e11c4078f3a234bfce294b0a0291363404de", size = 10310, upload-time = "2023-10-24T04:13:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl", hash = "sha256:7ecfff8f2fd72616f7481040475a65b2bf8af90a56c89140852d1120324e8686", size = 9755, upload-time = "2023-10-24T04:13:38.866Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "spconv-cu126" +version = "2.3.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "ccimport" }, + { name = "cumm-cu126" }, + { name = "fire" }, + { name = "numpy" }, + { name = "pccm" }, + { name = "pybind11" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/d2/c8/0b8e710b76b997f8c020dcac4ea0856c3702392464d2cb8ae63510a451fc/spconv_cu126-2.3.8-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:88002b375d33561ac956e29e00dade16cad11d0e3413630a8ef35e92ddfda6e1", size = 70470805, upload-time = "2024-12-15T18:03:31.347Z" }, + { url = "https://files.pythonhosted.org/packages/f1/14/880b7a50ff2e44f948847706a07d0841aa2e95e64c07bff3f40d0b9b22b6/spconv_cu126-2.3.8-cp312-cp312-win_amd64.whl", hash = "sha256:e5935a3948c0126d2e15042dc2a0c987d70e66b6198add08885590c78b682f7b", size = 68810321, upload-time = "2024-12-15T18:07:30.656Z" }, +] + +[[package]] +name = "sympy" +version = "1.14.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mpmath" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517", size = 7793921, upload-time = "2025-04-27T18:05:01.611Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" }, +] + +[[package]] +name = "termcolor" +version = "3.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/46/79/cf31d7a93a8fdc6aa0fbb665be84426a8c5a557d9240b6239e9e11e35fc5/termcolor-3.3.0.tar.gz", hash = "sha256:348871ca648ec6a9a983a13ab626c0acce02f515b9e1983332b17af7979521c5", size = 14434, upload-time = "2025-12-29T12:55:21.882Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/33/d1/8bb87d21e9aeb323cc03034f5eaf2c8f69841e40e4853c2627edf8111ed3/termcolor-3.3.0-py3-none-any.whl", hash = "sha256:cf642efadaf0a8ebbbf4bc7a31cec2f9b5f21a9f726f4ccbb08192c9c26f43a5", size = 7734, upload-time = "2025-12-29T12:55:20.718Z" }, +] + +[[package]] +name = "threadpoolctl" +version = "3.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b7/4d/08c89e34946fce2aec4fbb45c9016efd5f4d7f24af8e5d93296e935631d8/threadpoolctl-3.6.0.tar.gz", hash = "sha256:8ab8b4aa3491d812b623328249fab5302a68d2d71745c8a4c719a2fcaba9f44e", size = 21274, upload-time = "2025-03-13T13:49:23.031Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" }, +] + +[[package]] +name = "timm" +version = "1.0.28" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, + { name = "pyyaml" }, + { name = "safetensors" }, + { name = "torch" }, + { name = "torchvision" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/35/03/e41389ac641747bfec48d016fde8be1eade1901e6f2c1aedcb0c8cb4b5d9/timm-1.0.28.tar.gz", hash = "sha256:3789d313fdd5541a327b60180d70dbb4bdec73db8ff0655e413db3c3d134a9a4", size = 2451413, upload-time = "2026-07-11T17:24:32.615Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/76/de1bfac17d183c49c6d0887903d3064ced51cf1d9ba7a8d611c1a8808c4f/timm-1.0.28-py3-none-any.whl", hash = "sha256:e577b88da96b3a722ea5e2f042455ce6f715d398304d8e63b17d126ed7d89968", size = 2597944, upload-time = "2026-07-11T17:24:30.869Z" }, +] + +[[package]] +name = "tokenizers" +version = "0.22.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/73/6f/f80cfef4a312e1fb34baf7d85c72d4411afde10978d4657f8cdd811d3ccc/tokenizers-0.22.2.tar.gz", hash = "sha256:473b83b915e547aa366d1eee11806deaf419e17be16310ac0a14077f1e28f917", size = 372115, upload-time = "2026-01-05T10:45:15.988Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/92/97/5dbfabf04c7e348e655e907ed27913e03db0923abb5dfdd120d7b25630e1/tokenizers-0.22.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:544dd704ae7238755d790de45ba8da072e9af3eea688f698b137915ae959281c", size = 3100275, upload-time = "2026-01-05T10:41:02.158Z" }, + { url = "https://files.pythonhosted.org/packages/2e/47/174dca0502ef88b28f1c9e06b73ce33500eedfac7a7692108aec220464e7/tokenizers-0.22.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1e418a55456beedca4621dbab65a318981467a2b188e982a23e117f115ce5001", size = 2981472, upload-time = "2026-01-05T10:41:00.276Z" }, + { url = "https://files.pythonhosted.org/packages/d6/84/7990e799f1309a8b87af6b948f31edaa12a3ed22d11b352eaf4f4b2e5753/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2249487018adec45d6e3554c71d46eb39fa8ea67156c640f7513eb26f318cec7", size = 3290736, upload-time = "2026-01-05T10:40:32.165Z" }, + { url = "https://files.pythonhosted.org/packages/78/59/09d0d9ba94dcd5f4f1368d4858d24546b4bdc0231c2354aa31d6199f0399/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:25b85325d0815e86e0bac263506dd114578953b7b53d7de09a6485e4a160a7dd", size = 3168835, upload-time = "2026-01-05T10:40:38.847Z" }, + { url = "https://files.pythonhosted.org/packages/47/50/b3ebb4243e7160bda8d34b731e54dd8ab8b133e50775872e7a434e524c28/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bfb88f22a209ff7b40a576d5324bf8286b519d7358663db21d6246fb17eea2d5", size = 3521673, upload-time = "2026-01-05T10:40:56.614Z" }, + { url = "https://files.pythonhosted.org/packages/e0/fa/89f4cb9e08df770b57adb96f8cbb7e22695a4cb6c2bd5f0c4f0ebcf33b66/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1c774b1276f71e1ef716e5486f21e76333464f47bece56bbd554485982a9e03e", size = 3724818, upload-time = "2026-01-05T10:40:44.507Z" }, + { url = "https://files.pythonhosted.org/packages/64/04/ca2363f0bfbe3b3d36e95bf67e56a4c88c8e3362b658e616d1ac185d47f2/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:df6c4265b289083bf710dff49bc51ef252f9d5be33a45ee2bed151114a56207b", size = 3379195, upload-time = "2026-01-05T10:40:51.139Z" }, + { url = "https://files.pythonhosted.org/packages/2e/76/932be4b50ef6ccedf9d3c6639b056a967a86258c6d9200643f01269211ca/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:369cc9fc8cc10cb24143873a0d95438bb8ee257bb80c71989e3ee290e8d72c67", size = 3274982, upload-time = "2026-01-05T10:40:58.331Z" }, + { url = "https://files.pythonhosted.org/packages/1d/28/5f9f5a4cc211b69e89420980e483831bcc29dade307955cc9dc858a40f01/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:29c30b83d8dcd061078b05ae0cb94d3c710555fbb44861139f9f83dcca3dc3e4", size = 9478245, upload-time = "2026-01-05T10:41:04.053Z" }, + { url = "https://files.pythonhosted.org/packages/6c/fb/66e2da4704d6aadebf8cb39f1d6d1957df667ab24cff2326b77cda0dcb85/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:37ae80a28c1d3265bb1f22464c856bd23c02a05bb211e56d0c5301a435be6c1a", size = 9560069, upload-time = "2026-01-05T10:45:10.673Z" }, + { url = "https://files.pythonhosted.org/packages/16/04/fed398b05caa87ce9b1a1bb5166645e38196081b225059a6edaff6440fac/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:791135ee325f2336f498590eb2f11dc5c295232f288e75c99a36c5dbce63088a", size = 9899263, upload-time = "2026-01-05T10:45:12.559Z" }, + { url = "https://files.pythonhosted.org/packages/05/a1/d62dfe7376beaaf1394917e0f8e93ee5f67fea8fcf4107501db35996586b/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:38337540fbbddff8e999d59970f3c6f35a82de10053206a7562f1ea02d046fa5", size = 10033429, upload-time = "2026-01-05T10:45:14.333Z" }, + { url = "https://files.pythonhosted.org/packages/fd/18/a545c4ea42af3df6effd7d13d250ba77a0a86fb20393143bbb9a92e434d4/tokenizers-0.22.2-cp39-abi3-win32.whl", hash = "sha256:a6bf3f88c554a2b653af81f3204491c818ae2ac6fbc09e76ef4773351292bc92", size = 2502363, upload-time = "2026-01-05T10:45:20.593Z" }, + { url = "https://files.pythonhosted.org/packages/65/71/0670843133a43d43070abeb1949abfdef12a86d490bea9cd9e18e37c5ff7/tokenizers-0.22.2-cp39-abi3-win_amd64.whl", hash = "sha256:c9ea31edff2968b44a88f97d784c2f16dc0729b8b143ed004699ebca91f05c48", size = 2747786, upload-time = "2026-01-05T10:45:18.411Z" }, + { url = "https://files.pythonhosted.org/packages/72/f4/0de46cfa12cdcbcd464cc59fde36912af405696f687e53a091fb432f694c/tokenizers-0.22.2-cp39-abi3-win_arm64.whl", hash = "sha256:9ce725d22864a1e965217204946f830c37876eee3b2ba6fc6255e8e903d5fcbc", size = 2612133, upload-time = "2026-01-05T10:45:17.232Z" }, +] + +[[package]] +name = "torch" +version = "2.7.0+cu126" +source = { registry = "https://download.pytorch.org/whl/cu126" } +dependencies = [ + { name = "filelock" }, + { name = "fsspec" }, + { name = "jinja2" }, + { name = "networkx" }, + { name = "nvidia-cublas-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cuda-cupti-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cuda-nvrtc-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cuda-runtime-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cudnn-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cufft-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cufile-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-curand-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cusolver-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cusparse-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cusparselt-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-nccl-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-nvjitlink-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-nvtx-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "setuptools" }, + { name = "sympy" }, + { name = "triton", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "typing-extensions" }, +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cu126/torch-2.7.0%2Bcu126-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:4933a51bfb906f34b44c23c6ea28fdfef5bf14a3c79a43d5d798285e29eba295", upload-time = "2025-04-22T18:17:24Z" }, + { url = "https://download-r2.pytorch.org/whl/cu126/torch-2.7.0%2Bcu126-cp312-cp312-win_amd64.whl", hash = "sha256:30bd9e92038c391b3b08b541c9bc803cb54e45fda63b61f7469bba6de372b065", upload-time = "2025-04-22T18:17:29Z" }, +] + +[[package]] +name = "torch-cluster" +version = "1.6.3+pt27cu126" +source = { registry = "https://data.pyg.org/whl/torch-2.7.0+cu126.html" } +dependencies = [ + { name = "scipy" }, +] +wheels = [ + { url = "https://data.pyg.org/whl/torch-2.7.0%2Bcu126/torch_cluster-1.6.3%2Bpt27cu126-cp312-cp312-linux_x86_64.whl" }, + { url = "https://data.pyg.org/whl/torch-2.7.0%2Bcu126/torch_cluster-1.6.3%2Bpt27cu126-cp312-cp312-win_amd64.whl" }, +] + +[[package]] +name = "torch-scatter" +version = "2.1.2+pt27cu126" +source = { registry = "https://data.pyg.org/whl/torch-2.7.0+cu126.html" } +wheels = [ + { url = "https://data.pyg.org/whl/torch-2.7.0%2Bcu126/torch_scatter-2.1.2%2Bpt27cu126-cp312-cp312-linux_x86_64.whl" }, + { url = "https://data.pyg.org/whl/torch-2.7.0%2Bcu126/torch_scatter-2.1.2%2Bpt27cu126-cp312-cp312-win_amd64.whl" }, +] + +[[package]] +name = "torchvision" +version = "0.22.0+cu126" +source = { registry = "https://download.pytorch.org/whl/cu126" } +dependencies = [ + { name = "numpy" }, + { name = "pillow" }, + { name = "torch" }, +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cu126/torchvision-0.22.0%2Bcu126-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:d155ffb659357747ef6697aca41dffd6fc8831d73f7b1b91d142ce2cd6e490c9", upload-time = "2025-04-22T18:30:19Z" }, + { url = "https://download-r2.pytorch.org/whl/cu126/torchvision-0.22.0%2Bcu126-cp312-cp312-win_amd64.whl", hash = "sha256:62ecdbdd53a467f17923a8237d0396a930a4c65c2377f9afc56872d74d99522f", upload-time = "2025-04-22T18:30:19Z" }, +] + +[[package]] +name = "tqdm" +version = "4.70.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/21/3b/6c24bec5be5e743ffd99576daa5cc077722fc7d5bbc00bd133fa0c698dc6/tqdm-4.70.0.tar.gz", hash = "sha256:55b0b0dbd97462d06ebee91e4dac24ed4d4702be82b24f07e6c1d27e08cea220", size = 795438, upload-time = "2026-07-27T11:33:15.271Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f9/1c/01bfd571a64e7f270e6bab5e33777debe0edc56759233ce84f27dec92d14/tqdm-4.70.0-py3-none-any.whl", hash = "sha256:7f585706bfddbdebf89daac705b2dfcc16890130727d3197ca62c732b4310953", size = 80184, upload-time = "2026-07-27T11:33:13.167Z" }, +] + +[[package]] +name = "traitlets" +version = "5.16.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2c/2e/a7fbfe268c8a3b32546930c0297c101d65a4a14c304ad5790a9f478f0e4e/traitlets-5.16.1.tar.gz", hash = "sha256:ed900c2b631aa3a112811139fa97b8d2c3bad5e989656bba4b7e52c7852c18c1", size = 166137, upload-time = "2026-08-03T08:32:36.848Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ad/66/0d785f0bc5e4315a96c989bb476d0fc07ea4f85132550c7b156ca2035d52/traitlets-5.16.1-py3-none-any.whl", hash = "sha256:f775618166caa0396c8e337099240f2bd3e5e917d203b2e6fbe21a58d3cb1f6b", size = 86211, upload-time = "2026-08-03T08:32:34.48Z" }, +] + +[[package]] +name = "transformers" +version = "5.14.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "regex" }, + { name = "safetensors" }, + { name = "tokenizers" }, + { name = "tqdm" }, + { name = "typer" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5a/fb/2a2ba88f325e68a921d8b69ff63b477830b2e73ade9a3c8c8cab2f06d741/transformers-5.14.1.tar.gz", hash = "sha256:60d196c27781eacf8637e2b533f517582907ad6f9ae142046d6b69431a5b2173", size = 9295927, upload-time = "2026-07-16T09:41:57.773Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6f/67/8d85ca2323233ae3c0365a659c4e52ee1f587b440e4bc577e7d8e4416d0f/transformers-5.14.1-py3-none-any.whl", hash = "sha256:9db974c4079ede2d1a3ea7ca5a240df33f2cc26fc2b36ba64c5f2a4f43b6e725", size = 11625234, upload-time = "2026-07-16T09:41:54.143Z" }, +] + +[[package]] +name = "triton" +version = "3.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "setuptools" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/11/53/ce18470914ab6cfbec9384ee565d23c4d1c55f0548160b1c7b33000b11fd/triton-3.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b68c778f6c4218403a6bd01be7484f6dc9e20fe2083d22dd8aef33e3b87a10a3", size = 156504509, upload-time = "2025-04-09T20:27:40.413Z" }, +] + +[[package]] +name = "typer" +version = "0.27.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "rich" }, + { name = "shellingham" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ae/40/4a3db7990d1f62a53182aa96eaef57aeb2886a27f90a195bc66713565d31/typer-0.27.1.tar.gz", hash = "sha256:a79bef8469a79c45498e7b814ecf8d603cc7644e9acbd9e19cac0334240b18df", size = 203994, upload-time = "2026-08-03T14:41:03.438Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/43/89/9518bc0c3929bee36b3a4a8e3daddd6e03f92f9961c66d4983b837160543/typer-0.27.1-py3-none-any.whl", hash = "sha256:53150287edd11baeb4e4722c8e394fcdf8181c0ae89485cba8d25c778d5edd56", size = 122874, upload-time = "2026-08-03T14:41:04.391Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/6d/bc/4eae18cd40c65798a16267572ba346c11f599d44b01603dbd843342042bc/typing_inspection-0.4.3.tar.gz", hash = "sha256:c5f9ec1530b5c1e2c9bc34a84d9a3466ed1b2f3f2fa9f901368d9c5596210e4d", size = 76711, upload-time = "2026-08-10T09:39:18.063Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/42/f7/7a3935abdebd5cf18705a5f0335dd6a3a18bef3baa7cb9edc3b6b9922cc8/typing_inspection-0.4.3-py3-none-any.whl", hash = "sha256:5f42b23858a91e0b4ef521f5418f03a0da3c9216fd2995ef5e73463100e676cd", size = 14693, upload-time = "2026-08-10T09:39:16.693Z" }, +] + +[[package]] +name = "tzdata" +version = "2026.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/92/ff/5a28bdfd8c3ebec42564ac7d0e54ca3db65044a9314a97f9564fa7a1e926/tzdata-2026.3.tar.gz", hash = "sha256:4a1518b8993086a7982523e071643f3c0e5f213e75b21318e78bcabfff9d1415", size = 198674, upload-time = "2026-07-10T08:50:37.887Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/6d/b53b99a9f2766d095985947a5782f1702cabb129a34f7a802d7197af832f/tzdata-2026.3-py2.py3-none-any.whl", hash = "sha256:dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931", size = 348168, upload-time = "2026-07-10T08:50:36.46Z" }, +] + +[[package]] +name = "urllib3" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, +] + +[[package]] +name = "werkzeug" +version = "3.1.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/dd/b2/381be8cfdee792dd117872481b6e378f85c957dd7c5bca38897b08f765fd/werkzeug-3.1.8.tar.gz", hash = "sha256:9bad61a4268dac112f1c5cd4630a56ede601b6ed420300677a869083d70a4c44", size = 875852, upload-time = "2026-04-02T18:49:14.268Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/93/8c/2e650f2afeb7ee576912636c23ddb621c91ac6a98e66dc8d29c3c69446e1/werkzeug-3.1.8-py3-none-any.whl", hash = "sha256:63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50", size = 226459, upload-time = "2026-04-02T18:49:12.72Z" }, +] + +[[package]] +name = "yapf" +version = "0.43.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "platformdirs" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/23/97/b6f296d1e9cc1ec25c7604178b48532fa5901f721bcf1b8d8148b13e5588/yapf-0.43.0.tar.gz", hash = "sha256:00d3aa24bfedff9420b2e0d5d9f5ab6d9d4268e72afbf59bb3fa542781d5218e", size = 254907, upload-time = "2024-11-14T00:11:41.584Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/37/81/6acd6601f61e31cfb8729d3da6d5df966f80f374b78eff83760714487338/yapf-0.43.0-py3-none-any.whl", hash = "sha256:224faffbc39c428cb095818cf6ef5511fdab6f7430a10783fdfb292ccf2852ca", size = 256158, upload-time = "2024-11-14T00:11:39.37Z" }, +] + +[[package]] +name = "zipp" +version = "4.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b9/d8/eab98a517c14134c0b2eb4e2387bc5f457334293ec5d2dd3857ec2966802/zipp-4.1.0.tar.gz", hash = "sha256:4cb57381f544315db7688e976e922a2b18cdb513d21cc194eb42232ba2a3e602", size = 26214, upload-time = "2026-05-18T20:08:57.967Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3a/13/547360d81e6d88d58492968ffda9f9542854f11310ee556fef14260cc886/zipp-4.1.0-py3-none-any.whl", hash = "sha256:25ad4e16390cd314347dd8f1de67a2ac538ae658ed4ab9db16029c07c188e97f", size = 10238, upload-time = "2026-05-18T20:08:57.045Z" }, +] diff --git a/weights/model_best.pth b/weights/model_best.pth new file mode 100644 index 0000000000000000000000000000000000000000..5cf264bca52d623e35d1a5978ae3c9e14de703bc --- /dev/null +++ b/weights/model_best.pth @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b5aa2656c80c02b78f2c3a78087098bb50c777095c123c18295d92205e647659 +size 554610216