File size: 24,796 Bytes
26225c5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 |
# Visualization
We provide a tool for interactive 3D visualization of `Data` and `NAG` objects.
This allows you to inspect what your `Data` attributes look like, check whether your model's
predictions make sense, and even share your interactive visualizations as
simple, portable, HTML files.
> **Note**: If you are not familiar with our `Data` and `NAG` structures yet,
> check the [Data](docs/data_structures.md) documentation.
## Let the `show()` begin
Our `show()` function rules all visualizations. It can be used as follows:
```python
from src.visualization import show
# Visualize a Data object
show(data)
```
It is also possible to directly call `show()` as a `Data` or `NAG` method:
```python
# Visualize a Data object
data.show()
# Visualize a NAG object
nag.show()
```
Datasets inheriting from our `BaseDataset` class also have a `show_examples()`
method that returns some spherical samples for a given class, randomly picked
across the dataset.
```python
# Visualize `max_examples` random spherical samples of size `radius`,
# centered on points annotated as `label`
dataset.show_examples(label, radius=4, max_examples=5)
```
## Visualizing data attributes
Given a `Data` or a `NAG` object, `show()` will automatically look for and plot
some specific point attributes. It generates an interactive
[Plotly](https://plotly.com/python) graph with intuitive navigation controls and
buttons for accessing different **visualization modes 🔘**.
The `pos` attribute is used to plot point positions. The 🔘`Position RGB` mode
will display points colored with respect to their location in the scene.
<p align="center">
<img width="80%" src="../media/viz_pos.png">
</p>
If the points carry colors in the `rgb` attribute, the 🔘`RGB` mode will display
them.
<p align="center">
<img width="80%" src="../media/viz_rgb.png">
</p>
If the points carry semantic segmentation labels in the `y` attribute, the
🔘`Labels` mode will display them. In addition, `show(class_colors=..., class_names=...)`
can be passed to adjust the color palette and the name displayed
when mouse hovering the graph. Similarly, if the points carry panoptic
segmentation labels in the `obj` attribute, the 🔘`Panoptic` mode will show them.
Passing `show(stuff_classes=..., num_classes=...)` allows for adjusting how
_thing_, _stuff_, and _void_ classes are displayed.
<p align="center">
<img width="80%" src="../media/viz_y.png">
</p>
If the points carry some features in the `x` attribute, the 🔘`Features 3D` mode
will display them.
> **Note**: This only supports point-wise features stored as 1D or 2D
> tensors. If the tensor contains only 1 channel, the attribute will be
> represented with a grayscale colormap. If the tensor contains 2 or 3 channels,
> these will be represented as RGB, with an additional all-1 channel if need be.
> If the tensor contains more than 3 channels, a PCA projection to RGB will be
> shown. In any case, the attribute values will be rescaled with respect to their
> statistics before visualization, meaning that colors may not compare between two
> different plots.
<p align="center">
<img width="80%" src="../media/viz_x.png">
</p>
If the points carry superpoint partition indices in the `super_index` attribute,
the 🔘`Level i` mode will display partition level `i`. Several options can be
passed to `show()` to adjust how partitions are shown: with centroids, with
horizontal graph edges, with vertical graph edges, with feature-colored edges,
etc. See the
[exhaustive list of parameters of `show()`](#exhaustive-parameter-list-for-show)
for more details.
<p align="center">
<img width="80%" src="../media/viz_p2.png">
</p>
If the points carry semantic segmentation predictions in the `semantic_pred`
attribute, the 🔘`Semantic Pred.` mode will display them. Besides, if both
`semantic_pred` and `y` can be found in the input, the 🔘`Error` button can be
used to highlight the erroneous predictions, as shown in red here below.
Similarly, `panoptic_pred` attributes can be visualized in the
🔘`Panoptic Pred.` mode.
<p align="center">
<img width="80%" src="../media/viz_errors.png">
</p>
> **Tip 💡**: **Want to visualize other point attributes stored in your `Data`
> object ?** Pass them to `show(keys=...)` to have them displayed ! The same
> coloring mechanisms as mentioned above for the `x` attribute will be used.
## Points selection for visualization
Our [Plotly](https://plotly.com/python)-based visualization is mostly meant for
quick interactive experiments with up to ~10⁵ points. While it is possible to
visualize more points at once, Plotly will take a while to generate the plot. We
provide several tools for reducing the number of point displayed, hence
accelerating the visualization speed.
Say you have the following large-scale visualization.
<p align="center">
<img width="80%" src="../media/viz_100k.png">
</p>
You can choose to alleviate the visualization time by sampling some points.
This can be done in one of two ways:
- random sampling with respect to a maximum point budget with:
```python
show(max_points=...)
```
- grid sampling (ie voxelization) with:
```python
show(voxel=...)
```
This can make your visualization run faster, at the expense of resolution.
<p align="center">
<img width="80%" src="../media/viz_10k.png">
</p>
Another strategy is to simply visualize a smaller area. To this end, one can
visualize only a spherical crop of the scene with:
```python
show(radius=..., center=...)
```
<p align="center">
<img width="80%" src="../media/viz_radius_center.png">
</p>
We also provide a tool for displaying point selections with:
```python
show(select=..., alpha=...)
```
This internally calls `NAG.select()` and `Data.select()` and applies an
`alpha` opacity to the non-selected points. This is typically useful for
visualizing binary masks across the scene. Here is, for example, the mask
corresponding to the previous spherical cropping operation.
<p align="center">
<img width="80%" src="../media/viz_select.png">
</p>
## Sharing your visualization as an HTML
You can export your Plotly figure as an HTML file which can be conveniently
opened by virtually anyone with a browser. To this end, simply specify a `path`
to which your HTML should be saved, along with a figure `title`, if need be:
```python
show(path=..., title=...)
```
## Exhaustive parameter list for `show()`
Here is the list of supported parameters for `show()`:
| Parameter | Type | Description |
|:------------------|:---------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `input` | `Data` or `NAG` | `Data` or `NAG` object to visualize |
| `keys` | `List(str)` or `str` | By default, the following attributes will be parsed in `input` for visualization {`pos`, `rgb`, `y`, `obj`, `semantic_pred`, `obj_pred`}. Yet, if `input` contains other attributes that you want to visualize, these can be passed as `keys`. This only supports point-wise attributes stored as 1D or 2D tensors. If the tensor contains only 1 channel, the attribute will be represented with a grayscale colormap. If the tensor |contains 2 or 3 channels, these will be represented as RGB, with an additional all-1 channel if need be. If the tensor contains more than 3 channels, a PCA projection to RGB will be shown. In any case, the attribute values will be rescaled with respect to their statistics before visualization, meaning that colors may not compare between two different plots |
| `figsize` | `int` | Figure dimensions will be `(figsize, figsize/2)` if `width` and `height` are not specified |
| `width` | `int` | Figure width |
| `height` | `int` | Figure height |
| `class_names` | `List(str)` | Names for point labels found in attributes `y` and `semantic_pred` |
| `class_colors` | `List(List(int, int, int))` | Colors palette for point labels found in attributes `y` and `semantic_pred` |
| `stuff_classes` | `List(int)` | Semantic labels of the classes considered as `stuff` for instance and panoptic segmentation. If `y` and `obj` are found in the point attributes, the stuff annotations will appear accordingly. Otherwise, stuff instance labeling will appear as any other object |
| `num_classes` | `int` | Number of valid classes. By convention, we assume `y ∈ [0, num_classes-1]` are VALID LABELS, while `y < 0` AND `y >= num_classes` ARE VOID LABELS |
| `hide_void_pred` | `bool` | Whether predictions on points labeled as VOID be visualized |
| `voxel` | `float` | Voxel size to subsample the point cloud to facilitate visualization |
| `max_points` | `int` | Maximum number of points displayed to facilitate visualization |
| `point_size` | `int` or `float` | Size of point markers |
| `centroid_size` | `int` or `float` | Size of superpoint markers |
| `error_color` | `List(int, int, int)` | Color used to identify mis-predicted points |
| `centroids` | `bool` | Whether superpoint centroids should be displayed |
| `h_edge` | `bool` | Whether horizontal edges should be displayed (only if `centroids=True`) |
| `h_edge_attr` | `bool` | Whether the edges should be colored by their features found in `edge_attr` (only if `h_edge=True`) |
| `h_edge_width` | `float` | Width of the horizontal edges, if `h_edge=True`. Defaults to `None`, in which case `point_size` will be used for the edge width |
| `v_edge` | `bool` | Whether vertical edges should be displayed (only if `centroids=True` and `gap` is not `None`) |
| `v_edge_width` | `float` | Width of the vertical edges, if `v_edge=True`. Defaults to `None`, in which case `point_size` will be used for the edge width |
| `gap` | `List(float, float, float)` | If `None`, the hierarchical graphs will be overlaid on the points. If not `None`, a 3D tensor indicating the offset by which the hierarchical graphs should be plotted |
| `radius` | `float` | If not `None`, only visualize a spherical sampling of the input data, centered on `center` and with size `radius`. This option is not compatible with `select` |
| `center` | `List(float, float, float)` | If `radius` is provided, only visualize a spherical sampling of the input data, centered on `center` and with size `radius`. If `None`, the center of the scene will be used |
| `select` | `Tuple(int, Tensor)` | If not `None`, will call `Data.select(*select)` or `NAG.select(*select)` on the input data (depending on its nature) and the coloring schemes will illustrate it. This option is not compatible with `radius` |
| `alpha` | `float` | Rules the whitening of selected points, nodes and edges (only if select is not `None`) |
| `alpha_super` | `float` | Rules the whitening of superpoints (only if select is not `None`). If `None`, alpha will be used as fallback |
| `alpha_stuff` | `float` | Rules the whitening of stuff points (only if the input points have `obj` and `semantic_pred` attributes, and `stuff_classes` or `num_classes` is specified). If `None`, `alpha` will be used as fallback |
| `point_symbol` | `str` | Marker symbol used for points. Must be one of `{'circle', 'circle-open', 'square', 'square-open', 'diamond', 'diamond-open', 'cross', 'x'}`. Defaults to `'circle'` |
| `centroid_symbol` | `str` | Marker symbol used for centroids. Must be one of `{'circle', 'circle-open', 'square', 'square-open', 'diamond', 'diamond-open', 'cross', 'x'}`. Defaults to `'circle'` |
| `colorscale` | `str` | Plotly colorscale used for coloring 1D continuous features. See https://plotly.com/python/builtin-colorscales for options |
| `path` | `str` | If not `None`, `path` will be used for storing the Plotly figure as an HTML file |
| `title` | `str` | Title for the Plotly figure |
| `pt_path` | `str` | Path to save the visualization-ready `Data` object as a `*.pt`. In this `Data` object, the `pos` and all `*color*` attributes will be saved, the rest is discarded. This is typically useful for exporting the visualization layers to another visualization tool |
| `kwargs` | | |
## 🚀 Getting started with visualization
See the [`notebooks/demo_nag.ipynb`](../notebooks/demo_nag.ipynb) notebook to play with our visualization tool
on a provided `NAG`, without needing a whole dataset.
For more information, have a look at the code in `src.visualization`, it is
fairly commented and should help you gain a deeper understanding of the
visualization mechanisms.
|