| # Apply Net |
|
|
| `apply_net` is a tool to print or visualize DensePose results on a set of images. |
| It has two modes: `dump` to save DensePose model results to a pickle file |
| and `show` to visualize them on images. |
|
|
| The `image.jpg` file that is used as an example in this doc can be found [here](http://images.cocodataset.org/train2017/000000117508.jpg) |
|
|
| ## Dump Mode |
|
|
| The general command form is: |
| ```bash |
| python apply_net.py dump [-h] [-v] [--output <dump_file>] <config> <model> <input> |
| ``` |
|
|
| There are three mandatory arguments: |
| - `<config>`, configuration file for a given model; |
| - `<model>`, model file with trained parameters |
| - `<input>`, input image file name, pattern or folder |
|
|
| One can additionally provide `--output` argument to define the output file name, |
| which defaults to `output.pkl`. |
|
|
|
|
| Examples: |
|
|
| 1. Dump results of the [R_50_FPN_s1x](https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl) DensePose model for images in a folder `images` to file `dump.pkl`: |
| ```bash |
| python apply_net.py dump configs/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \ |
| images --output dump.pkl -v |
| ``` |
|
|
| 2. Dump results of the [R_50_FPN_s1x](https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl) DensePose model for images with file name matching a pattern `image*.jpg` to file `results.pkl`: |
| ```bash |
| python apply_net.py dump configs/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \ |
| "image*.jpg" --output results.pkl -v |
| ``` |
|
|
| If you want to load the pickle file generated by the above command: |
| ``` |
| # make sure DensePose is in your PYTHONPATH, or use the following line to add it: |
| sys.path.append("/your_detectron2_path/detectron2_repo/projects/DensePose/") |
| |
| f = open('/your_result_path/results.pkl', 'rb') |
| data = pickle.load(f) |
| ``` |
|
|
| The file `results.pkl` contains the list of results per image, for each image the result is a dictionary. |
|
|
| **If you use a [IUV model](DENSEPOSE_IUV.md#-model-zoo-and-baselines)**, the dumped data will have the following format: |
|
|
| ``` |
| data: [{'file_name': '/your_path/image1.jpg', |
| 'scores': tensor([0.9884]), |
| 'pred_boxes_XYXY': tensor([[ 69.6114, 0.0000, 706.9797, 706.0000]]), |
| 'pred_densepose': [DensePoseChartResultWithConfidences(labels=tensor(...), uv=tensor(...), sigma_1=None, |
| sigma_2=None, kappa_u=None, kappa_v=None, fine_segm_confidence=None, coarse_segm_confidence=None), |
| DensePoseChartResultWithConfidences, ...] |
| } |
| {'file_name': '/your_path/image2.jpg', |
| 'scores': tensor([0.9999, 0.5373, 0.3991]), |
| 'pred_boxes_XYXY': tensor([[ 59.5734, 7.7535, 579.9311, 932.3619], |
| [612.9418, 686.1254, 612.9999, 704.6053], |
| [164.5081, 407.4034, 598.3944, 920.4266]]), |
| 'pred_densepose': [DensePoseChartResultWithConfidences(labels=tensor(...), uv=tensor(...), sigma_1=None, |
| sigma_2=None, kappa_u=None, kappa_v=None, fine_segm_confidence=None, coarse_segm_confidence=None), |
| DensePoseChartResultWithConfidences, ...] |
| }] |
| ``` |
|
|
| `DensePoseChartResultWithConfidences` contains the following fields: |
| - `labels` - a tensor of size `[H, W]` of type `torch.long` which contains fine segmentation labels (previously called `I`) |
| - `uv` - a tensor of size `[2, H, W]` of type `torch.float` which contains `U` and `V` coordinates |
| - various optional confidence-related fields (`sigma_1`, `sigma_2`, `kappa_u`, `kappa_v`, `fine_segm_confidence`, `coarse_segm_confidence`) |
|
|
|
|
| **If you use a [CSE model](DENSEPOSE_CSE.md#-model-zoo-and-baselines)**, the dumped data will have the following format: |
| ``` |
| data: [{'file_name': '/your_path/image1.jpg', |
| 'scores': tensor([0.9984, 0.9961]), |
| 'pred_boxes_XYXY': tensor([[480.0093, 461.0796, 698.3614, 696.1011], |
| [78.1589, 168.6614, 307.1287, 653.8522]]), |
| 'pred_densepose': DensePoseEmbeddingPredictorOutput(embedding=tensor(...), coarse_segm=tensor(...))} |
| {'file_name': '/your_path/image2.jpg', |
| 'scores': tensor([0.9189, 0.9491]), |
| 'pred_boxes_XYXY': tensor([[734.9685, 534.2003, 287.3923, 254.8859], |
| [434.2853, 765.1219, 132.1029, 867.9283]]), |
| 'pred_densepose': DensePoseEmbeddingPredictorOutput(embedding=tensor(...), coarse_segm=tensor(...))}] |
| ``` |
|
|
| `DensePoseEmbeddingPredictorOutput` contains the following fields: |
| - `embedding` - a tensor of size `[N, D, sz, sz]` of type `torch.float`, which contains embeddings of size `D` of the `N` detections in the image |
| - `coarse_segm` - a tensor of size `[N, 2, sz, sz]` of type `torch.float` which contains segmentation scores of the `N` detections in the image; e.g. a mask can be obtained by `coarse_segm.argmax(dim=1)` |
|
|
| `sz` is a fixed size for the tensors; you can resize them to the size of the bounding box, if needed |
|
|
| We can use the following code, to parse the outputs of the first |
| detected instance on the first image (IUV model). |
| ``` |
| img_id, instance_id = 0, 0 # Look at the first image and the first detected instance |
| bbox_xyxy = data[img_id]['pred_boxes_XYXY'][instance_id] |
| result = data[img_id]['pred_densepose'][instance_id] |
| uv = result.uv |
| ``` |
| The array `bbox_xyxy` contains (x0, y0, x1, y1) of the bounding box. |
|
|
|
|
| ## Visualization Mode |
|
|
| The general command form is: |
| ```bash |
| python apply_net.py show [-h] [-v] [--min_score <score>] [--nms_thresh <threshold>] [--output <image_file>] <config> <model> <input> <visualizations> |
| ``` |
|
|
| There are four mandatory arguments: |
| - `<config>`, configuration file for a given model; |
| - `<model>`, model file with trained parameters |
| - `<input>`, input image file name, pattern or folder |
| - `<visualizations>`, visualizations specifier; currently available visualizations are: |
| * `bbox` - bounding boxes of detected persons; |
| * `dp_segm` - segmentation masks for detected persons; |
| * `dp_u` - each body part is colored according to the estimated values of the |
| U coordinate in part parameterization; |
| * `dp_v` - each body part is colored according to the estimated values of the |
| V coordinate in part parameterization; |
| * `dp_contour` - plots contours with color-coded U and V coordinates; |
| * `dp_iuv_texture` - transfers the texture from a given texture image file to detected instances, in IUV mode; |
| * `dp_vertex` - plots the rainbow visualization of the closest vertices prediction for a given mesh, in CSE mode; |
| * `dp_cse_texture` - transfers the texture from a given list of texture image files (one from each human or animal mesh) to detected instances, in CSE mode |
| |
|
|
| One can additionally provide the following optional arguments: |
| - `--min_score` to only show detections with sufficient scores that are not lower than provided value |
| - `--nms_thresh` to additionally apply non-maximum suppression to detections at a given threshold |
| - `--output` to define visualization file name template, which defaults to `output.png`. |
| To distinguish output file names for different images, the tool appends 1-based entry index, |
| e.g. output.0001.png, output.0002.png, etc... |
| - `--texture_atlas` to define the texture atlas image for IUV texture transfer |
| - `--texture_atlases_map` to define the texture atlas images map (a dictionary `{mesh name: texture atlas image}`) for CSE texture transfer |
|
|
|
|
| The following examples show how to output results of a DensePose model |
| with ResNet-50 FPN backbone using different visualizations for image `image.jpg`: |
|
|
| 1. Show bounding box and segmentation: |
| ```bash |
| python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \ |
| image.jpg bbox,dp_segm -v |
| ``` |
|  |
|
|
| 2. Show bounding box and estimated U coordinates for body parts: |
| ```bash |
| python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \ |
| image.jpg bbox,dp_u -v |
| ``` |
|  |
|
|
| 3. Show bounding box and estimated V coordinates for body parts: |
| ```bash |
| python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \ |
| image.jpg bbox,dp_v -v |
| ``` |
|  |
|
|
| 4. Show bounding box and estimated U and V coordinates via contour plots: |
| ```bash |
| python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \ |
| image.jpg dp_contour,bbox -v |
| ``` |
|  |
|
|
| 5. Show bounding box and texture transfer: |
| ```bash |
| python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \ |
| image.jpg dp_iuv_texture,bbox --texture_atlas texture_from_SURREAL.jpg -v |
| ``` |
|  |
|
|
| 6. Show bounding box and CSE rainbow visualization: |
| ```bash |
| python apply_net.py show configs/cse/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/cse/densepose_rcnn_R_50_FPN_s1x/251155172/model_final_c4ea5f.pkl \ |
| image.jpg dp_vertex,bbox -v |
| ``` |
|  |
|
|
| 7. Show bounding box and CSE texture transfer: |
| ```bash |
| python apply_net.py show configs/cse/densepose_rcnn_R_50_FPN_s1x.yaml \ |
| https://dl.fbaipublicfiles.com/densepose/cse/densepose_rcnn_R_50_FPN_s1x/251155172/model_final_c4ea5f.pkl \ |
| image.jpg dp_cse_texture,bbox --texture_atlases_map '{"smpl_27554": "smpl_uvSnapshot_colors.jpg"}' -v |
| ``` |
|  |
|
|
| The texture files can be found in the `doc/images` folder |
|
|
