Apply Net

apply_net is a tool to print or visualize DensePose results on a set of images. It has two modes: dump to save DensePose model results to a pickle file and show to visualize them on images.

The image.jpg file that is used as an example in this doc can be found here

Dump Mode

The general command form is:

python apply_net.py dump [-h] [-v] [--output <dump_file>] <config> <model> <input>

There are three mandatory arguments:

<config>, configuration file for a given model;
<model>, model file with trained parameters
<input>, input image file name, pattern or folder

One can additionally provide --output argument to define the output file name, which defaults to output.pkl.

Examples:

Dump results of the R_50_FPN_s1x DensePose model for images in a folder images to file dump.pkl:

python apply_net.py dump configs/densepose_rcnn_R_50_FPN_s1x.yaml \
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \
images --output dump.pkl -v

Dump results of the R_50_FPN_s1x DensePose model for images with file name matching a pattern image*.jpg to file results.pkl:

python apply_net.py dump configs/densepose_rcnn_R_50_FPN_s1x.yaml \
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \
"image*.jpg" --output results.pkl -v

If you want to load the pickle file generated by the above command:

# make sure DensePose is in your PYTHONPATH, or use the following line to add it:
sys.path.append("/your_detectron2_path/detectron2_repo/projects/DensePose/")

f = open('/your_result_path/results.pkl', 'rb')
data = pickle.load(f)

The file results.pkl contains the list of results per image, for each image the result is a dictionary.

If you use a IUV model, the dumped data will have the following format:

data: [{'file_name': '/your_path/image1.jpg',
        'scores': tensor([0.9884]),
        'pred_boxes_XYXY': tensor([[ 69.6114,   0.0000, 706.9797, 706.0000]]),
        'pred_densepose': [DensePoseChartResultWithConfidences(labels=tensor(...), uv=tensor(...), sigma_1=None,
            sigma_2=None, kappa_u=None, kappa_v=None, fine_segm_confidence=None, coarse_segm_confidence=None),
            DensePoseChartResultWithConfidences, ...]
        }
       {'file_name': '/your_path/image2.jpg',
        'scores': tensor([0.9999, 0.5373, 0.3991]),
        'pred_boxes_XYXY': tensor([[ 59.5734,   7.7535, 579.9311, 932.3619],
                                   [612.9418, 686.1254, 612.9999, 704.6053],
                                   [164.5081, 407.4034, 598.3944, 920.4266]]),
        'pred_densepose': [DensePoseChartResultWithConfidences(labels=tensor(...), uv=tensor(...), sigma_1=None,
            sigma_2=None, kappa_u=None, kappa_v=None, fine_segm_confidence=None, coarse_segm_confidence=None),
            DensePoseChartResultWithConfidences, ...]
        }]

DensePoseChartResultWithConfidences contains the following fields:

labels - a tensor of size [H, W] of type torch.long which contains fine segmentation labels (previously called I)
uv - a tensor of size [2, H, W] of type torch.float which contains U and V coordinates
various optional confidence-related fields (sigma_1, sigma_2, kappa_u, kappa_v, fine_segm_confidence, coarse_segm_confidence)

If you use a CSE model, the dumped data will have the following format:

data: [{'file_name': '/your_path/image1.jpg',
        'scores': tensor([0.9984, 0.9961]),
        'pred_boxes_XYXY': tensor([[480.0093, 461.0796, 698.3614, 696.1011],
                                   [78.1589, 168.6614, 307.1287, 653.8522]]),
        'pred_densepose': DensePoseEmbeddingPredictorOutput(embedding=tensor(...), coarse_segm=tensor(...))}
        {'file_name': '/your_path/image2.jpg',
        'scores': tensor([0.9189, 0.9491]),
        'pred_boxes_XYXY': tensor([[734.9685, 534.2003, 287.3923, 254.8859],
                                   [434.2853, 765.1219, 132.1029, 867.9283]]),
        'pred_densepose': DensePoseEmbeddingPredictorOutput(embedding=tensor(...), coarse_segm=tensor(...))}]

DensePoseEmbeddingPredictorOutput contains the following fields:

embedding - a tensor of size [N, D, sz, sz] of type torch.float, which contains embeddings of size D of the N detections in the image
coarse_segm - a tensor of size [N, 2, sz, sz] of type torch.float which contains segmentation scores of the N detections in the image; e.g. a mask can be obtained by coarse_segm.argmax(dim=1)

sz is a fixed size for the tensors; you can resize them to the size of the bounding box, if needed

We can use the following code, to parse the outputs of the first detected instance on the first image (IUV model).

img_id, instance_id = 0, 0  # Look at the first image and the first detected instance
bbox_xyxy = data[img_id]['pred_boxes_XYXY'][instance_id]
result = data[img_id]['pred_densepose'][instance_id]
uv = result.uv

The array bbox_xyxy contains (x0, y0, x1, y1) of the bounding box.

Visualization Mode

The general command form is:

python apply_net.py show [-h] [-v] [--min_score <score>] [--nms_thresh <threshold>] [--output <image_file>] <config> <model> <input> <visualizations>

There are four mandatory arguments:

<config>, configuration file for a given model;
<model>, model file with trained parameters
<input>, input image file name, pattern or folder
<visualizations>, visualizations specifier; currently available visualizations are:
- bbox - bounding boxes of detected persons;
- dp_segm - segmentation masks for detected persons;
- dp_u - each body part is colored according to the estimated values of the U coordinate in part parameterization;
- dp_v - each body part is colored according to the estimated values of the V coordinate in part parameterization;
- dp_contour - plots contours with color-coded U and V coordinates;
- dp_iuv_texture - transfers the texture from a given texture image file to detected instances, in IUV mode;
- dp_vertex - plots the rainbow visualization of the closest vertices prediction for a given mesh, in CSE mode;
- dp_cse_texture - transfers the texture from a given list of texture image files (one from each human or animal mesh) to detected instances, in CSE mode

One can additionally provide the following optional arguments:

--min_score to only show detections with sufficient scores that are not lower than provided value
--nms_thresh to additionally apply non-maximum suppression to detections at a given threshold
--output to define visualization file name template, which defaults to output.png. To distinguish output file names for different images, the tool appends 1-based entry index, e.g. output.0001.png, output.0002.png, etc...
--texture_atlas to define the texture atlas image for IUV texture transfer
--texture_atlases_map to define the texture atlas images map (a dictionary {mesh name: texture atlas image}) for CSE texture transfer

The following examples show how to output results of a DensePose model with ResNet-50 FPN backbone using different visualizations for image image.jpg:

Show bounding box and segmentation:

python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml \
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \
image.jpg bbox,dp_segm -v

Show bounding box and estimated U coordinates for body parts:

python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml  \
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \
image.jpg bbox,dp_u -v

Show bounding box and estimated V coordinates for body parts:

python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml  \
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \
image.jpg bbox,dp_v -v

Show bounding box and estimated U and V coordinates via contour plots:

python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml  \
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \
image.jpg dp_contour,bbox -v

Show bounding box and texture transfer:

python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml  \
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_50_FPN_s1x/165712039/model_final_162be9.pkl \
image.jpg dp_iuv_texture,bbox --texture_atlas texture_from_SURREAL.jpg -v

Show bounding box and CSE rainbow visualization:

python apply_net.py show configs/cse/densepose_rcnn_R_50_FPN_s1x.yaml  \
https://dl.fbaipublicfiles.com/densepose/cse/densepose_rcnn_R_50_FPN_s1x/251155172/model_final_c4ea5f.pkl \
image.jpg dp_vertex,bbox -v

Show bounding box and CSE texture transfer:

python apply_net.py show configs/cse/densepose_rcnn_R_50_FPN_s1x.yaml  \
https://dl.fbaipublicfiles.com/densepose/cse/densepose_rcnn_R_50_FPN_s1x/251155172/model_final_c4ea5f.pkl \
image.jpg dp_cse_texture,bbox  --texture_atlases_map '{"smpl_27554": "smpl_uvSnapshot_colors.jpg"}' -v

The texture files can be found in the doc/images folder