Upload 11 files

deployment/Instance segmentation task/README.md

@@ -0,0 +1,151 @@
+# Exportable code
+
+Exportable code is a .zip archive that contains simple demo to get and visualize result of model inference.
+
+## Structure of generated zip
+
+- `README.md`
+- model
+  - `model.xml`
+  - `model.bin`
+  - `config.json`
+- python
+  - model_wrappers (Optional)
+    - `__init__.py`
+    - model_wrappers required to run demo
+  - `LICENSE`
+  - `demo.py`
+  - `requirements.txt`
+
+> **NOTE**: Zip archive contains model_wrappers when [ModelAPI](https://github.com/openvinotoolkit/model_api) has no appropriate standard model wrapper for the model.
+
+## Prerequisites
+
+- [Python 3.8](https://www.python.org/downloads/)
+- [Git](https://git-scm.com/)
+
+## Install requirements to run demo
+
+1. Install [prerequisites](#prerequisites). You may also need to [install pip](https://pip.pypa.io/en/stable/installation/). For example, on Ubuntu execute the following command to get pip installed:
+
+   ```bash
+   sudo apt install python3-pip
+   ```
+
+1. Create clean virtual environment:
+
+   One of the possible ways for creating a virtual environment is to use `virtualenv`:
+
+   ```bash
+   python -m pip install virtualenv
+   python -m virtualenv <directory_for_environment>
+   ```
+
+   Before starting to work inside virtual environment, it should be activated:
+
+   On Linux and macOS:
+
+   ```bash
+   source <directory_for_environment>/bin/activate
+   ```
+
+   On Windows:
+
+   ```bash
+   .\<directory_for_environment>\Scripts\activate
+   ```
+
+   Please make sure that the environment contains [wheel](https://pypi.org/project/wheel/) by calling the following command:
+
+   ```bash
+   python -m pip install wheel
+   ```
+
+   > **NOTE**: On Linux and macOS, you may need to type `python3` instead of `python`.
+
+1. Install requirements in the environment:
+
+   ```bash
+   python -m pip install -r requirements.txt
+   ```
+
+## Usecase
+
+1. Running the `demo.py` application with the `-h` option yields the following usage message:
+
+   ```bash
+   usage: demo.py [-h] -i INPUT -m MODELS [MODELS ...] [-it {sync,async}] [-l] [--no_show] [-d {CPU,GPU}] [--output OUTPUT]
+
+   Options:
+   -h, --help            Show this help message and exit.
+   -i INPUT, --input INPUT
+                           Required. An input to process. The input must be a single image, a folder of images, video file or camera id.
+   -m MODELS [MODELS ...], --models MODELS [MODELS ...]
+                           Optional. Path to directory with trained model and configuration file. If you provide several models you will start the task chain pipeline with the provided models in the order in which they were specified. Default value points to deployed model folder '../model'.
+   -it {sync,async}, --inference_type {sync,async}
+                           Optional. Type of inference for single model.
+   -l, --loop            Optional. Enable reading the input in a loop.
+   --no_show             Optional. Disables showing inference results on UI.
+   -d {CPU,GPU}, --device {CPU,GPU}
+                           Optional. Device to infer the model.
+   --output OUTPUT       Optional. Output path to save input data with predictions.
+   ```
+
+2. As a `model` parameter the default value `../model` will be used. Or you can specify the other path to the model directory from generated zip. You can pass as `input` a single image, a folder of images, a video file, or a web camera id. So you can use the following command to do inference with a pre-trained model:
+
+   ```bash
+   python3 demo.py -i <path_to_video>/inputVideo.mp4
+   ```
+
+   You can press `Q` to stop inference during demo running.
+
+   > **NOTE**: If you provide a single image as input, the demo processes and renders it quickly, then exits. To continuously
+   > visualize inference results on the screen, apply the `--loop` option, which enforces processing a single image in a loop.
+   > In this case, you can stop the demo by pressing `Q` button or killing the process in the terminal (`Ctrl+C` for Linux).
+   >
+   > **NOTE**: Default configuration contains info about pre- and post processing for inference and is guaranteed to be correct.
+   > Also you can change `config.json` that specifies the confidence threshold and color for each class visualization, but any
+   > changes should be made with caution.
+
+3. To save inferenced results with predictions on it, you can specify the folder path, using `--output`.
+   It works for images, videos, image folders and web cameras. To prevent issues, do not specify it together with a `--loop` parameter.
+
+   ```bash
+   python3 demo.py \
+      --input <path_to_image>/inputImage.jpg \
+      --models ../model \
+      --output resulted_images
+   ```
+
+4. To run a demo on a web camera, you need to know its ID.
+   You can check a list of camera devices by running this command line on Linux system:
+
+   ```bash
+   sudo apt-get install v4l-utils
+   v4l2-ctl --list-devices
+   ```
+
+   The output will look like this:
+
+   ```bash
+   Integrated Camera (usb-0000:00:1a.0-1.6):
+      /dev/video0
+   ```
+
+   After that, you can use this `/dev/video0` as a camera ID for `--input`.
+
+## Troubleshooting
+
+1. If you have access to the Internet through the proxy server only, please use pip with proxy call as demonstrated by command below:
+
+   ```bash
+   python -m pip install --proxy http://<usr_name>:<password>@<proxyserver_name>:<port#> <pkg_name>
+   ```
+
+1. If you use Anaconda environment, you should consider that OpenVINO has limited [Conda support](https://docs.openvino.ai/2021.4/openvino_docs_install_guides_installing_openvino_conda.html) for Python 3.6 and 3.7 versions only. But the demo package requires python 3.8. So please use other tools to create the environment (like `venv` or `virtualenv`) and use `pip` as a package manager.
+
+1. If you have problems when you try to use `pip install` command, please update pip version by following command:
+
+   ```bash
+   python -m pip install --upgrade pip
+   ```

deployment/Instance segmentation task/model.json

@@ -1,37 +1,26 @@
 {
-   "id": "6483248259c02bd70e8df1f4",
-   "name": "MaskRCNN-ResNet50 OpenVINO INT8",
-   "version": 1,
-   "creation_date": "2023-06-09T13:09:22.699000+00:00",
+   "id": "660fb1e0824143cee70ff576",
+   "name": "MaskRCNN-ResNet50 OpenVINO FP16",
+   "version": 3,
+   "creation_date": "2024-04-05T08:10:08.633000+00:00",
    "model_format": "OpenVINO",
    "precision": [
-      "INT8"
+      "FP16"
    ],
    "has_xai_head": false,
    "target_device": "CPU",
    "target_device_type": null,
    "performance": {
-      "score": 0.9397590361445782
+      "score": 0.9641873278236913
    },
-   "size": 247759204,
+   "size": 90856771,
    "latency": 0,
    "fps_throughput": 0,
-   "optimization_type": "NNCF",
+   "optimization_type": "MO",
    "optimization_objectives": {},
    "model_status": "SUCCESS",
-   "configurations": [
-      {
-         "name": "max_accuracy_drop",
-         "value": 0.01
-      },
-      {
-         "name": "filter_pruning",
-         "value": false
-      }
-   ],
-   "previous_revision_id": "6483248259c02bd70e8df1f5",
-   "previous_trained_revision_id": "648311e459c02bd70e8db073",
-   "optimization_methods": [
-      "QUANTIZATION"
-   ]
+   "configurations": [],
+   "previous_revision_id": "660fb1e0824143cee70ff573",
+   "previous_trained_revision_id": "660fb1e0824143cee70ff573",
+   "optimization_methods": []
 }

deployment/Instance segmentation task/model/config.json

@@ -3,9 +3,8 @@
     "converter_type": "INSTANCE_SEGMENTATION",
     "model_parameters": {
         "result_based_confidence_threshold": true,
-        "confidence_threshold": 0.8500000238418579,
+        "confidence_threshold": 0.824999988079071,
         "use_ellipse_shapes": false,
-        "resize_type": "fit_to_window",
         "labels": {
             "label_tree": {
                 "type": "tree",

deployment/Instance segmentation task/model/model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3128a90931c4b7b51ec0218fe952bac97d13b1763fc1f9cef1ecb620a22412e3
-size 56328440
+oid sha256:22e3375e168a776d41eab926a797814a7c2e9cc5f2cd5c6134ca9820f191725a
+size 90038530

deployment/Instance segmentation task/python/demo.py

@@ -41,11 +41,12 @@ def build_argparser():
     args.add_argument(
         "-m",
         "--models",
-        help="Required. Path to directory with trained model and configuration file. "
+        help="Optional. Path to directory with trained model and configuration file. "
         "If you provide several models you will start the task chain pipeline with "
-        "the provided models in the order in which they were specified.",
+        "the provided models in the order in which they were specified. Default value "
+        "points to deployed model folder '../model'.",
         nargs="+",
-        required=True,
+        default=[Path("../model")],
         type=Path,
     )
     args.add_argument(

deployment/Instance segmentation task/python/model_wrappers/openvino_models.py

@@ -16,16 +16,11 @@
 
 from typing import Dict
 
+import cv2
 import numpy as np
-
-try:
-    from openvino.model_zoo.model_api.models.instance_segmentation import MaskRCNNModel
-    from openvino.model_zoo.model_api.models.ssd import SSD, find_layer_by_name
-    from openvino.model_zoo.model_api.models.utils import Detection
-except ImportError as e:
-    import warnings
-
-    warnings.warn(f"{e}: ModelAPI was not found.")
+from openvino.model_api.models.instance_segmentation import MaskRCNNModel, _expand_box, _segm_postprocess
+from openvino.model_api.models.ssd import SSD, find_layer_by_name
+from openvino.model_api.models.utils import Detection
 
 
 class OTXMaskRCNNModel(MaskRCNNModel):
@@ -92,24 +87,105 @@ class OTXMaskRCNNModel(MaskRCNNModel):
         masks = masks[detections_filter]
         classes = classes[detections_filter]
 
-        scale_x = meta["resized_shape"][1] / meta["original_shape"][1]
-        scale_y = meta["resized_shape"][0] / meta["original_shape"][0]
-        boxes[:, 0::2] /= scale_x
-        boxes[:, 1::2] /= scale_y
+        inputImgWidth, inputImgHeight = (
+            meta["original_shape"][1],
+            meta["original_shape"][0],
+        )
+        invertedScaleX, invertedScaleY = (
+            inputImgWidth / self.orig_width,
+            inputImgHeight / self.orig_height,
+        )
+        padLeft, padTop = 0, 0
+        if "fit_to_window" == self.resize_type or "fit_to_window_letterbox" == self.resize_type:
+            invertedScaleX = invertedScaleY = max(invertedScaleX, invertedScaleY)
+            if "fit_to_window_letterbox" == self.resize_type:
+                padLeft = (self.orig_width - round(inputImgWidth / invertedScaleX)) // 2
+                padTop = (self.orig_height - round(inputImgHeight / invertedScaleY)) // 2
+
+        boxes -= (padLeft, padTop, padLeft, padTop)
+        boxes *= (invertedScaleX, invertedScaleY, invertedScaleX, invertedScaleY)
+        np.around(boxes, out=boxes)
+        np.clip(
+            boxes,
+            0.0,
+            [inputImgWidth, inputImgHeight, inputImgWidth, inputImgHeight],
+            out=boxes,
+        )
 
         resized_masks = []
         for box, cls, raw_mask in zip(boxes, classes, masks):
             raw_cls_mask = raw_mask[cls, ...] if self.is_segmentoly else raw_mask
             if self.resize_mask:
-                resized_masks.append(self._segm_postprocess(box, raw_cls_mask, *meta["original_shape"][:-1]))
+                resized_masks.append(_segm_postprocess(box, raw_cls_mask, *meta["original_shape"][:-1]))
             else:
                 resized_masks.append(raw_cls_mask)
 
         return scores, classes, boxes, resized_masks
 
+    def get_saliency_map_from_prediction(self, outputs, meta, num_classes):
+        """Post process function for saliency map of OTX MaskRCNN model."""
+        boxes = outputs[self.output_blob_name["boxes"]]
+        if boxes.shape[0] == 1:
+            boxes = boxes.squeeze(0)
+        scores = boxes[:, 4]
+        boxes = boxes[:, :4]
+        masks = outputs[self.output_blob_name["masks"]]
+        if masks.shape[0] == 1:
+            masks = masks.squeeze(0)
+        classes = outputs[self.output_blob_name["labels"]].astype(np.uint32)
+        if classes.shape[0] == 1:
+            classes = classes.squeeze(0)
+
+        scale_x = meta["resized_shape"][0] / meta["original_shape"][1]
+        scale_y = meta["resized_shape"][1] / meta["original_shape"][0]
+        boxes[:, 0::2] /= scale_x
+        boxes[:, 1::2] /= scale_y
+
+        saliency_maps = [None for _ in range(num_classes)]
+        for box, score, cls, raw_mask in zip(boxes, scores, classes, masks):
+            resized_mask = self._resize_mask(box, raw_mask * score, *meta["original_shape"][:-1])
+            if saliency_maps[cls] is None:
+                saliency_maps[cls] = [resized_mask]
+            else:
+                saliency_maps[cls].append(resized_mask)
+
+        saliency_maps = self._average_and_normalize(saliency_maps, num_classes)
+        return saliency_maps
+
+    def _resize_mask(self, box, raw_cls_mask, im_h, im_w):
+        # Add zero border to prevent upsampling artifacts on segment borders.
+        raw_cls_mask = np.pad(raw_cls_mask, ((1, 1), (1, 1)), "constant", constant_values=0)
+        extended_box = _expand_box(box, raw_cls_mask.shape[0] / (raw_cls_mask.shape[0] - 2.0)).astype(int)
+        w, h = np.maximum(extended_box[2:] - extended_box[:2] + 1, 1)
+        x0, y0 = np.clip(extended_box[:2], a_min=0, a_max=[im_w, im_h])
+        x1, y1 = np.clip(extended_box[2:] + 1, a_min=0, a_max=[im_w, im_h])
+
+        raw_cls_mask = cv2.resize(raw_cls_mask.astype(np.float32), (w, h))
+        # Put an object mask in an image mask.
+        im_mask = np.zeros((im_h, im_w), dtype=np.float32)
+        im_mask[y0:y1, x0:x1] = raw_cls_mask[
+            (y0 - extended_box[1]) : (y1 - extended_box[1]), (x0 - extended_box[0]) : (x1 - extended_box[0])
+        ]
+        return im_mask
+
+    @staticmethod
+    def _average_and_normalize(saliency_maps, num_classes):
+        for i in range(num_classes):
+            if saliency_maps[i] is not None:
+                saliency_maps[i] = np.array(saliency_maps[i]).mean(0)
+
+        for i in range(num_classes):
+            per_class_map = saliency_maps[i]
+            if per_class_map is not None:
+                max_values = np.max(per_class_map)
+                per_class_map = 255 * (per_class_map) / (max_values + 1e-12)
+                per_class_map = per_class_map.astype(np.uint8)
+                saliency_maps[i] = per_class_map
+        return saliency_maps
+
     def segm_postprocess(self, *args, **kwargs):
         """Post-process for segmentation masks."""
-        return self._segm_postprocess(*args, **kwargs)
+        return _segm_postprocess(*args, **kwargs)
 
     def disable_mask_resizing(self):
         """Disable mask resizing.

deployment/Instance segmentation task/python/requirements.txt

@@ -1,4 +1,4 @@
-openvino==2022.3.0
-openmodelzoo-modelapi==2022.3.0
-otx=1.2.3.3
+openvino==2023.0
+openvino-model-api==0.1.3
+otx==1.4.3
 numpy>=1.21.0,<=1.23.5  # np.bool was removed in 1.24.0 which was used in openvino runtime

deployment/project.json

@@ -61,7 +61,8 @@
       }
    ],
    "thumbnail": "/api/v1/workspaces/6487656fb7efbf83c9b9ec35/projects/6483114c18fb8c1c529bd149/thumbnail",
+   "storage_info": {},
    "performance": {
-      "score": 0.9702380952380952
+      "score": 0.9641873278236913
    }
 }