example_code/demo_ovms.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "copyright": [
     "INTEL CONFIDENTIAL",
     "Copyright (C) 2023 Intel Corporation",
     "This software and the related documents are Intel copyrighted materials, and your use of them is governed by",
     "the express license under which they were provided to you (\"License\"). Unless the License provides otherwise,",
     "you may not use, modify, copy, publish, distribute, disclose or transmit this software or the related documents",
     "without Intel's prior written permission.",
     "This software and the related documents are provided as is, with no express or implied warranties,",
     "other than those that are expressly stated in the License."
    ]
   },
   "source": [
    "# Serving Intel® Geti™ models with OpenVINO Model Server\n",
    "This notebook shows how to set up an OpenVINO model server to serve the models trained\n",
    "in your Intel® Geti™ project. It also shows how to use the Geti SDK as a client to\n",
    "make inference requests to the model server.\n",
    "\n",
    "# Contents\n",
    "\n",
    "1. **OpenVINO Model Server**\n",
    "    1. Requirements\n",
    "    2. Generating the model server configuration\n",
    "    3. Launching the model server\n",
    "\n",
    "2. **OVMS inference with Geti SDK**\n",
    "    1. Loading inference model and sample image\n",
    "    2. Requesting inference\n",
    "    3. Inspecting the results\n",
    "\n",
    "3. **Conclusion**\n",
    "    1. Cleaning up\n",
    "\n",
    "> **NOTE**: This notebook will set up a model server on the same machine that will be\n",
    "> used as a client to request inference. In a real scenario you'd most likely\n",
    "> want the server and the client to be different physical machines. The steps to set up\n",
    "> OVMS on a remote server are the same as for the local server outlined in this\n",
    "> notebook, but additional network configuration and security measures are most likely\n",
    "> required.\n",
    "\n",
    "# OpenVINO Model Server\n",
    "## Requirements\n",
    "We will be running the OpenVINO Model Server (OVMS) with Docker. Please make sure you\n",
    "have docker available on your system. You can install it by following the instructions\n",
    "[here](https://docs.docker.com/get-docker/).\n",
    "\n",
    "## Generating the model server configuration\n",
    "The `deployment` that was downloaded from the Intel® Geti™ platform can be used to create\n",
    "the configuration files that are needed to set up an OpenVINO model server for your project.\n",
    "\n",
    "The cell below shows how to create the configuration. Running this cell should create\n",
    "a folder called `ovms_models` in a temporary directory. The `ovms_models` folder\n",
    "contains the models and the configuration files required to run OVMS for the Intel®\n",
    "Geti™ project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import tempfile\n",
    "\n",
    "from geti_sdk.deployment import Deployment\n",
    "\n",
    "deployment_path = os.path.join(\"..\", \"deployment\")\n",
    "\n",
    "# Load the Geti deployment\n",
    "deployment = Deployment.from_folder(deployment_path)\n",
    "\n",
    "# Creating the OVMS configuration for the deployment\n",
    "# First, we'll create a temporary directory to store the config files\n",
    "ovms_config_path = os.path.join(tempfile.mkdtemp(), \"ovms_models\")\n",
    "\n",
    "# Next, we generate the OVMS configuration and save it\n",
    "deployment.generate_ovms_config(output_folder=ovms_config_path)\n",
    "\n",
    "print(f\"Configuration for OpenVINO Model Server was created at '{ovms_config_path}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Launching the model server\n",
    "As mentioned before, we will run OVMS in a Docker container. First, we need to make sure\n",
    "that we have the latest OVMS image on our system. Run the cell below to pull the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "! docker pull openvino/model_server:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next, we have to start the container with the configuration that we just generated. This\n",
    "is done in the cell below.\n",
    "\n",
    "> NOTE: The cell below starts the OVMS container and sets it up to listen for inference\n",
    "> requests on port 9000 on your system. If this port is already occupied the `docker run`\n",
    "> command will fail and you may need to try a different port number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Launch the OVMS container\n",
    "result = ! docker run -d --rm -v {ovms_config_path}:/models -p 9000:9000 --name ovms_demo openvino/model_server:latest --port 9000 --config_path /models/ovms_model_config.json\n",
    "\n",
    "# Check that the container was created successfully\n",
    "if len(result) == 1:\n",
    "    container_id = result[0]\n",
    "    print(f\"OVMS container with ID '{container_id}' created.\")\n",
    "else:\n",
    "    # Anything other than 1 result indicates that something went wrong\n",
    "    raise RuntimeError(result)\n",
    "\n",
    "# Check that the container is running properly\n",
    "container_info = ! docker container inspect {container_id}\n",
    "container_status = str(container_info.grep(\"Status\"))\n",
    "\n",
    "if not container_status or not \"running\" in container_status:\n",
    "    raise RuntimeError(\n",
    "        f\"Invalid ovms docker container status found: {container_status}. Most \"\n",
    "        f\"likely the container has not started properly.\"\n",
    "    )\n",
    "print(\"OVMS container is up and running.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "That's it! If all went well the cell above should print the ID of the container that\n",
    "was created. This can be used to identify your container if you have a lot of docker\n",
    "containers running on your system.\n",
    "\n",
    "# OVMS inference with Geti SDK\n",
    "Now that the OVMS container is running, we can use the Geti SDK to talk to it and make an\n",
    "inference request. The remaining part of this notebook shows how to do so.\n",
    "\n",
    "## Loading inference model and sample image\n",
    "In the first part of this notebook we created configuration files for OVMS, using the\n",
    "`deployment` that was generated for your Intel® Geti™ project. To do inference, we need\n",
    "to connect the deployment to the OVMS container that is now running. This is done in the\n",
    "cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Load the inference models by connecting to OVMS on port 9000\n",
    "deployment.load_inference_models(device=\"http://localhost:9000\")\n",
    "\n",
    "print(\"Connected to OpenVINO Model Server.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "You should see some output indicating that the connection to OVMS was made successfully.\n",
    "If you see any errors at this stage, make sure your OVMS container is running and that the\n",
    "port number is correct.\n",
    "\n",
    "Next up, we'll load a sample image from the project to run inference on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "\n",
    "# Load the sample image\n",
    "image = cv2.imread(\"../sample_image.jpg\")\n",
    "image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "# Show the image in the notebook\n",
    "from IPython.display import display\n",
    "from PIL import Image\n",
    "\n",
    "display(Image.fromarray(image_rgb))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Requesting inference\n",
    "Now that everything is set up, making an inference request is very simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "t_start = time.time()\n",
    "prediction = deployment.infer(image_rgb)\n",
    "t_end = time.time()\n",
    "\n",
    "print(\n",
    "    f\"OVMS inference on sample image completed in {(t_end - t_start) * 1000:.1f} milliseconds.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Inspecting the results\n",
    "Note that the code to request inference is exactly the same as for the case when the model\n",
    "is loaded on the CPU (see `demo_notebook.ipynb`). Like The `prediction` can be shown using\n",
    "the Geti SDK visualization utility function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from geti_sdk.utils import show_image_with_annotation_scene\n",
    "\n",
    "show_image_with_annotation_scene(image_rgb, prediction, show_in_notebook=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Conclusion\n",
    "That's all there is to it! Of course in practice the client would request inference\n",
    "from an OpenVINO model server on a different physical machine, in contrast to the\n",
    "example here where client and server are running on the same machine.\n",
    "\n",
    "The steps outlined in this notebook can be used as a basis to set up a remote\n",
    "client/server combination, but please note that additional network configuration will\n",
    "be required (along with necessary security measures).\n",
    "\n",
    "## Cleaning up\n",
    "To clean up, we'll stop the OVMS docker container that we started. This will\n",
    "automatically remove the container. After that, we'll delete the temporary directory\n",
    "we created to store the config files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stop the container\n",
    "result = ! docker stop {container_id}\n",
    "\n",
    "# Check if removing the container worked correctly\n",
    "if result[0] == container_id:\n",
    "    print(f\"OVMS container '{container_id}' stopped and removed successfully.\")\n",
    "else:\n",
    "    print(\n",
    "       \"An error occurred while removing OVMS docker container. Most likely the container \"\n",
    "       \"was already removed. \"\n",
    "    )\n",
    "    print(f\"The docker daemon responded with the following error: \\n{result}\")\n",
    "    \n",
    "# Remove the temporary directory with the OVMS configuration\n",
    "import shutil\n",
    "\n",
    "temp_dir = os.path.dirname(ovms_config_path)\n",
    "try:\n",
    "    shutil.rmtree(temp_dir)\n",
    "    print(\"Temporary configuration directory removed successfully.\")\n",
    "except FileNotFoundError:\n",
    "    print(\n",
    "        f\"Temporary directory with OVMS configuration '{temp_dir}' was \"\n",
    "        f\"not found on the system. Most likely it is already removed.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}