{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "9d91c190-956f-44ef-9e3a-1428aa020493", "metadata": {}, "source": [ "# Quantize Data2Vec Speech Recognition Model using NNCF PTQ API\n", "This tutorial demonstrates how to use the NNCF (Neural Network Compression Framework) 8-bit quantization in post-training mode (without the fine-tuning pipeline) to optimize the speech recognition model, known as [Data2Vec](https://arxiv.org/abs/2202.03555) for the high-speed inference via OpenVINO™ Toolkit. This notebook uses a fine-tuned [data2vec-audio-base-960h](https://huggingface.co/facebook/data2vec-audio-base-960h) [PyTorch](https://pytorch.org/) model trained on the [LibriSpeech ASR corpus](https://www.openslr.org/12). The tutorial is designed to be extendable to custom models and datasets. It consists of the following steps:\n", "\n", "- Download and prepare model.\n", "- Define data loading and accuracy validation functionality.\n", "- Prepare the model for quantization and quantize.\n", "- Compare performance of the original and quantized models.\n", "- Compare Accuracy of the Original and Quantized Models.\n" ] }, { "attachments": {}, "cell_type": "markdown", "id": "664ecf1e", "metadata": {}, "source": [ "\n", "#### Table of contents:\n", "\n", "- [Download and prepare model](#Download-and-prepare-model)\n", " - [Obtain Pytorch model representation](#Obtain-Pytorch-model-representation)\n", " - [Convert model to OpenVINO Intermediate Representation](#Convert-model-to-OpenVINO-Intermediate-Representation)\n", " - [Prepare inference data](#Prepare-inference-data)\n", "- [Check model inference result](#Check-model-inference-result)\n", "- [Validate model accuracy on dataset](#Validate-model-accuracy-on-dataset)\n", "- [Quantization](#Quantization)\n", "- [Check INT8 model inference result](#Check-INT8-model-inference-result)\n", "- [Compare Performance of the Original and Quantized Models](#Compare-Performance-of-the-Original-and-Quantized-Models)\n", "- [Compare Accuracy of the Original and Quantized Models](#Compare-Accuracy-of-the-Original-and-Quantized-Models)\n", "\n" ] }, { "attachments": {}, "cell_type": "markdown", "id": "03f79f99-6baf-4f4b-8bc4-a2b0a4e40030", "metadata": {}, "source": [ "## Download and prepare model\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "data2vec is a framework for self-supervised representation learning for images, speech, and text as described in [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (Baevski et al., 2022)](https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language). The algorithm uses the same learning mechanism for different modalities.\n", "\n", "![pre-trained pipeline](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/data2vec.png)\n", "\n", "In our case, we will use `data2vec-audio-base-960h` model, which was finetuned on 960 hours of audio from LibriSpeech Automatic Speech Recognition corpus and distributed as part of HuggingFace transformers.\n", "\n", "### Obtain Pytorch model representation\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "For instantiating PyTorch model class, we should use `Data2VecAudioForCTC.from_pretrained` method with providing model ID for downloading from HuggingFace hub. Model weights and configuration files will be downloaded automatically in first time usage.\n", "Keep in mind that downloading the files can take several minutes and depends on your internet connection.\n", "\n", "Additionally, we can create processor class which is responsible for model specific pre- and post-processing steps." ] }, { "cell_type": "code", "execution_count": null, "id": "b3936821", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:03:56.334642261Z", "start_time": "2023-06-07T16:03:54.294143455Z" } }, "outputs": [], "source": [ "%pip install -q \"openvino>=2023.3.0\" \"nncf>=2.7\"\n", "%pip install -q datasets \"torchmetrics>=0.11.0\" \"torch>=2.1.0\" --extra-index-url https://download.pytorch.org/whl/cpu\n", "%pip install -q soundfile librosa \"transformers>=4.36.2\" --extra-index-url https://download.pytorch.org/whl/cpu" ] }, { "cell_type": "code", "execution_count": 2, "id": "b18e8a87-1064-4100-af21-8d392f236d89", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:03:58.574318746Z", "start_time": "2023-06-07T16:03:56.335717987Z" } }, "outputs": [], "source": [ "from transformers import Wav2Vec2Processor, Data2VecAudioForCTC\n", "\n", "processor = Wav2Vec2Processor.from_pretrained(\"facebook/data2vec-audio-base-960h\")\n", "model = Data2VecAudioForCTC.from_pretrained(\"facebook/data2vec-audio-base-960h\")" ] }, { "attachments": {}, "cell_type": "markdown", "id": "58f16b2f-fae8-4a4b-a8dc-5b49aec358a8", "metadata": {}, "source": [ "### Convert model to OpenVINO Intermediate Representation\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "90535366", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:03:58.577421575Z", "start_time": "2023-06-07T16:03:58.576121202Z" }, "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "# Set model directory\n", "MODEL_DIR = Path(\"model\")\n", "MODEL_DIR.mkdir(exist_ok=True)" ] }, { "cell_type": "code", "execution_count": null, "id": "9699f2ea", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:03:58.761231314Z", "start_time": "2023-06-07T16:03:58.582349854Z" }, "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "import openvino as ov\n", "import torch\n", "\n", "core = ov.Core()\n", "\n", "BATCH_SIZE = 1\n", "MAX_SEQ_LENGTH = 30480\n", "\n", "ir_model_path = MODEL_DIR / \"data2vec-audo-base.xml\"\n", "\n", "if not ir_model_path.exists():\n", " ov_model = ov.convert_model(model, example_input=torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float))\n", " ov.save_model(ov_model, str(ir_model_path))\n", " print(\"IR model saved to {}\".format(ir_model_path))\n", "else:\n", " print(\"Read IR model from {}\".format(ir_model_path))\n", " ov_model = core.read_model(ir_model_path)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "ea7106bc-7cfb-4158-a9c7-bbbb76787523", "metadata": {}, "source": [ "### Prepare inference data\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "For demonstration purposes, we will use short dummy version of LibriSpeech dataset - `patrickvonplaten/librispeech_asr_dummy` to speed up model evaluation. Model accuracy can be different from reported in the paper. For reproducing original accuracy, use `librispeech_asr` dataset." ] }, { "cell_type": "code", "execution_count": null, "id": "c312b02e-2c99-4211-b7f1-4e491550b568", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:04:00.743367856Z", "start_time": "2023-06-07T16:03:58.762193807Z" } }, "outputs": [], "source": [ "from datasets import load_dataset\n", "\n", "ds = load_dataset(\"patrickvonplaten/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n", "\n", "\n", "# define preprocessing function for converting audio to input values for model\n", "def map_to_input(batch):\n", " preprocessed_signal = processor(\n", " batch[\"audio\"][\"array\"],\n", " return_tensors=\"pt\",\n", " padding=\"longest\",\n", " sampling_rate=batch[\"audio\"][\"sampling_rate\"],\n", " )\n", " input_values = preprocessed_signal.input_values\n", " batch[\"input_values\"] = input_values\n", " return batch\n", "\n", "\n", "# apply preprocessing function to dataset and remove audio column, to save memory as we do not need it anymore\n", "dataset = ds.map(map_to_input, batched=False, remove_columns=[\"audio\"])\n", "\n", "test_sample = ds[0][\"audio\"]" ] }, { "attachments": {}, "cell_type": "markdown", "id": "dc35d830-4b64-4083-bd0c-aaef68ba3d86", "metadata": {}, "source": [ "## Check model inference result\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "The code below is used for running model inference on a single sample from the dataset. It contains the following steps:\n", "\n", "* Get the input_values tensor as model input.\n", "* Run model inference and obtain logits.\n", "* Find logits ids with highest probability, using argmax.\n", "* Decode predicted token ids, using processor.\n", "\n", "For reference, see the same function provided for OpenVINO model." ] }, { "cell_type": "code", "execution_count": 6, "id": "a6a11f44-2fe1-4aa9-a360-ba4b23bba87d", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:04:03.536084940Z", "start_time": "2023-06-07T16:04:03.534552418Z" } }, "outputs": [], "source": [ "import numpy as np\n", "\n", "\n", "# inference function for pytorch\n", "def torch_infer(model, sample):\n", " logits = model(torch.Tensor(sample[\"input_values\"])).logits\n", " # take argmax and decode\n", " predicted_ids = torch.argmax(logits, dim=-1)\n", " transcription = processor.batch_decode(predicted_ids)\n", " return transcription\n", "\n", "\n", "# inference function for openvino\n", "def ov_infer(model, sample):\n", " output = model.output(0)\n", " logits = model(np.array(sample[\"input_values\"]))[output]\n", " predicted_ids = np.argmax(logits, axis=-1)\n", " transcription = processor.batch_decode(torch.from_numpy(predicted_ids))\n", " return transcription" ] }, { "cell_type": "markdown", "id": "0bb514d4-2d00-4a8c-a858-76730c59e3f4", "metadata": {}, "source": [ "Select inference device for OpenVINO" ] }, { "cell_type": "code", "execution_count": 7, "id": "e77013b8-7afb-4e22-9f2f-734e312f7b79", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6f48b17c489046c3a4ecf1e1bf177822", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Dropdown(description='Device:', index=4, options=('CPU', 'GPU.0', 'GPU.1', 'GPU.2', 'AUTO'), value='AUTO')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import ipywidgets as widgets\n", "\n", "core = ov.Core()\n", "device = widgets.Dropdown(\n", " options=core.available_devices + [\"AUTO\"],\n", " value=\"AUTO\",\n", " description=\"Device:\",\n", " disabled=False,\n", ")\n", "\n", "device" ] }, { "cell_type": "code", "execution_count": 8, "id": "fb36df3c-b461-4400-9ebc-19d9f307032c", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:04:06.536948027Z", "start_time": "2023-06-07T16:04:03.536437467Z" } }, "outputs": [], "source": [ "core = ov.Core()\n", "\n", "pt_transcription = torch_infer(model, dataset[0])\n", "compiled_model = core.compile_model(ov_model, device.value)\n", "ov_transcription = ov_infer(compiled_model, dataset[0])" ] }, { "cell_type": "code", "execution_count": 9, "id": "6d02ca6e-9939-4d10-a65f-f03b19c4153a", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:04:06.604292313Z", "start_time": "2023-06-07T16:04:06.595005784Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Reference]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY ROSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW WHILE POOR SHAGGY SITS THERE A COOING DOVE\n", "[PyTorch]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY RUSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW A POOR SHAGGY SITS THERE ACCOOING DOVE\n", "[OpenVINO FP16]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY RUSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW A POOR SHAGGY SITS THERE ACCOOING DOVE\n" ] }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import IPython.display as ipd\n", "\n", "print(f\"[Reference]: {dataset[0]['text']}\")\n", "print(f\"[PyTorch]: {pt_transcription[0]}\")\n", "print(f\"[OpenVINO FP16]: {ov_transcription[0]}\")\n", "ipd.Audio(test_sample[\"array\"], rate=16000)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d7281085-47ef-4f6e-b778-8d5921a97644", "metadata": {}, "source": [ "## Validate model accuracy on dataset\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "For model accuracy evaluation, [Word Error Rate](https://en.wikipedia.org/wiki/Word_error_rate) metric can be used. Word Error Rate or WER is the ratio of errors in a transcript to the total words spoken. A lower WER in speech-to-text means better accuracy in recognizing speech.\n", "\n", "For WER calculation, we will use [`torchmetrics`](https://torchmetrics.readthedocs.io/en/stable/text/word_error_rate.html) library." ] }, { "cell_type": "code", "execution_count": 10, "id": "0cdf626f-a1c3-4bd7-b31d-4542fed04236", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:04:06.710441239Z", "start_time": "2023-06-07T16:04:06.603661897Z" } }, "outputs": [], "source": [ "from torchmetrics.text import WordErrorRate\n", "from tqdm.notebook import tqdm\n", "\n", "\n", "def compute_wer(dataset, model, infer_fn):\n", " wer = WordErrorRate()\n", " for sample in tqdm(dataset):\n", " # run infer function on sample\n", " transcription = infer_fn(model, sample)\n", " # update metric on sample result\n", " wer.update(transcription, [sample[\"text\"]])\n", " # finalize metric calculation\n", " result = wer.compute()\n", " return result" ] }, { "cell_type": "code", "execution_count": 11, "id": "da4ceda6-1535-4816-ab1b-bf3dc80a636f", "metadata": { "ExecuteTime": { "end_time": "2023-06-07T16:04:49.431119230Z", "start_time": "2023-06-07T16:04:06.712270520Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e11dbee699324434a17f2bfe9ab263dc", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/73 [00:00\n" ], "text/plain": [] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n",
       "
\n" ], "text/plain": [ "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e9216c796a3f4efea1b09e31be4a1126", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:2 ignored nodes were found by name in the NNCFGraph\n",
      "INFO:nncf:36 ignored nodes were found by name in the NNCFGraph\n",
      "INFO:nncf:50 ignored nodes were found by name in the NNCFGraph\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 3 __module.data2vec_audio.feature_extractor.conv_layers.0.conv/aten::_convolution/Convolution\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 10 __module.data2vec_audio.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "049f5ab2e4b04e73977bb1abed3129f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fac40c79ae634450b63b7618b4a39451",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import nncf\n",
    "from nncf.parameters import ModelType\n",
    "\n",
    "\n",
    "def transform_fn(data_item):\n",
    "    \"\"\"\n",
    "    Extract the model's input from the data item.\n",
    "    The data item here is the data item that is returned from the data source per iteration.\n",
    "    This function should be passed when the data item cannot be used as model's input.\n",
    "    \"\"\"\n",
    "    return np.array(data_item[\"input_values\"])\n",
    "\n",
    "\n",
    "calibration_dataset = nncf.Dataset(dataset, transform_fn)\n",
    "\n",
    "\n",
    "quantized_model = nncf.quantize(\n",
    "    ov_model,\n",
    "    calibration_dataset,\n",
    "    model_type=ModelType.TRANSFORMER,  # specify additional transformer patterns in the model\n",
    "    subset_size=len(dataset),\n",
    "    ignored_scope=nncf.IgnoredScope(\n",
    "        names=[\n",
    "            \"__module.data2vec_audio.feature_extractor.conv_layers.0.conv/aten::_convolution/Convolution\",\n",
    "            \"__module.data2vec_audio.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution\",\n",
    "        ],\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2fc9e942-58d4-49d3-b115-758583661093",
   "metadata": {},
   "source": [
    "After quantization is finished, compressed model representation can be saved using `ov.save_model` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "898ecbdd-1f89-47c7-8877-b7acf57ec8f3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-07T16:07:39.528434877Z",
     "start_time": "2023-06-07T16:07:39.527002213Z"
    }
   },
   "outputs": [],
   "source": [
    "MODEL_NAME = \"quantized_data2vec_base\"\n",
    "quantized_model_path = Path(f\"{MODEL_NAME}_openvino_model/{MODEL_NAME}_quantized.xml\")\n",
    "ov.save_model(quantized_model, quantized_model_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9fda5391-9a7c-4606-8d5e-fed5fd6cf0fc",
   "metadata": {},
   "source": [
    "## Check INT8 model inference result\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "`INT8` model is the same in usage like the original one. We need to read it, using the `core.read_model` method and load on the device, using `core.compile_model`. After that, we can reuse the same `ov_infer` function for getting model inference result on test sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d0ea3977-c5bd-42ec-b700-4bcc43a8cd29",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-07T16:07:40.022688559Z",
     "start_time": "2023-06-07T16:07:39.528896096Z"
    }
   },
   "outputs": [],
   "source": [
    "int8_compiled_model = core.compile_model(quantized_model, device.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "518b56cb-5617-4ad6-a7b5-747e9698829a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-07T16:07:40.248888136Z",
     "start_time": "2023-06-07T16:07:40.041525928Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Reference]:     BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY ROSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW WHILE POOR SHAGGY SITS THERE A COOING DOVE\n",
      "[PyTorch]:       BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY RUSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW A POOR SHAGGY SITS THERE ACCOOING DOVE\n",
      "[OpenVINO FP16]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY RUSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW A POOR SHAGGY SITS THERE ACCOOING DOVE\n",
      "[OpenVINO INT8]: BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY RUSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW A POORA SHAGGY SITS THERE ACCOOING DOVE\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                \n",
       "              "
      ],
      "text/plain": [
       ""
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcription = ov_infer(int8_compiled_model, dataset[0])\n",
    "print(f\"[Reference]:     {dataset[0]['text']}\")\n",
    "print(f\"[PyTorch]:       {pt_transcription[0]}\")\n",
    "print(f\"[OpenVINO FP16]: {ov_transcription[0]}\")\n",
    "print(f\"[OpenVINO INT8]: {transcription[0]}\")\n",
    "ipd.Audio(test_sample[\"array\"], rate=16000)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "98777233-6cc5-4d97-bbcd-f8338c75b55e",
   "metadata": {},
   "source": [
    "## Compare Performance of the Original and Quantized Models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "[Benchmark Tool](https://docs.openvino.ai/2024/learn-openvino/openvino-samples/benchmark-tool.html) is used to measure the inference performance of the `FP16` and `INT8` models.\n",
    "\n",
    "> **NOTE**: For more accurate performance, it is recommended to run `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app -m model.xml -d CPU` to benchmark async inference on CPU for one minute. Change `CPU` to `GPU` to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command-line options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c65ef565-5b65-4efb-b663-526c8e9f0eed",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-07T16:07:56.551074457Z",
     "start_time": "2023-06-07T16:07:40.249158437Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Step 1/11] Parsing and validating input arguments\n",
      "[ INFO ] Parsing input parameters\n",
      "[Step 2/11] Loading OpenVINO Runtime\n",
      "[ INFO ] OpenVINO:\n",
      "[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3\n",
      "[ INFO ] \n",
      "[ INFO ] Device info:\n",
      "[ INFO ] AUTO\n",
      "[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3\n",
      "[ INFO ] \n",
      "[ INFO ] \n",
      "[Step 3/11] Setting device configuration\n",
      "[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
      "[Step 4/11] Reading model files\n",
      "[ INFO ] Loading model files\n",
      "[ INFO ] Read model took 26.84 ms\n",
      "[ INFO ] Original model I/O parameters:\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     input_values (node: input_values) : f32 / [...] / [?,?]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     1289 , logits (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [?,?,32]\n",
      "[Step 5/11] Resizing model to match image sizes and given batch\n",
      "[ INFO ] Model batch size: 1\n",
      "[ INFO ] Reshaping model: 'input_values': [1,30480]\n",
      "[ INFO ] Reshape model took 31.21 ms\n",
      "[Step 6/11] Configuring input of the model\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     input_values (node: input_values) : f32 / [...] / [1,30480]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     1289 , logits (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [1,95,32]\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Compile model took 871.22 ms\n",
      "[Step 8/11] Querying optimal runtime parameters\n",
      "[ INFO ] Model:\n",
      "[ INFO ]   NETWORK_NAME: Model0\n",
      "[ INFO ]   EXECUTION_DEVICES: ['CPU']\n",
      "[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT\n",
      "[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
      "[ INFO ]   MULTI_DEVICE_PRIORITIES: CPU\n",
      "[ INFO ]   CPU:\n",
      "[ INFO ]     AFFINITY: Affinity.CORE\n",
      "[ INFO ]     CPU_DENORMALS_OPTIMIZATION: False\n",
      "[ INFO ]     CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0\n",
      "[ INFO ]     ENABLE_CPU_PINNING: True\n",
      "[ INFO ]     ENABLE_HYPER_THREADING: True\n",
      "[ INFO ]     EXECUTION_DEVICES: ['CPU']\n",
      "[ INFO ]     EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
      "[ INFO ]     INFERENCE_NUM_THREADS: 36\n",
      "[ INFO ]     INFERENCE_PRECISION_HINT: \n",
      "[ INFO ]     NETWORK_NAME: Model0\n",
      "[ INFO ]     NUM_STREAMS: 12\n",
      "[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
      "[ INFO ]     PERFORMANCE_HINT: THROUGHPUT\n",
      "[ INFO ]     PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
      "[ INFO ]     PERF_COUNT: NO\n",
      "[ INFO ]     SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
      "[ INFO ]   MODEL_PRIORITY: Priority.MEDIUM\n",
      "[ INFO ]   LOADED_FROM_CACHE: False\n",
      "[Step 9/11] Creating infer requests and preparing input tensors\n",
      "[ WARNING ] No input files were given for input 'input_values'!. This input will be filled with random values!\n",
      "[ INFO ] Fill input 'input_values' with random values \n",
      "[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 15000 ms duration)\n",
      "[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
      "[ INFO ] First inference took 90.48 ms\n",
      "[Step 11/11] Dumping statistics report\n",
      "[ INFO ] Execution Devices:['CPU']\n",
      "[ INFO ] Count:            732 iterations\n",
      "[ INFO ] Duration:         15334.79 ms\n",
      "[ INFO ] Latency:\n",
      "[ INFO ]    Median:        250.16 ms\n",
      "[ INFO ]    Average:       250.01 ms\n",
      "[ INFO ]    Min:           193.17 ms\n",
      "[ INFO ]    Max:           277.38 ms\n",
      "[ INFO ] Throughput:   47.73 FPS\n"
     ]
    }
   ],
   "source": [
    "# Inference FP16 model (OpenVINO IR)\n",
    "! benchmark_app -m $ir_model_path -shape [1,30480] -d $device.value -api async -t 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "32960bce-373a-4421-ba7c-d9a7e3f9b678",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-07T16:08:13.109162303Z",
     "start_time": "2023-06-07T16:07:56.585814927Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Step 1/11] Parsing and validating input arguments\n",
      "[ INFO ] Parsing input parameters\n",
      "[Step 2/11] Loading OpenVINO Runtime\n",
      "[ INFO ] OpenVINO:\n",
      "[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3\n",
      "[ INFO ] \n",
      "[ INFO ] Device info:\n",
      "[ INFO ] AUTO\n",
      "[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3\n",
      "[ INFO ] \n",
      "[ INFO ] \n",
      "[Step 3/11] Setting device configuration\n",
      "[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
      "[Step 4/11] Reading model files\n",
      "[ INFO ] Loading model files\n",
      "[ INFO ] Read model took 65.75 ms\n",
      "[ INFO ] Original model I/O parameters:\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     input_values (node: input_values) : f32 / [...] / [?,?]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     logits , 1289 (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [?,?,32]\n",
      "[Step 5/11] Resizing model to match image sizes and given batch\n",
      "[ INFO ] Model batch size: 1\n",
      "[ INFO ] Reshaping model: 'input_values': [1,30480]\n",
      "[ INFO ] Reshape model took 37.95 ms\n",
      "[Step 6/11] Configuring input of the model\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     input_values (node: input_values) : f32 / [...] / [1,30480]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     logits , 1289 (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [1,95,32]\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Compile model took 1339.85 ms\n",
      "[Step 8/11] Querying optimal runtime parameters\n",
      "[ INFO ] Model:\n",
      "[ INFO ]   NETWORK_NAME: Model0\n",
      "[ INFO ]   EXECUTION_DEVICES: ['CPU']\n",
      "[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT\n",
      "[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
      "[ INFO ]   MULTI_DEVICE_PRIORITIES: CPU\n",
      "[ INFO ]   CPU:\n",
      "[ INFO ]     AFFINITY: Affinity.CORE\n",
      "[ INFO ]     CPU_DENORMALS_OPTIMIZATION: False\n",
      "[ INFO ]     CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0\n",
      "[ INFO ]     ENABLE_CPU_PINNING: True\n",
      "[ INFO ]     ENABLE_HYPER_THREADING: True\n",
      "[ INFO ]     EXECUTION_DEVICES: ['CPU']\n",
      "[ INFO ]     EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
      "[ INFO ]     INFERENCE_NUM_THREADS: 36\n",
      "[ INFO ]     INFERENCE_PRECISION_HINT: \n",
      "[ INFO ]     NETWORK_NAME: Model0\n",
      "[ INFO ]     NUM_STREAMS: 12\n",
      "[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
      "[ INFO ]     PERFORMANCE_HINT: THROUGHPUT\n",
      "[ INFO ]     PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
      "[ INFO ]     PERF_COUNT: NO\n",
      "[ INFO ]     SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
      "[ INFO ]   MODEL_PRIORITY: Priority.MEDIUM\n",
      "[ INFO ]   LOADED_FROM_CACHE: False\n",
      "[Step 9/11] Creating infer requests and preparing input tensors\n",
      "[ WARNING ] No input files were given for input 'input_values'!. This input will be filled with random values!\n",
      "[ INFO ] Fill input 'input_values' with random values \n",
      "[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 15000 ms duration)\n",
      "[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
      "[ INFO ] First inference took 68.59 ms\n",
      "[Step 11/11] Dumping statistics report\n",
      "[ INFO ] Execution Devices:['CPU']\n",
      "[ INFO ] Count:            1092 iterations\n",
      "[ INFO ] Duration:         15108.16 ms\n",
      "[ INFO ] Latency:\n",
      "[ INFO ]    Median:        165.80 ms\n",
      "[ INFO ]    Average:       165.38 ms\n",
      "[ INFO ]    Min:           63.30 ms\n",
      "[ INFO ]    Max:           200.85 ms\n",
      "[ INFO ] Throughput:   72.28 FPS\n"
     ]
    }
   ],
   "source": [
    "# Inference INT8 model (OpenVINO IR)\n",
    "! benchmark_app -m $quantized_model_path -shape [1,30480] -d $device.value -api async -t 15"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "34347a28-73bb-4bfc-8048-25e03d70f80e",
   "metadata": {},
   "source": [
    "## Compare Accuracy of the Original and Quantized Models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Finally, calculate WER metric for the `INT8` model representation and compare it with the `FP16` result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fb4937fb-ba00-490b-b116-d6c731d35450",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-07T16:08:29.719577956Z",
     "start_time": "2023-06-07T16:08:13.124222392Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49e7626e5ef446f49342d050f94e2ab1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/73 [00:00