{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Post-Training Quantization of OpenAI Whisper model with NNCF\n", "\n", "The goal of this tutorial is to demonstrate how to speed up the model by applying 8-bit post-training quantization from [NNCF](https://github.com/openvinotoolkit/nncf/) (Neural Network Compression Framework) and infer quantized model via OpenVINO™ Toolkit. The optimization process contains the following steps:\n", "\n", "1. Quantize the converted OpenVINO model from [whisper-convert notebook](whisper-convert.ipynb) with NNCF.\n", "2. Check model result for the demo video.\n", "3. Compare model size, performance and accuracy of FP32 and quantized INT8 models.\n", "\n", "> **NOTE**: you should run [whisper-convert](whisper-convert.ipynb) notebook first to generate OpenVINO IR model that is used for quantization.\n", "\n", "\n", "#### Table of contents:\n", "\n", "- [Prerequisites](#Prerequisites)\n", "- [Create and initialize quantization](#Create-and-initialize-quantization)\n", " - [Prepare calibration datasets](#Prepare-calibration-datasets)\n", " - [Quantize Whisper encoder and decoder models](#Quantize-Whisper-encoder-and-decoder-models)\n", "- [Transcribe video with quantized OpenVINO model](#Transcribe-video-with-quantized-OpenVINO-model)\n", "- [Compare performance and accuracy of the FP32 and INT8 IRs](#Compare-performance-and-accuracy-of-the-FP32-and-INT8-IRs)\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prerequisites\n", "[back to top ⬆️](#Table-of-contents:)\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Install dependencies." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T07:22:38.138055200Z", "start_time": "2023-08-24T07:22:38.000326500Z" }, "tags": [] }, "outputs": [], "source": [ "%pip install -q \"openvino>=2023.1.0\"\n", "%pip install -q \"nncf>=2.6.0\"\n", "%pip install -q datasets librosa soundfile\n", "%pip install -q evaluate jiwer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Select model for quantization" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2e6dde4a5c3344b88772ddc9b5d94510", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Dropdown(description='Model:', options=('large-v2', 'large-v3'), value='large-v2')" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from pathlib import Path\n", "import ipywidgets as widgets\n", "\n", "\n", "def get_model_id(model_path):\n", " return model_path.name.replace(\"whisper_\", \"\").replace(\"encoder.xml\", \"\").replace(\"_\", \"\")\n", "\n", "\n", "model_list = [get_model_id(model_path) for model_path in Path(\".\").glob(\"whisper_*encoder.xml\")]\n", "model_list = [model_name for model_name in model_list if model_name]\n", "\n", "if not model_list:\n", " raise RuntimeError(\"Please run conversion notebook first\")\n", "\n", "model_id = widgets.Dropdown(\n", " options=model_list,\n", " value=model_list[0],\n", " description=\"Model:\",\n", " disabled=False,\n", ")\n", "\n", "model_id" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Select device from dropdown list for running inference using OpenVINO." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T07:22:38.357268200Z", "start_time": "2023-08-24T07:22:38.250550900Z" }, "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "09e511523d2f4cf09655829a27347e61", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Dropdown(description='Device:', index=2, options=('CPU', 'GPU', 'AUTO'), value='AUTO')" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import ipywidgets as widgets\n", "\n", "from openvino import Core\n", "\n", "core = Core()\n", "\n", "device = widgets.Dropdown(\n", " options=core.available_devices + [\"AUTO\"],\n", " value=\"AUTO\",\n", " description=\"Device:\",\n", " disabled=False,\n", ")\n", "\n", "device" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Select the task for the model:\n", "\n", "* **transcribe** - generate audio transcription in the source language (automatically detected).\n", "* **translate** - generate audio transcription with translation to English language." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T07:22:38.359102800Z", "start_time": "2023-08-24T07:22:38.357540500Z" }, "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "83871b40db7e48b693dfca1f39c58eb0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Select(description='Select task:', index=1, options=('transcribe', 'translate'), value='translate')" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "task = widgets.Select(\n", " options=[\"transcribe\", \"translate\"],\n", " value=\"translate\",\n", " description=\"Select task:\",\n", " disabled=False,\n", ")\n", "task" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "## Create and initialize quantization\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "[NNCF](https://github.com/openvinotoolkit/nncf/) enables post-training quantization by adding the quantization layers into the model graph and then using a subset of the training dataset to initialize the parameters of these additional quantization layers. The framework is designed so that modifications to your original training code are minor. Quantization is the simplest scenario and requires a few modifications.\n", "\n", "The optimization process contains the following steps:\n", "\n", "1. Create a calibration dataset for quantization.\n", "2. Run `nncf.quantize` to obtain quantized models.\n", "3. Serialize the `INT8` model using `openvino.runtime.serialize` function." ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Set paths to the model converted in [whisper-convert](whisper-convert.ipynb) notebook and the paths where quantized models will be saved." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "WHISPER_ENCODER_OV = Path(f\"whisper_{model_id.value}_encoder.xml\")\n", "WHISPER_DECODER_OV = Path(f\"whisper_{model_id.value}_decoder.xml\")\n", "\n", "WHISPER_ENCODER_OV_INT8 = Path(f\"whisper_{model_id.value}_encoder_int8.xml\")\n", "WHISPER_DECODER_OV_INT8 = Path(f\"whisper_{model_id.value}_decoder_int8.xml\")" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Load FP32 model IR." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T07:22:39.574746400Z", "start_time": "2023-08-24T07:22:38.358318Z" } }, "outputs": [], "source": [ "import whisper\n", "\n", "# Fetch `notebook_utils` module\n", "import requests\n", "\n", "r = requests.get(\n", " url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n", ")\n", "open(\"notebook_utils.py\", \"w\").write(r.text)\n", "from notebook_utils import download_file\n", "\n", "if not Path(\"./utils.py\").exists():\n", " download_file(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/whisper-subtitles-generation/utils.py\")\n", "\n", "from utils import (\n", " patch_whisper_for_ov_inference,\n", " OpenVINOAudioEncoder,\n", " OpenVINOTextDecoder,\n", ")\n", "\n", "model_fp32 = whisper.load_model(model_id.value, \"cpu\").eval()\n", "patch_whisper_for_ov_inference(model_fp32)\n", "\n", "model_fp32.encoder = OpenVINOAudioEncoder(core, WHISPER_ENCODER_OV, device=device.value)\n", "model_fp32.decoder = OpenVINOTextDecoder(core, WHISPER_DECODER_OV, device=device.value)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "### Prepare calibration datasets\n", "[back to top ⬆️](#Table-of-contents:)\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Whisper consists of an encoder and a decoder models. We need to collect calibration data for both of them.\n", "\n", "Below we overwrite encoder/decoder forward methods in order to collect calibration samples." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T07:22:39.623947800Z", "start_time": "2023-08-24T07:22:39.575286700Z" }, "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "from contextlib import contextmanager\n", "from functools import partial\n", "import openvino as ov\n", "from typing import Optional\n", "import torch\n", "\n", "COLLECT_CALIBRATION_DATA = False\n", "encoder_calibration_data = []\n", "decoder_calibration_data = []\n", "\n", "\n", "@contextmanager\n", "def calibration_data_collection():\n", " global COLLECT_CALIBRATION_DATA\n", " try:\n", " COLLECT_CALIBRATION_DATA = True\n", " yield\n", " finally:\n", " COLLECT_CALIBRATION_DATA = False\n", "\n", "\n", "def encoder_forward(self, mel: torch.Tensor):\n", " if COLLECT_CALIBRATION_DATA:\n", " encoder_calibration_data.append(mel)\n", " return torch.from_numpy(self.compiled_model(mel)[self.output_blob])\n", "\n", "\n", "def decoder_forward(self, x: torch.Tensor, xa: torch.Tensor, kv_cache: Optional[dict] = None):\n", " feed_dict = {\"x\": ov.Tensor(x.numpy()), \"xa\": ov.Tensor(xa.numpy())}\n", " feed_dict = self.preprocess_kv_cache_inputs(feed_dict, kv_cache)\n", " if COLLECT_CALIBRATION_DATA:\n", " decoder_calibration_data.append(feed_dict)\n", " res = self.compiled_model(feed_dict)\n", " return self.postprocess_outputs(res)\n", "\n", "\n", "model_fp32.encoder.forward = partial(encoder_forward, model_fp32.encoder)\n", "model_fp32.decoder.forward = partial(decoder_forward, model_fp32.decoder)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "We use a portion of validation [librispeech_asr](https://huggingface.co/datasets/librispeech_asr) dataset from Hugging Face as calibration data." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T07:23:05.269312200Z", "start_time": "2023-08-24T07:22:39.623947800Z" }, "collapsed": false, "jupyter": { "outputs_hidden": false }, "test_replace": { "CALIBRATION_DATASET_SIZE = 30": "CALIBRATION_DATASET_SIZE = 1" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "50abbb740a6f49e386c8963bb2cd9b79", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Collecting calibration data: 0%| | 0/30 [00:00 00:00:05,000\n", " What's that?\n", "\n", "2\n", "00:00:05,000 --> 00:00:07,000\n", " Oh, wow.\n", "\n", "3\n", "00:00:09,000 --> 00:00:11,000\n", " Hello, humans.\n", "\n", "4\n", "00:00:13,000 --> 00:00:15,000\n", " Focus on me.\n", "\n", "5\n", "00:00:15,000 --> 00:00:17,000\n", " Focus on the guard.\n", "\n", "6\n", "00:00:17,000 --> 00:00:20,000\n", " Don't tell anyone what you see in here.\n", "\n", "7\n", "00:00:22,000 --> 00:00:24,000\n", " Have you seen what's in there?\n", "\n", "8\n", "00:00:24,000 --> 00:00:25,000\n", " They have...\n", "\n", "9\n", "00:00:25,000 --> 00:00:27,000\n", " Intel. This is where it all changes.\n", "\n", "\n" ] } ], "source": [ "print(\"\".join(srt_lines))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "As you can see the result is almost the same." ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "## Compare performance and accuracy of the FP32 and INT8 IRs\n", "[back to top ⬆️](#Table-of-contents:)\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Compare model file size." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T07:25:39.370126700Z", "start_time": "2023-08-24T07:25:39.336253900Z" }, "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: whisper_large-v2_encoder\n", " * FP32 IR model size: 1244080.07 KB\n", " * INT8 IR model size: 626971.58 KB\n", " * Model compression rate: 1.984\n", "Model: whisper_large-v2_decoder\n", " * FP32 IR model size: 1900607.09 KB\n", " * INT8 IR model size: 955679.81 KB\n", " * Model compression rate: 1.989\n" ] } ], "source": [ "def calculate_compression_rate(model_path_ov, model_path_ov_int8):\n", " model_size_fp32 = model_path_ov.with_suffix(\".bin\").stat().st_size / 1024\n", " model_size_int8 = model_path_ov_int8.with_suffix(\".bin\").stat().st_size / 1024\n", " print(f\"Model: {model_path_ov.stem}\")\n", " print(f\" * FP32 IR model size: {model_size_fp32:.2f} KB\")\n", " print(f\" * INT8 IR model size: {model_size_int8:.2f} KB\")\n", " print(f\" * Model compression rate: {model_size_fp32 / model_size_int8:.3f}\")\n", "\n", "\n", "calculate_compression_rate(WHISPER_ENCODER_OV, WHISPER_ENCODER_OV_INT8)\n", "calculate_compression_rate(WHISPER_DECODER_OV, WHISPER_DECODER_OV_INT8)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "To measure the inference performance of the `FP32` and `INT8` encoder/decoder models, we use median inference time on calibration dataset.\n", "So we can approximately estimate the speed-up of the dynamic quantized models.\n", "\n", "> **NOTE**: For the most accurate performance estimation, it is recommended to run `benchmark_app` with static shapes in a terminal/command prompt after closing other applications." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T07:26:39.712460600Z", "start_time": "2023-08-24T07:25:39.370126700Z" }, "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "39707d4e1bed4ca3a56aa05b6158c124", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Measuring performance: 0%| | 0/60 [00:00 **NOTE**: Accuracy drop can generally be improved by increasing calibration dataset size." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "openvino_notebooks": { "imageUrl": "https://user-images.githubusercontent.com/29454499/204548693-1304ef33-c790-490d-8a8b-d5766acb6254.png", "tags": { "categories": [ "Model Demos", "AI Trends" ], "libraries": [], "other": [], "tasks": [ "Speech Recognition" ] } }, "vscode": { "interpreter": { "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1" } }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": { "2e6dde4a5c3344b88772ddc9b5d94510": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "DropdownModel", "state": { "_options_labels": [ "large-v2", "large-v3" ], "description": "Model:", "index": 0, "layout": "IPY_MODEL_b01054569522434d865aed9f2e2c0f83", "style": "IPY_MODEL_68ab334740754f9591d40c5bfdad8128" } }, "68ab334740754f9591d40c5bfdad8128": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "b01054569522434d865aed9f2e2c0f83": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} } }, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }