{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "cacbe6b4",
"metadata": {
"id": "rQc-wXjqrEuR"
},
"source": [
"# Quantize Wav2Vec Speech Recognition Model using NNCF PTQ API\n",
"This tutorial demonstrates how to apply `INT8` quantization to the speech recognition model, known as [Wav2Vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2), using the NNCF (Neural Network Compression Framework) 8-bit quantization in post-training mode (without the fine-tuning pipeline). This notebook uses a fine-tuned [Wav2Vec2-Base-960h](https://huggingface.co/facebook/wav2vec2-base-960h) [PyTorch](https://pytorch.org/) model trained on the [LibriSpeech ASR corpus](https://www.openslr.org/12). The tutorial is designed to be extendable to custom models and datasets. It consists of the following steps:\n",
"\n",
"- Download and prepare the Wav2Vec2 model and LibriSpeech dataset.\n",
"- Define data loading and accuracy validation functionality.\n",
"- Model quantization.\n",
"- Compare Accuracy of original PyTorch model, OpenVINO FP16 and INT8 models.\n",
"- Compare performance of the original and quantized models.\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b6bb1c0f",
"metadata": {},
"source": [
"\n",
"#### Table of contents:\n",
"\n",
"- [Imports](#Imports)\n",
"- [Settings](#Settings)\n",
"- [Prepare the Model](#Prepare-the-Model)\n",
"- [Prepare LibriSpeech Dataset](#Prepare-LibriSpeech-Dataset)\n",
"- [Run Quantization](#Run-Quantization)\n",
"- [Model Usage Example with Inference Pipeline](#Model-Usage-Example-with-Inference-Pipeline)\n",
"- [Validate model accuracy on dataset](#Validate-model-accuracy-on-dataset)\n",
"- [Compare Performance of the Original and Quantized Models](#Compare-Performance-of-the-Original-and-Quantized-Models)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3f1f601b-c012-4e48-ab50-262ec3c0af2d",
"metadata": {},
"outputs": [],
"source": [
"%pip install -q \"openvino>=2023.3.0\" \"nncf>=2.7\"\n",
"%pip install datasets \"torchmetrics>=0.11.0\" \"torch>=2.1.0\" --extra-index-url https://download.pytorch.org/whl/cpu\n",
"%pip install -q soundfile librosa \"transformers>=4.36.2\" --extra-index-url https://download.pytorch.org/whl/cpu"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4d6b41e6-132b-40da-b3b9-91bacba29e31",
"metadata": {},
"source": [
"## Imports\n",
"[back to top ⬆️](#Table-of-contents:)\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "629e2d2c",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"import numpy as np\n",
"import openvino as ov\n",
"import torch\n",
"import IPython.display as ipd\n",
"\n",
"from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e9e66896-d439-4065-868a-65b44d31525a",
"metadata": {},
"source": [
"## Settings\n",
"[back to top ⬆️](#Table-of-contents:)\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "284e9a4b",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"# Set the data and model directories, model source URL and model filename.\n",
"MODEL_DIR = Path(\"model\")\n",
"MODEL_DIR.mkdir(exist_ok=True)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "44dc335d",
"metadata": {
"id": "YytHDzLE0uOJ"
},
"source": [
"## Prepare the Model\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"Perform the following:\n",
"- Download and unpack a pre-trained Wav2Vec2 model.\n",
"- Run model conversion API to convert the model from the PyTorch representation to the OpenVINO Intermediate Representation (OpenVINO IR)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9d7401c0",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"torch_model = Wav2Vec2ForCTC.from_pretrained(\"facebook/wav2vec2-base-960h\", ctc_loss_reduction=\"mean\")\n",
"processor = Wav2Vec2Processor.from_pretrained(\"facebook/wav2vec2-base-960h\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a2684e8a",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"BATCH_SIZE = 1\n",
"MAX_SEQ_LENGTH = 30480\n",
"ov_model = ov.convert_model(torch_model, example_input=torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float))\n",
"\n",
"ir_model_path = MODEL_DIR / \"wav2vec2_base.xml\"\n",
"ov.save_model(ov_model, ir_model_path)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "635f4b0d",
"metadata": {
"id": "LBbY7c4NsHzT"
},
"source": [
"## Prepare LibriSpeech Dataset\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"For demonstration purposes, we will use short dummy version of LibriSpeech dataset - `patrickvonplaten/librispeech_asr_dummy` to speed up model evaluation. Model accuracy can be different from reported in the paper. For reproducing original accuracy, use `librispeech_asr` dataset."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "43070514",
"metadata": {
"id": "NN-qRME1a-Sm"
},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"\n",
"\n",
"dataset = load_dataset(\"patrickvonplaten/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n",
"test_sample = dataset[0][\"audio\"]\n",
"\n",
"\n",
"# define preprocessing function for converting audio to input values for model\n",
"def map_to_input(batch):\n",
" preprocessed_signal = processor(\n",
" batch[\"audio\"][\"array\"],\n",
" return_tensors=\"pt\",\n",
" padding=\"longest\",\n",
" sampling_rate=batch[\"audio\"][\"sampling_rate\"],\n",
" )\n",
" input_values = preprocessed_signal.input_values\n",
" batch[\"input_values\"] = input_values\n",
" return batch\n",
"\n",
"\n",
"# apply preprocessing function to dataset and remove audio column, to save memory as we do not need it anymore\n",
"dataset = dataset.map(map_to_input, batched=False, remove_columns=[\"audio\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "9bbbca4a",
"metadata": {
"id": "CclWk-fVd9Wi"
},
"source": [
"## Run Quantization\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"[NNCF](https://github.com/openvinotoolkit/nncf) provides a suite of advanced algorithms for Neural Networks inference optimization in OpenVINO with minimal accuracy drop.\n",
"\n",
"Create a quantized model from the pre-trained `FP16` model and the calibration dataset. The optimization process contains the following steps:\n",
"\n",
"1. Create a Dataset for quantization.\n",
"2. Run `nncf.quantize` for getting an optimized model. The `nncf.quantize` function provides an interface for model quantization. It requires an instance of the OpenVINO Model and quantization dataset. Optionally, some additional parameters for the configuration quantization process (number of samples for quantization, preset, ignored scope, etc.) can be provided. For more accurate results, we should keep the operation in the postprocessing subgraph in floating point precision, using the `ignored_scope` parameter. For more information see [Tune quantization parameters](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.html#tune-quantization-parameters). For this model, ignored scope was selected experimentally, based on result of quantization with accuracy control. For understanding how it works please check following [notebook](../quantizing-model-with-accuracy-control/speech-recognition-quantization-wav2vec2.ipynb)\n",
"3. Serialize OpenVINO IR model using `ov.save_model` function."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "16457bd4",
"metadata": {
"id": "PiAvrwo0tr6Z",
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, openvino\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "69792104728e4d6f988d5a8f05ec9645",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"
\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import nncf\n",
"from nncf.parameters import ModelType\n",
"\n",
"\n",
"def transform_fn(data_item):\n",
" \"\"\"\n",
" Extract the model's input from the data item.\n",
" The data item here is the data item that is returned from the data source per iteration.\n",
" This function should be passed when the data item cannot be used as model's input.\n",
" \"\"\"\n",
" return np.array(data_item[\"input_values\"])\n",
"\n",
"\n",
"calibration_dataset = nncf.Dataset(dataset, transform_fn)\n",
"\n",
"quantized_model = nncf.quantize(\n",
" ov_model,\n",
" calibration_dataset,\n",
" model_type=ModelType.TRANSFORMER, # specify additional transformer patterns in the model\n",
" ignored_scope=nncf.IgnoredScope(\n",
" names=[\n",
" \"__module.wav2vec2.feature_extractor.conv_layers.1.conv/aten::_convolution/Convolution\",\n",
" \"__module.wav2vec2.feature_extractor.conv_layers.2.conv/aten::_convolution/Convolution\",\n",
" \"__module.wav2vec2.feature_extractor.conv_layers.3.conv/aten::_convolution/Convolution\",\n",
" \"__module.wav2vec2.feature_extractor.conv_layers.0.conv/aten::_convolution/Convolution\",\n",
" ],\n",
" ),\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "a05b3999",
"metadata": {
"id": "hPj_fcDAG8xG"
},
"outputs": [],
"source": [
"MODEL_NAME = \"quantized_wav2vec2_base\"\n",
"quantized_model_path = Path(f\"{MODEL_NAME}_openvino_model/{MODEL_NAME}_quantized.xml\")\n",
"ov.save_model(quantized_model, quantized_model_path)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "754b4d84",
"metadata": {},
"source": [
"## Model Usage Example with Inference Pipeline\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"Both initial (`FP16`) and quantized (`INT8`) models are exactly the same in use.\n",
"\n",
"Start with taking one example from the dataset to show inference steps for it."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "0431ac4f",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ipd.Audio(test_sample[\"array\"], rate=16000)"
]
},
{
"cell_type": "markdown",
"id": "21ae1447-e1e5-4cba-a469-1ce4a5a6c622",
"metadata": {},
"source": [
"For running model with OpenVINO, we should select inference device first. Please select one from available devices from dropdown list:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "26bc7dfa-dfcc-4222-9e12-c5239a2f42ac",
"metadata": {},
"outputs": [],
"source": [
"import ipywidgets as widgets\n",
"\n",
"core = ov.Core()\n",
"device = widgets.Dropdown(\n",
" options=core.available_devices + [\"AUTO\"],\n",
" value=\"AUTO\",\n",
" description=\"Device:\",\n",
" disabled=False,\n",
")\n",
"\n",
"device"
]
},
{
"cell_type": "markdown",
"id": "a3dd02aa-ea36-46d4-8ac6-bfd3c80305b0",
"metadata": {},
"source": [
"Next, load the quantized model to the inference pipeline."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "99f89c0f",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"compiled_model = core.compile_model(model=quantized_model, device_name=device.value)\n",
"\n",
"input_data = np.expand_dims(test_sample[\"array\"], axis=0)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7d8274a6",
"metadata": {},
"source": [
"Next, make a prediction."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "4fc43396",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['BECAUSE YOU ARE SLEEPING INSTEAD OF CONQUERING THE LOVELY ROSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BEOW WHILE POOR SHAGGY SITS THERE A COOING DOVE']\n"
]
}
],
"source": [
"predictions = compiled_model(input_data)[0]\n",
"predicted_ids = np.argmax(predictions, axis=-1)\n",
"transcription = processor.batch_decode(torch.from_numpy(predicted_ids))\n",
"print(transcription)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4f67f6a2",
"metadata": {
"id": "vQACMfAUo52V",
"tags": []
},
"source": [
"## Validate model accuracy on dataset\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"For model accuracy evaluation, [Word Error Rate](https://en.wikipedia.org/wiki/Word_error_rate) metric can be used. Word Error Rate or WER is the ratio of errors in a transcript to the total words spoken. A lower WER in speech-to-text means better accuracy in recognizing speech.\n",
"\n",
"For WER calculation, we will use [`torchmetrics`](https://torchmetrics.readthedocs.io/en/stable/text/word_error_rate.html) library."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "a6047d49",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"from torchmetrics import WordErrorRate\n",
"from tqdm.notebook import tqdm\n",
"\n",
"\n",
"# inference function for pytorch\n",
"def torch_infer(model, sample):\n",
" logits = model(torch.Tensor(sample[\"input_values\"])).logits\n",
" # take argmax and decode\n",
" predicted_ids = torch.argmax(logits, dim=-1)\n",
" transcription = processor.batch_decode(predicted_ids)\n",
" return transcription\n",
"\n",
"\n",
"# inference function for openvino\n",
"def ov_infer(model, sample):\n",
" logits = model(np.array(sample[\"input_values\"]))[0]\n",
" predicted_ids = np.argmax(logits, axis=-1)\n",
" transcription = processor.batch_decode(torch.from_numpy(predicted_ids))\n",
" return transcription\n",
"\n",
"\n",
"def compute_wer(dataset, model, infer_fn):\n",
" wer = WordErrorRate()\n",
" for sample in tqdm(dataset):\n",
" # run infer function on sample\n",
" transcription = infer_fn(model, sample)\n",
" # update metric on sample result\n",
" wer.update(transcription, [sample[\"text\"]])\n",
" # finalize metric calculation\n",
" result = wer.compute()\n",
" return result"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ca4095f9",
"metadata": {},
"source": [
"Now, you just need to decode predicted probabilities to text, using tokenizer `decode_logits`.\n",
"\n",
"Alternatively, use a built-in `Wav2Vec2Processor` tokenizer from the `transformers` package."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3430624f",
"metadata": {},
"source": [
"Now, compute WER for the original PyTorch model, OpenVINO IR model and quantized model."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "6a0ced60",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/maleksandr/test_notebooks/speech-recognition/openvino_notebooks/notebooks/speech-recognition-quantization/venv/lib/python3.10/site-packages/torchmetrics/utilities/prints.py:62: FutureWarning: Importing `WordErrorRate` from `torchmetrics` was deprecated and will be removed in 2.0. Import `WordErrorRate` from `torchmetrics.text` instead.\n",
" _future_warning(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e3ebf27209884bca8bf5c35a081bea9e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/73 [00:00 **NOTE**: For more accurate performance, it is recommended to run `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app -m model.xml -d CPU` to benchmark async inference on CPU for one minute. Change `CPU` to `GPU` to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command-line options."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "b46b4a90",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Step 1/11] Parsing and validating input arguments\n",
"[ INFO ] Parsing input parameters\n",
"[Step 2/11] Loading OpenVINO Runtime\n",
"[ INFO ] OpenVINO:\n",
"[ INFO ] Build ................................. 2023.3.0-13649-bbddb891712\n",
"[ INFO ] \n",
"[ INFO ] Device info:\n",
"[ INFO ] CPU\n",
"[ INFO ] Build ................................. 2023.3.0-13649-bbddb891712\n",
"[ INFO ] \n",
"[ INFO ] \n",
"[Step 3/11] Setting device configuration\n",
"[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
"[Step 4/11] Reading model files\n",
"[ INFO ] Loading model files\n",
"[ INFO ] Read model took 39.87 ms\n",
"[ INFO ] Original model I/O parameters:\n",
"[ INFO ] Model inputs:\n",
"[ INFO ] input_values (node: input_values) : f32 / [...] / [?,?]\n",
"[ INFO ] Model outputs:\n",
"[ INFO ] logits , 1170 (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [?,?,32]\n",
"[Step 5/11] Resizing model to match image sizes and given batch\n",
"[ INFO ] Model batch size: 1\n",
"[ INFO ] Reshaping model: 'input_values': [1,30480]\n",
"[ INFO ] Reshape model took 30.61 ms\n",
"[Step 6/11] Configuring input of the model\n",
"[ INFO ] Model inputs:\n",
"[ INFO ] input_values (node: input_values) : f32 / [...] / [1,30480]\n",
"[ INFO ] Model outputs:\n",
"[ INFO ] logits , 1170 (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [1,95,32]\n",
"[Step 7/11] Loading the model to the device\n",
"[ INFO ] Compile model took 743.32 ms\n",
"[Step 8/11] Querying optimal runtime parameters\n",
"[ INFO ] Model:\n",
"[ INFO ] NETWORK_NAME: Model0\n",
"[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
"[ INFO ] NUM_STREAMS: 12\n",
"[ INFO ] AFFINITY: Affinity.CORE\n",
"[ INFO ] INFERENCE_NUM_THREADS: 36\n",
"[ INFO ] PERF_COUNT: NO\n",
"[ INFO ] INFERENCE_PRECISION_HINT: \n",
"[ INFO ] PERFORMANCE_HINT: THROUGHPUT\n",
"[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
"[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
"[ INFO ] ENABLE_CPU_PINNING: True\n",
"[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
"[ INFO ] ENABLE_HYPER_THREADING: True\n",
"[ INFO ] EXECUTION_DEVICES: ['CPU']\n",
"[ INFO ] CPU_DENORMALS_OPTIMIZATION: False\n",
"[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0\n",
"[Step 9/11] Creating infer requests and preparing input tensors\n",
"[ WARNING ] No input files were given for input 'input_values'!. This input will be filled with random values!\n",
"[ INFO ] Fill input 'input_values' with random values \n",
"[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 60000 ms duration)\n",
"[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
"[ INFO ] First inference took 78.50 ms\n",
"[Step 11/11] Dumping statistics report\n",
"[ INFO ] Execution Devices:['CPU']\n",
"[ INFO ] Count: 3228 iterations\n",
"[ INFO ] Duration: 60314.79 ms\n",
"[ INFO ] Latency:\n",
"[ INFO ] Median: 223.21 ms\n",
"[ INFO ] Average: 223.81 ms\n",
"[ INFO ] Min: 92.71 ms\n",
"[ INFO ] Max: 261.87 ms\n",
"[ INFO ] Throughput: 53.52 FPS\n"
]
}
],
"source": [
"# Inference FP16 model (OpenVINO IR)\n",
"! benchmark_app -m $ir_model_path -shape [1,30480] -d $device.value -api async"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "83bf78ec",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Step 1/11] Parsing and validating input arguments\n",
"[ INFO ] Parsing input parameters\n",
"[Step 2/11] Loading OpenVINO Runtime\n",
"[ INFO ] OpenVINO:\n",
"[ INFO ] Build ................................. 2023.3.0-13649-bbddb891712\n",
"[ INFO ] \n",
"[ INFO ] Device info:\n",
"[ INFO ] CPU\n",
"[ INFO ] Build ................................. 2023.3.0-13649-bbddb891712\n",
"[ INFO ] \n",
"[ INFO ] \n",
"[Step 3/11] Setting device configuration\n",
"[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
"[Step 4/11] Reading model files\n",
"[ INFO ] Loading model files\n",
"[ INFO ] Read model took 55.69 ms\n",
"[ INFO ] Original model I/O parameters:\n",
"[ INFO ] Model inputs:\n",
"[ INFO ] input_values (node: input_values) : f32 / [...] / [?,?]\n",
"[ INFO ] Model outputs:\n",
"[ INFO ] logits , 1170 (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [?,?,32]\n",
"[Step 5/11] Resizing model to match image sizes and given batch\n",
"[ INFO ] Model batch size: 1\n",
"[ INFO ] Reshaping model: 'input_values': [1,30480]\n",
"[ INFO ] Reshape model took 36.51 ms\n",
"[Step 6/11] Configuring input of the model\n",
"[ INFO ] Model inputs:\n",
"[ INFO ] input_values (node: input_values) : f32 / [...] / [1,30480]\n",
"[ INFO ] Model outputs:\n",
"[ INFO ] logits , 1170 (node: __module.lm_head/aten::linear/Add) : f32 / [...] / [1,95,32]\n",
"[Step 7/11] Loading the model to the device\n",
"[ INFO ] Compile model took 1300.12 ms\n",
"[Step 8/11] Querying optimal runtime parameters\n",
"[ INFO ] Model:\n",
"[ INFO ] NETWORK_NAME: Model0\n",
"[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
"[ INFO ] NUM_STREAMS: 12\n",
"[ INFO ] AFFINITY: Affinity.CORE\n",
"[ INFO ] INFERENCE_NUM_THREADS: 36\n",
"[ INFO ] PERF_COUNT: NO\n",
"[ INFO ] INFERENCE_PRECISION_HINT: \n",
"[ INFO ] PERFORMANCE_HINT: THROUGHPUT\n",
"[ INFO ] EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
"[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
"[ INFO ] ENABLE_CPU_PINNING: True\n",
"[ INFO ] SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
"[ INFO ] ENABLE_HYPER_THREADING: True\n",
"[ INFO ] EXECUTION_DEVICES: ['CPU']\n",
"[ INFO ] CPU_DENORMALS_OPTIMIZATION: False\n",
"[ INFO ] CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0\n",
"[Step 9/11] Creating infer requests and preparing input tensors\n",
"[ WARNING ] No input files were given for input 'input_values'!. This input will be filled with random values!\n",
"[ INFO ] Fill input 'input_values' with random values \n",
"[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 60000 ms duration)\n",
"[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
"[ INFO ] First inference took 81.38 ms\n",
"[Step 11/11] Dumping statistics report\n",
"[ INFO ] Execution Devices:['CPU']\n",
"[ INFO ] Count: 4500 iterations\n",
"[ INFO ] Duration: 60142.86 ms\n",
"[ INFO ] Latency:\n",
"[ INFO ] Median: 159.61 ms\n",
"[ INFO ] Average: 160.08 ms\n",
"[ INFO ] Min: 81.89 ms\n",
"[ INFO ] Max: 204.87 ms\n",
"[ INFO ] Throughput: 74.82 FPS\n"
]
}
],
"source": [
"# Inference INT8 model (OpenVINO IR)\n",
"! benchmark_app -m $quantized_model_path -shape [1,30480] -d $device.value -api async"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"openvino_notebooks": {
"imageUrl": "",
"tags": {
"categories": [
"Optimize"
],
"libraries": [],
"other": [],
"tasks": [
"Speech Recognition"
]
}
},
"vscode": {
"interpreter": {
"hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
}
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}