{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cacbe6b4",
   "metadata": {
    "id": "rQc-wXjqrEuR"
   },
   "source": [
    "# Accelerate Inference of Sparse Transformer Models with OpenVINO™ and 4th Gen Intel® Xeon® Scalable Processors\n",
    "This tutorial demonstrates how to improve performance of sparse Transformer models with [OpenVINO](https://docs.openvino.ai/) on 4th Gen Intel® Xeon® Scalable processors.\n",
    "\n",
    "The tutorial downloads [a BERT-base model](https://huggingface.co/OpenVINO/bert-base-uncased-sst2-int8-unstructured80) which has been quantized, sparsified, and tuned for [SST2 datasets](https://huggingface.co/datasets/sst2) using [Optimum-Intel](https://github.com/huggingface/optimum-intel). It demonstrates the inference performance advantage on 4th Gen Intel® Xeon® Scalable Processors by running it with [Sparse Weight Decompression](https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.html#sparse-weights-decompression-intel-x86-64), a runtime option that seizes model sparsity for efficiency. The notebook consists of the following steps:\n",
    "\n",
    "- Install prerequisites\n",
    "- Download and quantize sparse public BERT model, using the OpenVINO integration with Hugging Face Optimum.\n",
    "- Compare sparse 8-bit vs. dense 8-bit inference performance.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5f9d5f14",
   "metadata": {},
   "source": [
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Imports](#Imports)\n",
    "    - [Download, quantize and sparsify the model, using Hugging Face Optimum API](#Download,-quantize-and-sparsify-the-model,-using-Hugging-Face-Optimum-API)\n",
    "- [Benchmark quantized dense inference performance](#Benchmark-quantized-dense-inference-performance)\n",
    "- [Benchmark quantized sparse inference performance](#Benchmark-quantized-sparse-inference-performance)\n",
    "- [When this might be helpful](#When-this-might-be-helpful)\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7bef22e9",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fc9afb5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install -q \"openvino>=2023.1.0\"\n",
    "%pip install -q \"git+https://github.com/huggingface/optimum-intel.git\" \"torch>=2.1\" datasets onnx transformers>=4.33.0 --extra-index-url https://download.pytorch.org/whl/cpu"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4d6b41e6-132b-40da-b3b9-91bacba29e31",
   "metadata": {},
   "source": [
    "## Imports\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "771388d6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "from pathlib import Path\n",
    "\n",
    "from optimum.intel.openvino import OVModelForSequenceClassification\n",
    "from transformers import AutoTokenizer, pipeline\n",
    "from huggingface_hub import hf_hub_download"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7603a481",
   "metadata": {},
   "source": [
    "### Download, quantize and sparsify the model, using Hugging Face Optimum API\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "040a1020-0c12-4f3d-ae9f-9464d2e18b10",
   "metadata": {},
   "source": [
    "The first step is to download a quantized sparse transformers which has been translated to OpenVINO IR. Then, it will be put through a classification as a simple validation of a working downloaded model. To find out how the model is being quantized and sparsified, refer to the [OpenVINO/bert-base-uncased-sst2-int8-unstructured80](https://huggingface.co/OpenVINO/bert-base-uncased-sst2-int8-unstructured80) model card on Hugging Face.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b897c926",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# The following model has been quantized, sparsified using Optimum-Intel 1.7 which is enabled by OpenVINO and NNCF\n",
    "# for reproducibility, refer https://huggingface.co/OpenVINO/bert-base-uncased-sst2-int8-unstructured80\n",
    "model_id = \"OpenVINO/bert-base-uncased-sst2-int8-unstructured80\"\n",
    "\n",
    "# The following two steps will set up the model and download them to HF Cache folder\n",
    "ov_model = OVModelForSequenceClassification.from_pretrained(model_id)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "\n",
    "# Let's take the model for a spin!\n",
    "sentiment_classifier = pipeline(\"text-classification\", model=ov_model, tokenizer=tokenizer)\n",
    "\n",
    "text = \"He's a dreadful magician.\"\n",
    "outputs = sentiment_classifier(text)\n",
    "\n",
    "print(outputs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "227d97eb-e91e-48bb-bba9-30598e49bb4f",
   "metadata": {},
   "source": [
    "For benchmarking, we will use OpenVINO's benchmark application and put the IRs into a single folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2caade15-cb72-4d49-8400-21ce56b9c220",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# create a folder\n",
    "quantized_sparse_dir = Path(\"bert_80pc_sparse_quantized_ir\")\n",
    "quantized_sparse_dir.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "# following return path to specified filename in cache folder (which we've with the\n",
    "ov_ir_xml_path = hf_hub_download(repo_id=model_id, filename=\"openvino_model.xml\")\n",
    "ov_ir_bin_path = hf_hub_download(repo_id=model_id, filename=\"openvino_model.bin\")\n",
    "\n",
    "# copy IRs to the folder\n",
    "shutil.copy(ov_ir_xml_path, quantized_sparse_dir)\n",
    "shutil.copy(ov_ir_bin_path, quantized_sparse_dir)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a6830eb7",
   "metadata": {},
   "source": [
    "## Benchmark quantized dense inference performance\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Benchmark dense inference performance using parallel execution on four CPU cores to simulate a small instance in the cloud infrastructure. Sequence length is dependent on use cases, 16 is common for conversational AI while 160 for question answering task. It is set to 64 as an example. It is recommended to tune based on your applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa895f88",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Dump benchmarking config for dense inference\n",
    "with (quantized_sparse_dir / \"perf_config.json\").open(\"w\") as outfile:\n",
    "    outfile.write(\n",
    "        \"\"\"\n",
    "        {\n",
    "            \"CPU\": {\"NUM_STREAMS\": 4, \"INFERENCE_NUM_THREADS\": 4}\n",
    "        }\n",
    "        \"\"\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f6c7526",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!benchmark_app -m $quantized_sparse_dir/openvino_model.xml -shape \"input_ids[1,64],attention_mask[1,64],token_type_ids[1,64]\" -load_config $quantized_sparse_dir/perf_config.json"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f9151b11",
   "metadata": {},
   "source": [
    "## Benchmark quantized sparse inference performance\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c305661a-27a5-45aa-b3df-777862584adf",
   "metadata": {},
   "source": [
    "To enable sparse weight decompression feature, users can add it to runtime config like below. `CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE` takes values between 0.5 and 1.0. It is a layer-level sparsity threshold for which a layer will be enabled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad77ae5f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Dump benchmarking config for dense inference\n",
    "# \"CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE\" controls minimum sparsity rate for weights to consider\n",
    "# for sparse optimization at the runtime.\n",
    "with (quantized_sparse_dir / \"perf_config_sparse.json\").open(\"w\") as outfile:\n",
    "    outfile.write(\n",
    "        \"\"\"\n",
    "        {\n",
    "            \"CPU\": {\"NUM_STREAMS\": 4, \"INFERENCE_NUM_THREADS\": 4, \"CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE\": 0.75}\n",
    "        }\n",
    "        \"\"\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ddd8b10",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!benchmark_app -m $quantized_sparse_dir/openvino_model.xml -shape \"input_ids[1,64],attention_mask[1,64],token_type_ids[1,64]\" -load_config $quantized_sparse_dir/perf_config_sparse.json"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cc1d4d61",
   "metadata": {},
   "source": [
    "## When this might be helpful\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "135c8526",
   "metadata": {},
   "source": [
    "This feature can improve inference performance for models with sparse weights in the scenarios when the model is deployed to handle multiple requests in parallel asynchronously. It is especially helpful with a small sequence length, for example, 32 and lower.\n",
    "\n",
    "For more details about asynchronous inference with OpenVINO, refer to the following documentation:\n",
    "\n",
    "- [Deployment Optimization Guide](https://docs.openvino.ai/2024/openvino-workflow/running-inference/optimize-inference/general-optimizations.html)\n",
    "- [Inference Request API](https://docs.openvino.ai/2024/openvino-workflow/running-inference/integrate-openvino-with-your-application/inference-request.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "openvino_notebooks": {
   "imageUrl": "",
   "tags": {
    "categories": [
     "Optimize"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Text Classification"
    ]
   }
  },
  "vscode": {
   "interpreter": {
    "hash": "cec18e25feb9469b5ff1085a8097bdcd86db6a4ac301d6aeff87d0f3e7ce4ca5"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}