ASR/transformer-deploy/docs/t5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Inference acceleration of `T5` for large batch size / long sequence length / > large models\n",
    "\n",
    "Every week or so, a new impressive few shots learning work taking advantage of autoregressive models is released by some team around the world.  \n",
    "Still `LLM` inference is rarely discussed and few projects are focusing on this aspect.  \n",
    "In this notebook, we describe our take to significantly improve autoregressive model latency.\n",
    "\n",
    "We plan to intensively test large autoregressive models, so we want something:\n",
    "\n",
    "* which **scales**: the improvement exists on small and large models, for short and long sequences, in greedy and beam search;\n",
    "    * This is very important in a few shots learning where sequences are most of the time hundreds or thousands tokens long and beam search is used to improve text quality.\n",
    "* that has **no hidden cost**: no big increase in memory usage, no degradation in quality of generated text, support state-of-the-art decoding algorithms;\n",
    "* that is **generic**: works for any transformer based architecture, and not specific to an inference engine;\n",
    "* that is **easy to maintain**: no hard-coded behaviors or other technical debt if it doesn't bring a clear advantage.\n",
    "\n",
    "To be clear, **we are not targeting the best performance ever but the right trade off** (for us at least) between simplicity to use/maintain and acceptable latency.\n",
    "\n",
    "## The challenge\n",
    "\n",
    "In most situations, performing inference with `ONNX Runtime` or `TensorRT` usually bring large improvement over `Pytorch` implementations.\n",
    "It's very true with `transformer` based models.\n",
    "\n",
    "The main reason is that these tools will perform `kernel fusions` (merging several operations into a single one) and therefore reduce the number of memory bounded operations. Sometimes they also replace some operations by a much faster approximation.\n",
    "In the very specific case of autoregressive languages, things are a bit more complicated.\n",
    "\n",
    "On most `Pytorch` implementations of these models, there is a `cache` of `K` and `V` values.\n",
    "Let's remind us that in attention blocks, each token is projected on 3 matrices called `Query`, `Key`, and `Value`.\n",
    "Then, those projections will be used to compute a representation of each token which takes into account the information from the related other tokens of the sequence.\n",
    "\n",
    "As autoregressive models generate the sequence one token at a time, they should recompute final representation of all past tokens for each new token to generate.\n",
    "Because each token can only attend to the past, the result of these computations never changes; therefore one simple trick to reduce latency is to just memorize them and reuse them later, avoiding lots of computation.\n",
    "\n",
    "Out of the box, the cache mechanism can't be exported to `ONNX` from `Hugging Face` models (and all other `Pytorch` implementations we are aware of).\n",
    "The reason is that those models are not `torchscript` scripting compliant (it requires `Pytorch` code to follow some [restrictive rules](https://pytorch.org/docs/stable/jit_builtin_functions.html)).\n",
    "Because of that, `ONNX` export is done through `tracing` which erases any control flow instructions (including the `If` instruction to enable or not a cache).\n",
    "\n",
    "## Existing solutions\n",
    "\n",
    "Some interesting solutions targeting inference latency that we have considered and/or tested:\n",
    "\n",
    "* [TensorRT](https://developer.nvidia.com/blog/optimizing-t5-and-gpt-2-for-real-time-inference-with-`TensorRT`/), which targets `GPU`, heavily optimizes the computation graph, making `T5` inference very fast (they report X10 speedup on `small-T5`). The trick is that it doesn't use any cache (see below for more details), so it's very fast on short sequence and small models, as it avoids many memory bounded operations by redoing full computation again and again... but as several users have already found ([1](https://github.com/NVIDIA/TensorRT/issues/1807), [2](https://github.com/NVIDIA/TensorRT/issues/1642), [3](https://github.com/NVIDIA/TensorRT/issues/1799), [4](https://github.com/NVIDIA/TensorRT/issues/1845), ...), this approach doesn't scale when the computation intensity increases, i.e., when base or large models are used instead of a small one, when generation is done on moderately long sequence of few hundred of tokens or if beam search is used instead of a greedy search;\n",
    "* [FastT5](https://github.com/Ki6an/fastT5), which targets `CPU`, exports 2 versions of the decoder, one with cache and one without. You need the `no cache` version to compute the first token and the first `past state` tensors (aka the cached tensors), and for all the other tokens you use the `cache` version of the computation graph. Basically, it makes the memory foot print 2 times bigger as all weights are duplicated. As generative models tend to be huge, they work around the memory issue by using dynamic `int-8` quantization, the final memory foot print of the decoders is now the same as `Hugging Face` in `FP16`... but 1/ dynamic quantization only works on `CPU`, and 2/ according to several reports dynamic quantization degrades significantly generative model output, to a point where it may make them useless ([1](https://github.com/huggingface/transformers/issues/2466#issuecomment-572781378), [2](https://github.com/huggingface/transformers/issues/2466#issuecomment-982710520), and [here](https://github.com/microsoft/onnxruntime/issues/6549#issuecomment-1016948837) you can find a report in the `GPT-2` context from a Microsoft engineer: \"*int8 quantization are not recommended due to accuracy loss*\").\n",
    "* [ONNX Runtime T5 export tool](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/models/t5) targets both `GPU` and `CPU`. It works in a similar way than `FastT5`: `decoder` module is exported 2 times. Like `FastT5`, the memory footprint of the decoder part is doubled (this time there is no `int-8` quantization).\n",
    "* [FasterTransformer](https://github.com/NVIDIA/FasterTransformer/blob/main/docs/t5_guide.md#translation-process) targets `GPU` and is a mix of `Pytorch` and `CUDA`/`C++` dedicated code. The performance boost is huge on `T5`, they report a 10X speedup like `TensorRT`. However, it may significantly decrease the accuracy of the model ([here](https://github.com/NVIDIA/FasterTransformer/blob/main/docs/t5_guide.md#translation-process) when sampling is enabled, it reduces BLEU score of translation task by 8 points, the cause may be a bug in the decoding algorithm or an approximation a bit too aggressive) plus the speedup is computed on a [translation task](https://github.com/NVIDIA/FasterTransformer/blob/main/examples/pytorch/decoding/utils/translation/test.en) where sequences are 25 tokens long on average. In our experience, improvement on very short sequences tends to decrease by large margins on longer sequences. It seems to us that their objectives are different from ours.\n",
    "\n",
    "With the existing solutions, you need to choose one or two items of the following:\n",
    "\n",
    "* double decoder memory footprint;\n",
    "* be slower than `Hugging Face` for moderately long sequence length / beam search;\n",
    "* degrade output quality.\n",
    "\n",
    "## Our approach\n",
    "\n",
    "Our approach to make autoregressive `transformer` based models 2.8X faster than `Hugging Face` `Pytorch` implementation (the base line) is based on 3 key ingredients:\n",
    "\n",
    "* storing 2 computation graphs in a single `ONNX` file: this let us have both cache and no cache support without having any duplicated weights,\n",
    "* `zero copy` to retrieve output from `ONNX Runtime`: we built over our past work to connect in the most efficient way `Pytorch` tensors (used in the decoding part) and `ONNX Runtime`. Our previous work was to avoid `host` <-> `GPU` tensor copy, but it still required a `GPU` <-> `GPU`. It is now part of the official `ONNX Runtime` documentation (apparently [thanks of our project](https://github.com/microsoft/onnxruntime/pull/10651)!). This time we found out a way to directly expose the internal state of `ONNX Runtime` through a `Pytorch` tensor in zero copy way. Combined with cache mechanism, this is responsible for most of the speedup we have obtained.\n",
    "* a generic tool to convert any model (whatever the architecture) to `FP16` without any risk of having out of range values or rounding to zero: `FP16` is still the way to reduce memory footprint of a model. The main issue is that some nodes may output values outside of `FP16` range or round others to zero, resulting in `NaN` output; moreover, very small values may be rounded to zero which is an issue for log and div operations. We have built a tool which detect those nodes so we can keep their precision in `FP32`. It's quite important to reduce memory footprint of these models, not just because they tend to be huge, but also because past states (that we cache) and internal buffers can be even bigger than the weights of the model itself.\n",
    "\n",
    "## Results\n",
    "\n",
    "As demonstrated at the end of this notebook, **we are able to provide a X2.8 speedup** whatever the batch size, the sequence length or the model size.\n",
    "\n",
    "> For `TensorRT` we have our own implementation of our approach described above which helps to provide similar latency to `ONNX Runtime`. It's in a Python script in the same folder as this notebook. We had to work around a documented limitation. Because of that the code is slightly more complex and we wanted to keep this notebook easy to follow.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:02:20.975344Z",
     "iopub.status.busy": "2022-06-25T23:02:20.974743Z",
     "iopub.status.idle": "2022-06-25T23:02:21.579668Z",
     "shell.execute_reply": "2022-06-25T23:02:21.578207Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sun Jun 26 01:02:21 2022       \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7     |\r\n",
      "|-------------------------------+----------------------+----------------------+\r\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n",
      "|                               |                      |               MIG M. |\r\n",
      "|===============================+======================+======================|\r\n",
      "|   0  NVIDIA GeForce ...  On   | 00000000:03:00.0  On |                  N/A |\r\n",
      "| 45%   49C    P8    42W / 350W |    234MiB / 24576MiB |     24%      Default |\r\n",
      "|                               |                      |                  N/A |\r\n",
      "+-------------------------------+----------------------+----------------------+\r\n",
      "                                                                               \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| Processes:                                                                  |\r\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\r\n",
      "|        ID   ID                                                   Usage      |\r\n",
      "|=============================================================================|\r\n",
      "|    0   N/A  N/A      2422      G   /usr/lib/xorg/Xorg                 96MiB |\r\n",
      "|    0   N/A  N/A      8156      G   /usr/bin/gnome-shell               35MiB |\r\n",
      "|    0   N/A  N/A      9002      G   ...on/Bin/AgentConnectix.bin        4MiB |\r\n",
      "|    0   N/A  N/A     85822      G   ...458042145133945570,131072       94MiB |\r\n",
      "+-----------------------------------------------------------------------------+\r\n"
     ]
    }
   ],
   "source": [
    "! nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## `ONNX Runtime`\n",
    "\n",
    "Version 1.11.1 of `ONNX Runtime` and older have a bug which makes them much slower when most inputs are used by subgraphs of an `If` node.  \n",
    "Use a version >= 1.12.0 instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:02:21.585535Z",
     "iopub.status.busy": "2022-06-25T23:02:21.585167Z",
     "iopub.status.idle": "2022-06-25T23:02:24.417678Z",
     "shell.execute_reply": "2022-06-25T23:02:24.417030Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import random\n",
    "from transformer_deploy.backends.ort_utils import get_keep_fp32_nodes\n",
    "from transformer_deploy.backends.ort_utils import convert_fp16\n",
    "import time\n",
    "from typing import Callable, Dict, Optional, List\n",
    "import matplotlib.pylab as plt\n",
    "from onnxruntime import IOBinding\n",
    "import numpy as np\n",
    "import torch\n",
    "from pathlib import Path\n",
    "from typing import Tuple\n",
    "from transformer_deploy.backends.onnx_utils import save_onnx, merge_autoregressive_model_graphs\n",
    "from transformer_deploy.backends.ort_utils import search_fp32_nodes\n",
    "from transformer_deploy.backends.ort_utils import add_output_nodes\n",
    "import onnx\n",
    "\n",
    "from torch.nn import Linear\n",
    "from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, PretrainedConfig, T5ForConditionalGeneration, TensorType\n",
    "from transformers.generation_utils import GenerationMixin\n",
    "from transformers.modeling_outputs import BaseModelOutputWithPastAndCrossAttentions, Seq2SeqLMOutput\n",
    "from transformers.models.t5.modeling_t5 import T5Stack\n",
    "from nvtx import nvtx\n",
    "from copy import copy\n",
    "\n",
    "from transformer_deploy.backends.ort_utils import create_model_for_provider, inference_onnx_binding\n",
    "from transformer_deploy.backends.pytorch_utils import convert_to_onnx\n",
    "import seaborn as sns\n",
    "import operator\n",
    "from collections import defaultdict\n",
    "import gc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Loading `Hugging Face` model / tokenizer\n",
    "\n",
    "Below we load the model and set global variables of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:02:24.423237Z",
     "iopub.status.busy": "2022-06-25T23:02:24.422920Z",
     "iopub.status.idle": "2022-06-25T23:03:19.364579Z",
     "shell.execute_reply": "2022-06-25T23:03:19.364011Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(123)\n",
    "torch.random.manual_seed(123)\n",
    "# other possible values: t5-small, t5-base, t5-large. t5-3b should work when ORT library is fixed\n",
    "model_name = \"t5-3b\"\n",
    "# shape of output generation for benchmarks (seqlen, batch).\n",
    "# You can provide 2 tasks to benchmark.\n",
    "# On 3b model, 128 tokens with batch 2 is the maximum without OOM Exception on GPU with 24Gb of RAM.\n",
    "# On smaller models/more ram GPUs, use bigger values like [(1024, 1), (1024, 4)]\n",
    "benchmark_tasks = [(128, 1), (128, 2)]\n",
    "# T5 has no max len\n",
    "# https://github.com/huggingface/transformers/issues/5204\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name, model_max_length=512)\n",
    "input_ids: torch.Tensor = tokenizer(\n",
    "    'translate English to French: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new \"Colossal Clean Crawled Corpus\", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.',\n",
    "    return_tensors=TensorType.PYTORCH,\n",
    ").input_ids\n",
    "input_ids = input_ids.type(torch.int32)\n",
    "pytorch_model: T5ForConditionalGeneration = AutoModelForSeq2SeqLM.from_pretrained(model_name)\n",
    "pytorch_model = pytorch_model.eval()\n",
    "\n",
    "pytorch_model.config.use_cache = True  # not really needed, just to make things obvious\n",
    "num_layers = pytorch_model.config.num_layers\n",
    "# tolerance between ONNX FP16 and Pytorch FP32.\n",
    "# Rounding errors increase with number of layers: 1e-1 for t5-small, 5e-1 for large, 3 for 3b (depending of how aggressive you are on FP16, it can raises up to 5-7). 11b not tested.\n",
    "# Do not impact final quality\n",
    "fp16_default_tolerance = 10\n",
    "\n",
    "\n",
    "def are_equal(a: torch.Tensor, b: torch.Tensor, atol: float = fp16_default_tolerance) -> None:\n",
    "    assert np.allclose(a=a.detach().cpu().numpy(), b=b.detach().cpu().numpy(), atol=atol), f\"{a}\\n\\nVS\\n\\n{b}\"\n",
    "\n",
    "\n",
    "def prepare_folder(path: str) -> Tuple[str, str]:\n",
    "    p = Path(path)\n",
    "    p.mkdir(parents=True, exist_ok=True)\n",
    "    [item.unlink() for item in Path(path).glob(\"*\") if item.is_file()]\n",
    "    return path + \"/model.onnx\", path + \"/model_fp16.onnx\"\n",
    "\n",
    "\n",
    "# create/clean folders where each model will be stored.\n",
    "# as multiple files will be saved for T5-3B and 11B, we use different folders for the encoder and the decoders.\n",
    "encoder_model_path, encoder_fp16_model_path = prepare_folder(path=\"./test-enc\")\n",
    "dec_cache_model_path, dec_cache_fp16_model_path = prepare_folder(path=\"./test-dec-cache\")\n",
    "dec_no_cache_model_path, dec_no_cache_fp16_model_path = prepare_folder(path=\"./test-dec-no-cache\")\n",
    "dec_if_model_path, dec_if_fp16_model_path = prepare_folder(path=\"./test-dec-if\")\n",
    "\n",
    "# some outputs to compare with\n",
    "out_enc: BaseModelOutputWithPastAndCrossAttentions = pytorch_model.encoder(input_ids=input_ids)\n",
    "out_full: Seq2SeqLMOutput = pytorch_model(input_ids=input_ids, decoder_input_ids=input_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Export to ONNX\n",
    "\n",
    "First step is to export the model to `ONNX` graph.\n",
    "`T5` is made of 2 parts, an `encoder` and a `decoder`.\n",
    "\n",
    "## Export encoder part\n",
    "\n",
    "The `encoder` part export doesn't imply any specific challenge.\n",
    "We use export function built for `Bert` like model, exported model is in `FP16`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:03:19.367413Z",
     "iopub.status.busy": "2022-06-25T23:03:19.367191Z",
     "iopub.status.idle": "2022-06-25T23:03:52.884148Z",
     "shell.execute_reply": "2022-06-25T23:03:52.883494Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "convert_to_onnx(\n",
    "    model_pytorch=pytorch_model.encoder,\n",
    "    output_path=encoder_model_path,\n",
    "    inputs_pytorch={\"input_ids\": input_ids},\n",
    "    var_output_seq=True,\n",
    "    quantization=False,\n",
    "    output_names=[\"output\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Export decoder\n",
    "\n",
    "The decoder export part is more challenging:\n",
    "\n",
    "* we first need to wrap it in a `Pytorch` model to add the final layer so it's output provide scores for each vocabulary token and can be directly used by the `Hugging Face` `decoding` algorithm\n",
    "* then, we need to manipulate the `ONNX` graph to add support of `Key`/`Value` cache\n",
    "\n",
    "The second point is the key ingredient of the observed acceleration of `ONNX` vs `Hugging Face` inference.\n",
    "\n",
    "\n",
    "### Wrapper to include some post-processing on the decoder output\n",
    "\n",
    "The post-processing is mainly a projection of the decoder output on a matrix with one of its dimensions equal to model vocabulary size, so we have scores for each possible token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:03:52.889647Z",
     "iopub.status.busy": "2022-06-25T23:03:52.889304Z",
     "iopub.status.idle": "2022-06-25T23:03:55.282126Z",
     "shell.execute_reply": "2022-06-25T23:03:55.281480Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class ExportT5(torch.nn.Module):\n",
    "    def __init__(self, decoder: T5Stack, lm_head: Linear):\n",
    "        super(ExportT5, self).__init__()\n",
    "        self.decoder = decoder\n",
    "        self.lm_head = lm_head\n",
    "\n",
    "    def forward(self, input_ids: torch.Tensor, encoder_hidden_states: torch.Tensor, past_key_values: Tuple = None):\n",
    "        out_dec = self.decoder.forward(\n",
    "            input_ids=input_ids, encoder_hidden_states=encoder_hidden_states, past_key_values=past_key_values\n",
    "        )\n",
    "        # weight tying -> rescale output before projecting on vocab\n",
    "        # to comment for T0 for instance\n",
    "        out_dec[\"last_hidden_state\"] = out_dec[\"last_hidden_state\"] * (pytorch_model.model_dim**-0.5)\n",
    "        out_dec[\"last_hidden_state\"] = self.lm_head(out_dec[\"last_hidden_state\"])\n",
    "        return out_dec\n",
    "\n",
    "\n",
    "model_decoder = ExportT5(decoder=pytorch_model.decoder, lm_head=pytorch_model.lm_head).eval()\n",
    "out_model_export: torch.Tensor = model_decoder(input_ids=input_ids, encoder_hidden_states=out_enc.last_hidden_state)\n",
    "\n",
    "are_equal(a=out_model_export[\"last_hidden_state\"], b=out_full.logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Export decoder part to `ONNX`\n",
    "\n",
    "Below we export 2 versions of the decoder, one without cache support and one with it.\n",
    "\n",
    "Model inputs with past states (cache support):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:03:55.286261Z",
     "iopub.status.busy": "2022-06-25T23:03:55.285980Z",
     "iopub.status.idle": "2022-06-25T23:03:57.600567Z",
     "shell.execute_reply": "2022-06-25T23:03:57.599993Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# decoder output one step before\n",
    "out_dec_pytorch = model_decoder(input_ids=input_ids[:, :-1], encoder_hidden_states=out_enc.last_hidden_state)\n",
    "\n",
    "model_inputs = {\n",
    "    \"input_ids\": input_ids[:, -1:].type(torch.int32),\n",
    "    \"encoder_hidden_states\": out_enc.last_hidden_state,\n",
    "    \"past_key_values\": out_dec_pytorch.past_key_values,\n",
    "}\n",
    "\n",
    "input_names = [\"input_ids\", \"encoder_hidden_states\"]\n",
    "\n",
    "for i in range(num_layers):\n",
    "    input_names.append(f\"past_key_values.{i}.decoder.key\")\n",
    "    input_names.append(f\"past_key_values.{i}.decoder.value\")\n",
    "    input_names.append(f\"past_key_values.{i}.encoder.key\")\n",
    "    input_names.append(f\"past_key_values.{i}.encoder.value\")\n",
    "\n",
    "output_names = [\"logits\"]\n",
    "\n",
    "for i in range(num_layers):\n",
    "    output_names.append(f\"present.{i}.decoder.key\")\n",
    "    output_names.append(f\"present.{i}.decoder.value\")\n",
    "    output_names.append(f\"present.{i}.encoder.key\")\n",
    "    output_names.append(f\"present.{i}.encoder.value\")\n",
    "\n",
    "dynamic_axis = {\n",
    "    \"input_ids\": {0: \"batch\", 1: \"encoder_sequence\"},\n",
    "    \"encoder_hidden_states\": {0: \"batch\", 1: \"encoder_sequence\"},\n",
    "    \"logits\": {0: \"batch\", 1: \"decoder_sequence\"},\n",
    "}\n",
    "\n",
    "\n",
    "for i in range(num_layers):\n",
    "    dynamic_axis[f\"past_key_values.{i}.decoder.key\"] = {0: \"batch\", 2: \"past_decoder_sequence\"}\n",
    "    dynamic_axis[f\"past_key_values.{i}.decoder.value\"] = {0: \"batch\", 2: \"past_decoder_sequence\"}\n",
    "    dynamic_axis[f\"past_key_values.{i}.encoder.key\"] = {0: \"batch\", 2: \"encoder_sequence_length\"}\n",
    "    dynamic_axis[f\"past_key_values.{i}.encoder.value\"] = {0: \"batch\", 2: \"encoder_sequence_length\"}\n",
    "\n",
    "    dynamic_axis[f\"present.{i}.decoder.key\"] = {0: \"batch\", 2: \"decoder_sequence\"}\n",
    "    dynamic_axis[f\"present.{i}.decoder.value\"] = {0: \"batch\", 2: \"decoder_sequence\"}\n",
    "    dynamic_axis[f\"present.{i}.encoder.key\"] = {0: \"batch\", 2: \"encoder_sequence_length\"}\n",
    "    dynamic_axis[f\"present.{i}.encoder.value\"] = {0: \"batch\", 2: \"encoder_sequence_length\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Export of the model with cache support:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:03:57.603501Z",
     "iopub.status.busy": "2022-06-25T23:03:57.603304Z",
     "iopub.status.idle": "2022-06-25T23:04:51.077733Z",
     "shell.execute_reply": "2022-06-25T23:04:51.077086Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/geantvert/.local/share/virtualenvs/fast_transformer/lib/python3.9/site-packages/transformers/modeling_utils.py:781: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if causal_mask.shape[1] < attention_mask.shape[1]:\n",
      "In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode\n",
      "In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    pytorch_model.config.return_dict = True\n",
    "    pytorch_model.eval()\n",
    "\n",
    "    # export can works with named args but the dict containing named args as to be last element of the args tuple\n",
    "    torch.onnx.export(\n",
    "        model_decoder,\n",
    "        (model_inputs,),\n",
    "        f=dec_cache_model_path,\n",
    "        input_names=input_names,\n",
    "        output_names=output_names,\n",
    "        dynamic_axes=dynamic_axis,\n",
    "        do_constant_folding=True,\n",
    "        opset_version=13,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Export of the model computing Key/Values for the whole sequence (we basically just remove past states from the input, the `Pytorch` code will recompute them):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:04:51.080696Z",
     "iopub.status.busy": "2022-06-25T23:04:51.080503Z",
     "iopub.status.idle": "2022-06-25T23:05:52.002319Z",
     "shell.execute_reply": "2022-06-25T23:05:52.001626Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "model_inputs_no_cache = {\n",
    "    \"input_ids\": input_ids,\n",
    "    \"encoder_hidden_states\": out_enc.last_hidden_state,\n",
    "}\n",
    "\n",
    "with torch.no_grad():\n",
    "    pytorch_model.config.return_dict = True\n",
    "    pytorch_model.eval()\n",
    "\n",
    "    # export can works with named args but the dict containing named args as to be last element of the args tuple\n",
    "    torch.onnx.export(\n",
    "        model_decoder,\n",
    "        (model_inputs_no_cache,),\n",
    "        f=dec_no_cache_model_path,\n",
    "        input_names=list(model_inputs_no_cache.keys()),\n",
    "        output_names=output_names,\n",
    "        dynamic_axes={k: v for k, v in dynamic_axis.items() if \"past_key_values\" not in k},\n",
    "        do_constant_folding=True,\n",
    "        opset_version=13,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Conversion to mixed precision\n",
    "\n",
    "### Why mixed precision?\n",
    "\n",
    "As `T5` can have up to 11 billion parameters, it requires lots of computation, and even more important, it takes lots of space in device memory.\n",
    "We convert the `encoder` to half precision.\n",
    "\n",
    "If we blindly convert the whole graph to `FP16`, we will have 2 issues:\n",
    "* `overflow`: some nodes, like exponential nodes, will try to output values out of the `FP16` range, at the end you get some `NaN`.\n",
    "* `underflow`: values very close to 0 will be rounded to 0, which may be an issue for some operations like `Div` and `Log` .\n",
    "\n",
    "### The challenge\n",
    "\n",
    "Mixed precision is done out of the box by `Pytorch` and follow some strict rules described in https://pytorch.org/docs/stable/amp.html\n",
    "\n",
    "Those rules are generic and quite conservative. Many nodes will be kept in `FP32` even if their output is always in the `FP16` range.\n",
    "\n",
    "Other approaches we have found:\n",
    "* `ONNX Runtime T5` [demo](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/models/t5/t5_helper.py): provide a list of operations to keep in `FP32` (Pow, ReduceMean, Add, Sqrt, Div, Mul, Softmax, Relu). We have found this approach to need more an more tweaking on larger networks and encoder part (decoder part seems simpler to manage, https://github.com/microsoft/onnxruntime/issues/11119);\n",
    "* `TensorRT T5` [demo](https://github.com/NVIDIA/TensorRT/tree/main/demo/HuggingFace/notebooks): provide the exact pattern of nodes to keep in `FP32`. This approach is much more effective, but imply lots of code to describe the patterns and may not generalize well, basically what works for a `base` model may not work for 11 billion parameters model. And it does not scale to other architectures without adaptations, for a library like `transformer-deploy`, it would lead to unmaintainable technical debt.\n",
    "\n",
    "\n",
    "### Our approach\n",
    "\n",
    "We have chosen an architecture agnostic approach: we inject random input sequences and audit the output of each computation graph node; finally, we make a list of all nodes that have output values out of the `FP16` range /close to zero values and perform some cleaning (to avoid unnecessary casting).\n",
    "\n",
    "We have chosen to use random values only for the `input_ids` field as the search space is limited: positive integers lower than the vocabulary size.\n",
    "You can also decide to send real data from a dataset you want to work on.\n",
    "\n",
    "To finish, we provide the list of nodes to keep in `FP32` to the conversion function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:05:52.007617Z",
     "iopub.status.busy": "2022-06-25T23:05:52.007426Z",
     "iopub.status.idle": "2022-06-25T23:09:26.776124Z",
     "shell.execute_reply": "2022-06-25T23:09:26.775503Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3790"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_random_input_encoder() -> Dict[str, torch.Tensor]:\n",
    "    max_seq = 128\n",
    "    seq_len = random.randint(a=1, b=max_seq)\n",
    "    batch = max_seq // seq_len\n",
    "    random_input_ids = torch.randint(\n",
    "        low=0, high=tokenizer.vocab_size, size=(batch, seq_len), dtype=torch.int32, device=\"cuda\"\n",
    "    )\n",
    "    inputs = {\"input_ids\": random_input_ids}\n",
    "    return inputs\n",
    "\n",
    "\n",
    "keep_fp32_encoder = get_keep_fp32_nodes(onnx_model_path=encoder_model_path, get_input=get_random_input_encoder)\n",
    "assert len(keep_fp32_encoder) > 0\n",
    "enc_model_onnx = convert_fp16(onnx_model=encoder_model_path, nodes_to_exclude=keep_fp32_encoder)\n",
    "save_onnx(proto=enc_model_onnx, model_path=encoder_fp16_model_path, clean=False)\n",
    "\n",
    "del enc_model_onnx\n",
    "torch.cuda.empty_cache()\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:09:26.781118Z",
     "iopub.status.busy": "2022-06-25T23:09:26.780819Z",
     "iopub.status.idle": "2022-06-25T23:09:26.786044Z",
     "shell.execute_reply": "2022-06-25T23:09:26.785319Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 first nodes to keep in FP32 (total 1255):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Pow_29',\n",
       " 'Log_89',\n",
       " 'Div_91',\n",
       " 'Mul_93',\n",
       " 'Softmax_109',\n",
       " 'MatMul_110',\n",
       " 'Transpose_111',\n",
       " 'Reshape_115',\n",
       " 'Pow_120',\n",
       " 'Mul_128',\n",
       " 'Pow_135',\n",
       " 'Mul_143',\n",
       " 'Softmax_168',\n",
       " 'Pow_179',\n",
       " 'Pow_194',\n",
       " 'Softmax_227',\n",
       " 'Pow_238',\n",
       " 'Mul_246',\n",
       " 'Pow_253',\n",
       " 'Softmax_286']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(f\"20 first nodes to keep in FP32 (total {len(keep_fp32_encoder)}):\")\n",
    "keep_fp32_encoder[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Compare the output of the `ONNX` `FP16` model with `Pytorch` one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:09:26.792026Z",
     "iopub.status.busy": "2022-06-25T23:09:26.791846Z",
     "iopub.status.idle": "2022-06-25T23:09:28.357582Z",
     "shell.execute_reply": "2022-06-25T23:09:28.356930Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "enc_fp16_onnx = create_model_for_provider(encoder_fp16_model_path, \"CUDAExecutionProvider\", log_severity=3)\n",
    "enc_fp16_onnx_binding: IOBinding = enc_fp16_onnx.io_binding()\n",
    "enc_onnx_out = inference_onnx_binding(\n",
    "    model_onnx=enc_fp16_onnx,\n",
    "    binding=enc_fp16_onnx_binding,\n",
    "    inputs={\"input_ids\": input_ids},\n",
    "    device=input_ids.device.type,\n",
    ")[\"output\"]\n",
    "are_equal(a=enc_onnx_out, b=out_enc.last_hidden_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Conversion of decoder module to mixed precision\n",
    "\n",
    "Decoder module has different kinds of inputs, `input_ids` but also some float tensors.\n",
    "It would a bit more complicated to generate random values for those tensors: in theory it can be of any value in the FP32 range, but because of how models are initialized and trained, most of them are close to 0.\n",
    "\n",
    "To avoid too  much guessing, we have decided to just take the output of the real model being fed with random `input_ids`.\n",
    "\n",
    "### Conversion of the decoder module without cache support\n",
    "\n",
    "This is the easiest part, it takes as input encoder output and `input_ids`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:09:28.360273Z",
     "iopub.status.busy": "2022-06-25T23:09:28.360078Z",
     "iopub.status.idle": "2022-06-25T23:21:19.962781Z",
     "shell.execute_reply": "2022-06-25T23:21:19.962002Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def get_random_input_no_cache() -> Dict[str, torch.Tensor]:\n",
    "    inputs = get_random_input_encoder()\n",
    "    encoder_hidden_states = inference_onnx_binding(\n",
    "        model_onnx=enc_fp16_onnx,\n",
    "        binding=enc_fp16_onnx_binding,\n",
    "        inputs=inputs,\n",
    "        device=\"cuda\",\n",
    "        clone_tensor=False,\n",
    "    )[\"output\"]\n",
    "    # it will serve as input of a FP32 model\n",
    "    inputs[\"encoder_hidden_states\"] = encoder_hidden_states.type(torch.float32)\n",
    "    return inputs\n",
    "\n",
    "\n",
    "keep_fp32_no_cache = get_keep_fp32_nodes(onnx_model_path=dec_no_cache_model_path, get_input=get_random_input_no_cache)\n",
    "\n",
    "onnx_model_no_cache_fp16 = convert_fp16(onnx_model=dec_no_cache_model_path, nodes_to_exclude=keep_fp32_no_cache)\n",
    "save_onnx(proto=onnx_model_no_cache_fp16, model_path=dec_no_cache_fp16_model_path, clean=False)\n",
    "del onnx_model_no_cache_fp16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:21:19.967983Z",
     "iopub.status.busy": "2022-06-25T23:21:19.967479Z",
     "iopub.status.idle": "2022-06-25T23:21:19.973357Z",
     "shell.execute_reply": "2022-06-25T23:21:19.972838Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 first nodes to keep in FP32 (total 2063):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Constant_74',\n",
       " 'Pow_78',\n",
       " 'Log_135',\n",
       " 'Div_137',\n",
       " 'Mul_139',\n",
       " 'Softmax_154',\n",
       " 'Pow_165',\n",
       " 'Softmax_212',\n",
       " 'Pow_223',\n",
       " 'Pow_238',\n",
       " 'MatMul_250',\n",
       " 'Reshape_254',\n",
       " 'Transpose_255',\n",
       " 'Softmax_272',\n",
       " 'Pow_283',\n",
       " 'Softmax_317',\n",
       " 'Pow_328',\n",
       " 'Pow_343',\n",
       " 'Softmax_377',\n",
       " 'Pow_388']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(f\"20 first nodes to keep in FP32 (total {len(keep_fp32_no_cache)}):\")\n",
    "keep_fp32_no_cache[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Conversion of the decoder module with cache support\n",
    "\n",
    "This module requires output from `encoder` but also from `decoder` module without cache support (as the cache is supposed not to be empty).\n",
    "\n",
    "It introduces a new challenge: memory footprint. Indeed, we need to load 3 `Onnx` files in RAM: \n",
    "\n",
    "* encoder: to generate module input;\n",
    "* decoder without cache support: to generate module input;\n",
    "* decoder with cache support: to perform the search of nodes to keep in `FP32`.\n",
    "\n",
    "To limit RAM usage, we will use a trick explained in detail below where we share weights between the 2 decoder modules.\n",
    "\n",
    "Below we perform this merge:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:21:19.975670Z",
     "iopub.status.busy": "2022-06-25T23:21:19.975468Z",
     "iopub.status.idle": "2022-06-25T23:22:13.229475Z",
     "shell.execute_reply": "2022-06-25T23:22:13.228253Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6775"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dec_cache_model: onnx.ModelProto = onnx.load_model(f=dec_cache_model_path, load_external_data=False)\n",
    "dec_no_cache_model: onnx.ModelProto = onnx.load_model(f=dec_no_cache_model_path, load_external_data=False)\n",
    "assert len(dec_cache_model.graph.output) == len(dec_no_cache_model.graph.output)\n",
    "original_nb_output_nodes = len(dec_cache_model.graph.output)\n",
    "\n",
    "dec_cache_model_fp32_all_nodes = add_output_nodes(model=dec_cache_model)\n",
    "dec_cache_model_fp32_all_nodes_path = dec_cache_model_path + \"_all_nodes.onnx\"\n",
    "save_onnx(proto=dec_cache_model_fp32_all_nodes, model_path=dec_cache_model_fp32_all_nodes_path, clean=False)\n",
    "# reload after shape inference\n",
    "dec_cache_model_fp32_all_nodes = onnx.load_model(f=dec_cache_model_fp32_all_nodes_path, load_external_data=False)\n",
    "\n",
    "ort_np_type_mapping = {\n",
    "    onnx.TensorProto.FLOAT: float,\n",
    "    onnx.TensorProto.INT64: np.int64,\n",
    "    onnx.TensorProto.INT32: np.int32,\n",
    "    onnx.TensorProto.BOOL: bool,\n",
    "}\n",
    "\n",
    "\n",
    "# If node requires that the 2 models merged have the exact same number/type of output nodes\n",
    "# Above we added many output nodes to the model with cache support...\n",
    "# ... we need to add fake output nodes to the other decoder model.\n",
    "no_cache_output_nodes = {item.name: item for item in dec_no_cache_model.graph.output}\n",
    "\n",
    "while dec_no_cache_model.graph.output:\n",
    "    dec_no_cache_model.graph.output.pop()\n",
    "\n",
    "nb_outputs_to_create = len(dec_cache_model_fp32_all_nodes.graph.output)\n",
    "nodes_to_be_added = list()\n",
    "for i in range(nb_outputs_to_create):\n",
    "    node_name = dec_cache_model_fp32_all_nodes.graph.output[i].name\n",
    "    if node_name in no_cache_output_nodes:\n",
    "        node_to_insert = no_cache_output_nodes[node_name]\n",
    "        nodes_to_be_added.append(node_to_insert)\n",
    "    else:\n",
    "        fake_node_name = f\"output_{node_name}\"\n",
    "        fake_node_ort_type = dec_cache_model_fp32_all_nodes.graph.output[i].type.tensor_type.elem_type\n",
    "        fake_node_np_type = ort_np_type_mapping[fake_node_ort_type]\n",
    "        fake_data = np.array([1.0], dtype=fake_node_np_type)\n",
    "        fake_node = onnx.helper.make_node(\n",
    "            op_type=\"Constant\",\n",
    "            inputs=[],\n",
    "            outputs=[fake_node_name],\n",
    "            value=onnx.helper.make_tensor(\n",
    "                name=\"const_tensor\",\n",
    "                data_type=fake_node_ort_type,\n",
    "                dims=fake_data.shape,\n",
    "                vals=fake_data.flatten(),\n",
    "            ),\n",
    "            name=fake_node_name,\n",
    "        )\n",
    "        dec_no_cache_model.graph.node.append(fake_node)\n",
    "        nodes_to_be_added.append(onnx.ValueInfoProto(name=fake_node_name))\n",
    "\n",
    "\n",
    "dec_no_cache_model.graph.output.extend(nodes_to_be_added)\n",
    "\n",
    "dec_no_cache_model_fp32_all_nodes_path = dec_no_cache_model_path + \"_all_nodes.onnx\"\n",
    "save_onnx(proto=dec_no_cache_model, model_path=dec_no_cache_model_fp32_all_nodes_path, clean=False)\n",
    "\n",
    "# now that each model has the same number of output nodes, we can merge them!\n",
    "merge_autoregressive_model_graphs(\n",
    "    model_cache_path=dec_cache_model_fp32_all_nodes_path,\n",
    "    model_no_cache_path=dec_no_cache_model_fp32_all_nodes_path,\n",
    "    output_path=dec_if_model_path,\n",
    ")\n",
    "del dec_cache_model_fp32_all_nodes, dec_no_cache_model, dec_cache_model\n",
    "\n",
    "torch.cuda.empty_cache()\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:22:13.233449Z",
     "iopub.status.busy": "2022-06-25T23:22:13.233130Z",
     "iopub.status.idle": "2022-06-25T23:30:20.275528Z",
     "shell.execute_reply": "2022-06-25T23:30:20.274874Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dec_if_ort_model = create_model_for_provider(dec_if_model_path, \"CUDAExecutionProvider\", log_severity=3)\n",
    "\n",
    "# use info from tokenizer size and max shape provided through the command line\n",
    "\n",
    "\n",
    "def get_random_input_cache() -> Dict[str, torch.Tensor]:\n",
    "    inputs = get_random_input_no_cache()\n",
    "    inputs[\"enable_cache\"] = torch.tensor([False], device=\"cuda\")\n",
    "    dec_past_states = inference_onnx_binding(\n",
    "        model_onnx=dec_if_ort_model,\n",
    "        inputs=inputs,\n",
    "        device=\"cuda\",\n",
    "        clone_tensor=False,\n",
    "    )\n",
    "    for k, v in dec_past_states.items():\n",
    "        if \"present\" not in k:\n",
    "            continue\n",
    "        new_k = k.replace(\"present\", \"past_key_values\")\n",
    "        inputs[new_k] = v\n",
    "    batch, _ = inputs[\"input_ids\"].shape\n",
    "    complement = torch.randint(low=0, high=tokenizer.vocab_size, size=(batch, 1), dtype=torch.int32, device=\"cuda\")\n",
    "    inputs[\"input_ids\"] = torch.concat(tensors=[inputs[\"input_ids\"], complement], dim=1)\n",
    "    inputs[\"enable_cache\"] = torch.tensor([True], device=\"cuda\")\n",
    "    return inputs\n",
    "\n",
    "\n",
    "keep_fp32_cache = search_fp32_nodes(\n",
    "    original_model=dec_if_model_path,\n",
    "    modified_model_session=dec_if_ort_model,\n",
    "    get_input=get_random_input_cache,\n",
    "    early_stop=100,\n",
    ")\n",
    "\n",
    "\n",
    "# the output node names are those from the decoder module without cache support\n",
    "# basically it's the fake node names we added above, we need to remove the output_ prefix to their names\n",
    "keep_fp32_cache = [item.replace(\"output_\", \"\") for item in keep_fp32_cache]\n",
    "\n",
    "\n",
    "del dec_if_ort_model, enc_fp16_onnx, enc_fp16_onnx_binding\n",
    "torch.cuda.empty_cache()\n",
    "gc.collect()\n",
    "\n",
    "onnx_model_cache_fp16 = convert_fp16(onnx_model=dec_cache_model_path, nodes_to_exclude=keep_fp32_cache)\n",
    "save_onnx(proto=onnx_model_cache_fp16, model_path=dec_cache_fp16_model_path, clean=False)\n",
    "\n",
    "del onnx_model_cache_fp16\n",
    "gc.collect()\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:30:20.279738Z",
     "iopub.status.busy": "2022-06-25T23:30:20.279249Z",
     "iopub.status.idle": "2022-06-25T23:30:20.284321Z",
     "shell.execute_reply": "2022-06-25T23:30:20.283807Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 first nodes to keep in FP32 (total 1306):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['onnx::Mul_515',\n",
       " 'onnx::ReduceMean_519',\n",
       " 'onnx::Div_612',\n",
       " 'onnx::Mul_614',\n",
       " 'onnx::Cast_616',\n",
       " 'onnx::MatMul_645',\n",
       " 'onnx::ReduceMean_666',\n",
       " 'onnx::MatMul_732',\n",
       " 'onnx::ReduceMean_750',\n",
       " 'onnx::ReduceMean_767',\n",
       " 'onnx::Add_768',\n",
       " 'onnx::Sqrt_770',\n",
       " 'onnx::MatMul_833',\n",
       " 'onnx::ReduceMean_851',\n",
       " 'onnx::Add_852',\n",
       " 'onnx::Sqrt_854',\n",
       " 'onnx::MatMul_883',\n",
       " 'onnx::ReduceMean_901',\n",
       " 'onnx::Add_902',\n",
       " 'onnx::Sqrt_904']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(f\"20 first nodes to keep in FP32 (total {len(keep_fp32_cache)}):\")\n",
    "keep_fp32_cache[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Merge `ONNX` computation graph to deduplicate weights\n",
    "\n",
    "Finally, we will merge the 2 decoders together.\n",
    "The idea is simple:\n",
    "\n",
    "* we prefix the node / edge names of one of them to avoid naming collision\n",
    "* we deduplicate the weights (the same weight matrix will have different names in the 2 models)\n",
    "* we join the 2 computation graphs through an `If` node\n",
    "* we generate the `ONNX` file\n",
    "\n",
    "The new model will take a new input, `enable_cache`. When it contains a `True` value, computation graph with cache support is used.\n",
    "\n",
    "> code below is written to be easy to read, but could be made much faster to run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:30:20.287722Z",
     "iopub.status.busy": "2022-06-25T23:30:20.287499Z",
     "iopub.status.idle": "2022-06-25T23:30:55.187877Z",
     "shell.execute_reply": "2022-06-25T23:30:55.187290Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merge_autoregressive_model_graphs(\n",
    "    model_cache_path=dec_cache_fp16_model_path,\n",
    "    model_no_cache_path=dec_no_cache_fp16_model_path,\n",
    "    output_path=dec_if_fp16_model_path,\n",
    ")\n",
    "\n",
    "torch.cuda.empty_cache()\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Check `ONNX` decoder output\n",
    "\n",
    "Compare `ONNX` output with and without cache, plus compare with `Pytorch` output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:30:55.192078Z",
     "iopub.status.busy": "2022-06-25T23:30:55.191597Z",
     "iopub.status.idle": "2022-06-25T23:30:56.622668Z",
     "shell.execute_reply": "2022-06-25T23:30:56.621901Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pytorch_model = pytorch_model.cuda()\n",
    "model_decoder = model_decoder.cuda()\n",
    "input_ids = input_ids.cuda()\n",
    "pytorch_model = pytorch_model.eval()\n",
    "model_decoder = model_decoder.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Zero copy output\n",
    "\n",
    "Below, we check that the new model output is similar to the ones from `Pytorch`.\n",
    "\n",
    "We use our new implementation of inference call.  \n",
    "The idea is the following:\n",
    "\n",
    "* we ask `ONNX Runtime` to output a pointer to the `CUDA` array containing the result of the inference;\n",
    "* we use `Cupy` API to wrap the array and provide information regarding tensor shape and type. `Cupy` doesn't own the data;\n",
    "* we use `Dlpack` support to convert the `Cupy` tensor to `Pytorch`, another zero copy process.\n",
    "\n",
    "This pipeline is unsafe, as the content of the tensor may change or disappear silently: only `ONNX Runtime` has the control of the array containing the data. It will happen at the next inference call. Because we know that during the text generation we discard each output before recalling `ONNX Runtime`, it works well in our case.\n",
    "\n",
    "A second benefit of this approach is that we do not have anymore to guess the output shape.  \n",
    "Before using this approach, to avoid the output to be stored on host memory (RAM) which made inference slower, we had to provide `ONNX Runtime` with a pointer to `Pytorch` tensor with the right size. As the size change with the sequence length (so it changes for each generated token), we had to store the logic to guess the size somewhere in the code. The new approach frees us from this burden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:30:56.627680Z",
     "iopub.status.busy": "2022-06-25T23:30:56.627098Z",
     "iopub.status.idle": "2022-06-25T23:30:58.826841Z",
     "shell.execute_reply": "2022-06-25T23:30:58.826297Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "with torch.inference_mode():\n",
    "    out_enc_pytorch: BaseModelOutputWithPastAndCrossAttentions = pytorch_model.encoder(input_ids=input_ids)\n",
    "    previous_step_pytorch: BaseModelOutputWithPastAndCrossAttentions = model_decoder(\n",
    "        input_ids=input_ids[:, :-1], encoder_hidden_states=out_enc_pytorch.last_hidden_state\n",
    "    )\n",
    "    out_dec_pytorch: BaseModelOutputWithPastAndCrossAttentions = model_decoder(\n",
    "        input_ids=input_ids, encoder_hidden_states=out_enc_pytorch.last_hidden_state\n",
    "    )\n",
    "pytorch_model = pytorch_model.cpu()\n",
    "model_decoder = model_decoder.cpu()\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:30:58.830668Z",
     "iopub.status.busy": "2022-06-25T23:30:58.830474Z",
     "iopub.status.idle": "2022-06-25T23:31:28.358065Z",
     "shell.execute_reply": "2022-06-25T23:31:28.357374Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "enc_fp16_onnx = create_model_for_provider(encoder_fp16_model_path, \"CUDAExecutionProvider\", log_severity=3)\n",
    "enc_fp16_onnx_binding: IOBinding = enc_fp16_onnx.io_binding()\n",
    "dec_onnx = create_model_for_provider(dec_if_fp16_model_path, \"CUDAExecutionProvider\", log_severity=3)\n",
    "dec_onnx_binding: IOBinding = dec_onnx.io_binding()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:31:28.362691Z",
     "iopub.status.busy": "2022-06-25T23:31:28.362409Z",
     "iopub.status.idle": "2022-06-25T23:31:29.072363Z",
     "shell.execute_reply": "2022-06-25T23:31:29.071563Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def decoder_pytorch_inference(decoder_input_ids: torch.Tensor, encoder_hidden_states: torch.Tensor, **_):\n",
    "    with torch.inference_mode():\n",
    "        return model_decoder(input_ids=decoder_input_ids, encoder_hidden_states=encoder_hidden_states)\n",
    "\n",
    "\n",
    "def decoder_onnx_inference(\n",
    "    decoder_input_ids: torch.Tensor,\n",
    "    encoder_hidden_states: torch.Tensor,\n",
    "    enable_cache: torch.Tensor,\n",
    "    past_key_values: Optional[torch.Tensor],\n",
    "):\n",
    "    inputs_onnx_dict = {\n",
    "        \"input_ids\": decoder_input_ids,\n",
    "        \"encoder_hidden_states\": encoder_hidden_states,\n",
    "        \"enable_cache\": enable_cache,\n",
    "    }\n",
    "\n",
    "    if past_key_values is not None:\n",
    "        for index, (k_dec, v_dec, k_enc, v_enc) in enumerate(past_key_values):\n",
    "            inputs_onnx_dict[f\"past_key_values.{index}.decoder.key\"] = k_dec\n",
    "            inputs_onnx_dict[f\"past_key_values.{index}.decoder.value\"] = v_dec\n",
    "            inputs_onnx_dict[f\"past_key_values.{index}.encoder.key\"] = k_enc\n",
    "            inputs_onnx_dict[f\"past_key_values.{index}.encoder.value\"] = v_enc\n",
    "\n",
    "    result_dict = inference_onnx_binding(\n",
    "        model_onnx=dec_onnx,\n",
    "        inputs=inputs_onnx_dict,\n",
    "        binding=dec_onnx_binding,  # recycle the binding\n",
    "        device=decoder_input_ids.device.type,\n",
    "        clone_tensor=False,  # no memory copy -> best perf and lowest memory footprint!\n",
    "    )\n",
    "    past_states = list()\n",
    "    for index in range(pytorch_model.config.num_layers):\n",
    "        kv = (\n",
    "            result_dict[f\"present.{index}.decoder.key\"],\n",
    "            result_dict[f\"present.{index}.decoder.value\"],\n",
    "            result_dict[f\"present.{index}.encoder.key\"],\n",
    "            result_dict[f\"present.{index}.encoder.value\"],\n",
    "        )\n",
    "        past_states.append(kv)\n",
    "    return BaseModelOutputWithPastAndCrossAttentions(\n",
    "        last_hidden_state=result_dict[\"logits\"],\n",
    "        past_key_values=past_states,\n",
    "    )\n",
    "\n",
    "\n",
    "out_dec_onnx_no_cache = decoder_onnx_inference(\n",
    "    decoder_input_ids=input_ids,\n",
    "    encoder_hidden_states=out_enc_pytorch.last_hidden_state.half(),\n",
    "    enable_cache=torch.tensor([False], device=\"cuda\", dtype=torch.bool),\n",
    "    past_key_values=None,\n",
    ")\n",
    "are_equal(a=out_dec_onnx_no_cache.last_hidden_state[:, -1:, :], b=out_dec_pytorch.last_hidden_state[:, -1:, :])\n",
    "\n",
    "# check that past states are identical between ONNX and Pytorch\n",
    "assert len(out_dec_onnx_no_cache.past_key_values) == len(out_dec_pytorch.past_key_values)\n",
    "for (o_dec_k, o_dev_v, o_enc_k, o_enc_v), (p_dec_k, p_dev_v, p_enc_k, p_enc_v) in zip(\n",
    "    out_dec_onnx_no_cache.past_key_values, out_dec_pytorch.past_key_values\n",
    "):\n",
    "    are_equal(a=o_dec_k, b=p_dec_k)\n",
    "    are_equal(a=o_dev_v, b=p_dev_v)\n",
    "    are_equal(a=o_enc_k, b=p_enc_k)\n",
    "    are_equal(a=o_enc_v, b=p_enc_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:31:29.077196Z",
     "iopub.status.busy": "2022-06-25T23:31:29.076902Z",
     "iopub.status.idle": "2022-06-25T23:31:29.758912Z",
     "shell.execute_reply": "2022-06-25T23:31:29.758374Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# convert ONNX inputs to FP16\n",
    "previous_step_pytorch.past_key_values = tuple(\n",
    "    [tuple([past.half() for past in layer_state]) for layer_state in previous_step_pytorch.past_key_values]\n",
    ")\n",
    "out_enc_pytorch.last_hidden_state = out_enc_pytorch.last_hidden_state.half()\n",
    "\n",
    "out_dec_onnx_cache = decoder_onnx_inference(\n",
    "    decoder_input_ids=input_ids[:, -1:],\n",
    "    encoder_hidden_states=out_enc_pytorch.last_hidden_state,\n",
    "    enable_cache=torch.tensor([True], device=\"cuda\", dtype=torch.bool),\n",
    "    past_key_values=previous_step_pytorch.past_key_values,\n",
    ")\n",
    "\n",
    "are_equal(a=out_dec_onnx_cache.last_hidden_state[:, -1:, :], b=out_dec_pytorch.last_hidden_state[:, -1:, :])\n",
    "\n",
    "# check that past states are identical between ONNX and Pytorch\n",
    "assert len(out_dec_onnx_cache.past_key_values) == len(out_dec_pytorch.past_key_values)\n",
    "for (o_dec_k, o_dev_v, o_enc_k, o_enc_v), (p_dec_k, p_dev_v, p_enc_k, p_enc_v) in zip(\n",
    "    out_dec_onnx_cache.past_key_values, out_dec_pytorch.past_key_values\n",
    "):\n",
    "    are_equal(a=o_dec_k, b=p_dec_k)\n",
    "    are_equal(a=o_dev_v, b=p_dev_v)\n",
    "    are_equal(a=o_enc_k, b=p_enc_k)\n",
    "    are_equal(a=o_enc_v, b=p_enc_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Benchmarks!\n",
    "\n",
    "Finally, we will compare the performances of 4 setup in end-to-end scenarii:\n",
    "\n",
    "* `Pytorch`\n",
    "* `Pytorch` + cache\n",
    "* `ONNX`\n",
    "* `ONNX` + cache\n",
    "\n",
    "For the comparison, we first do a sanity check by just generating a short sequence (we already have checked that output tensors are OK).\n",
    "\n",
    "Then we force each model to generate:\n",
    "\n",
    "* 256 tokens + batch size 1 (similar to `TensorRT` demo)\n",
    "* 1000 tokens + batch size 4\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:31:29.763346Z",
     "iopub.status.busy": "2022-06-25T23:31:29.763084Z",
     "iopub.status.idle": "2022-06-25T23:31:31.363367Z",
     "shell.execute_reply": "2022-06-25T23:31:31.362735Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text generated by ONNX:\n",
      "L'apprentissage par transfert, où un modèle est d'abord préentraîné sur une tâche riche en données avant sa mise au point à l'égard de tâches ultérieures, est devenue une technique puissante dans le domaine du traitement de langue naturelle (PLN).\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def encoder_onnx_inference(input_ids: torch.Tensor, **_) -> BaseModelOutputWithPastAndCrossAttentions:\n",
    "    last_hidden_state = inference_onnx_binding(\n",
    "        model_onnx=enc_fp16_onnx,  # noqa: F821\n",
    "        inputs={\"input_ids\": input_ids},\n",
    "        device=input_ids.device.type,\n",
    "        binding=enc_fp16_onnx_binding,\n",
    "    )[\"output\"]\n",
    "    return BaseModelOutputWithPastAndCrossAttentions(last_hidden_state=last_hidden_state.type(torch.float16))\n",
    "\n",
    "\n",
    "def encoder_pytorch_inference(input_ids, **_) -> BaseModelOutputWithPastAndCrossAttentions:\n",
    "    with torch.inference_mode():\n",
    "        res = pytorch_model.encoder(input_ids=input_ids).type(torch.float16)\n",
    "        return res\n",
    "\n",
    "\n",
    "# https://github.com/NVIDIA/TensorRT/blob/main/demo/HuggingFace/T5/export.py\n",
    "class ExtT5(torch.nn.Module, GenerationMixin):\n",
    "    def __init__(self, config: PretrainedConfig, device: torch.device, encoder_func: Callable, decoder_func: Callable):\n",
    "        super(ExtT5, self).__init__()\n",
    "        self.main_input_name = \"input_ids\"  # https://github.com/huggingface/transformers/pull/14803\n",
    "        self.config: PretrainedConfig = config\n",
    "        self.device: torch.device = device\n",
    "\n",
    "        self.encoder_func = encoder_func\n",
    "        self.decoder_func = decoder_func\n",
    "        self.use_cache = True\n",
    "        self.timings = list()\n",
    "\n",
    "    def get_encoder(self):\n",
    "        return self.encoder_func\n",
    "\n",
    "    def get_decoder(self):\n",
    "        return self.decoder_func\n",
    "\n",
    "    def set_cache(self, enable: bool) -> None:\n",
    "        self.use_cache = enable\n",
    "\n",
    "    # from transformers library (modeling_t5.py)\n",
    "    def _reorder_cache(self, past, beam_idx):\n",
    "        reordered_decoder_past = ()\n",
    "        for layer_past_states in past:\n",
    "            # get the correct batch idx from layer past batch dim\n",
    "            # batch dim of `past` is at 2nd position\n",
    "            reordered_layer_past_states = ()\n",
    "            for layer_past_state in layer_past_states:\n",
    "                # need to set correct `past` for each of the four key / value states\n",
    "                reordered_layer_past_states = reordered_layer_past_states + (\n",
    "                    layer_past_state.index_select(0, beam_idx),\n",
    "                )\n",
    "\n",
    "            assert reordered_layer_past_states[0].shape == layer_past_states[0].shape\n",
    "            assert len(reordered_layer_past_states) == len(layer_past_states)\n",
    "\n",
    "            reordered_decoder_past = reordered_decoder_past + (reordered_layer_past_states,)\n",
    "        return reordered_decoder_past\n",
    "\n",
    "    def prepare_inputs_for_generation(self, input_ids, past=None, use_cache=None, **kwargs) -> Dict[str, torch.Tensor]:\n",
    "        params = {\n",
    "            \"encoder_hidden_states\": kwargs[\"encoder_outputs\"][\"last_hidden_state\"],\n",
    "        }\n",
    "        if past is None:  # this is the 1st inferred token\n",
    "            self.timings = list()\n",
    "        if not self.use_cache:\n",
    "            past = None\n",
    "        if past is None:\n",
    "            params[self.main_input_name] = input_ids\n",
    "            params[\"enable_cache\"] = torch.tensor([False], device=\"cuda\", dtype=torch.bool)\n",
    "        else:\n",
    "            params[self.main_input_name] = input_ids[:, -1:]\n",
    "            params[\"enable_cache\"] = torch.tensor([True], device=\"cuda\", dtype=torch.bool)\n",
    "            params[\"past_key_values\"] = past\n",
    "\n",
    "        return params\n",
    "\n",
    "    def forward(\n",
    "        self,\n",
    "        input_ids: torch.Tensor,\n",
    "        encoder_hidden_states: torch.Tensor,\n",
    "        enable_cache: torch.Tensor,\n",
    "        past_key_values: Optional[torch.Tensor] = None,\n",
    "        **_,\n",
    "    ):\n",
    "        start_timer = time.monotonic()\n",
    "        dec_output = self.get_decoder()(\n",
    "            decoder_input_ids=input_ids,\n",
    "            encoder_hidden_states=encoder_hidden_states,\n",
    "            enable_cache=enable_cache,\n",
    "            past_key_values=past_key_values,\n",
    "        )\n",
    "        self.timings.append(time.monotonic() - start_timer)\n",
    "        return Seq2SeqLMOutput(logits=dec_output.last_hidden_state, past_key_values=dec_output.past_key_values)\n",
    "\n",
    "\n",
    "model_gen = (\n",
    "    ExtT5(\n",
    "        config=pytorch_model.config,\n",
    "        device=pytorch_model.device,\n",
    "        encoder_func=encoder_onnx_inference,  # encoder_pytorch_inference\n",
    "        decoder_func=decoder_onnx_inference,  # decoder_pytorch_inference\n",
    "    )\n",
    "    .cuda()\n",
    "    .eval()\n",
    ")\n",
    "\n",
    "torch.cuda.synchronize()\n",
    "with torch.inference_mode():\n",
    "    print(\"text generated by ONNX:\")\n",
    "    onnx_tokens = model_gen.generate(\n",
    "        inputs=input_ids,\n",
    "        min_length=10,\n",
    "        max_length=128,\n",
    "        num_beams=2,\n",
    "        no_repeat_ngram_size=2,\n",
    "    )[0]\n",
    "    print(tokenizer.decode(onnx_tokens, skip_special_tokens=True, clean_up_tokenization_spaces=True))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:31:31.367833Z",
     "iopub.status.busy": "2022-06-25T23:31:31.367648Z",
     "iopub.status.idle": "2022-06-25T23:31:42.820566Z",
     "shell.execute_reply": "2022-06-25T23:31:42.820023Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seq len: 128 / # beam (batch size): 1\n",
      "ONNX: 2.9, including inference: 2.8 (97.2%)\n",
      "ONNX + cache: 1.4, including inference: 1.3 (94.4%)\n",
      "seq len: 128 / # beam (batch size): 2\n",
      "ONNX: 4.7, including inference: 4.4 (94.5%)\n",
      "ONNX + cache: 1.7, including inference: 1.4 (85.1%)\n"
     ]
    }
   ],
   "source": [
    "def print_timings(name: str, total: float, inference: float):\n",
    "    percent_inference = 100 * inference / total\n",
    "    print(f\"{name}: {total:.1f}, including inference: {inference:.1f} ({percent_inference:.1f}%)\")\n",
    "\n",
    "\n",
    "all_timings: Dict[str, Dict[str, List[float]]] = defaultdict(dict)\n",
    "for seq_len, num_beam in benchmark_tasks:\n",
    "    timings_key = f\"{seq_len} / {num_beam}\"\n",
    "\n",
    "    print(f\"seq len: {seq_len} / # beam (batch size): {num_beam}\")\n",
    "    task = \"ONNX\"\n",
    "    with nvtx.annotate(\n",
    "        task, color=\"red\"\n",
    "    ):  # nvtx is for Nvidia nsight profiler, you can remove the line or install the library\n",
    "        model_gen.set_cache(enable=False)\n",
    "        # warmup\n",
    "        model_gen.generate(inputs=input_ids, max_length=10, num_beams=num_beam, min_length=10)\n",
    "        start = time.monotonic()\n",
    "        model_gen.generate(inputs=input_ids, max_length=seq_len, num_beams=num_beam, min_length=seq_len)\n",
    "        total_time = time.monotonic() - start\n",
    "        print_timings(name=task, total=total_time, inference=sum(model_gen.timings))\n",
    "        all_timings[timings_key][f\"{task}\"] = model_gen.timings\n",
    "\n",
    "    task = \"ONNX + cache\"\n",
    "    with nvtx.annotate(task, color=\"red\"):\n",
    "        model_gen.set_cache(enable=True)\n",
    "        # warmup\n",
    "        model_gen.generate(inputs=input_ids, max_length=10, num_beams=num_beam, min_length=10)\n",
    "        start = time.monotonic()\n",
    "        model_gen.generate(inputs=input_ids, max_length=seq_len, num_beams=num_beam, min_length=seq_len)\n",
    "        total_time = time.monotonic() - start\n",
    "        print_timings(name=task, total=total_time, inference=sum(model_gen.timings))\n",
    "        all_timings[timings_key][f\"{task}\"] = model_gen.timings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:31:42.823167Z",
     "iopub.status.busy": "2022-06-25T23:31:42.822970Z",
     "iopub.status.idle": "2022-06-25T23:31:44.648023Z",
     "shell.execute_reply": "2022-06-25T23:31:44.647389Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "del enc_fp16_onnx, enc_fp16_onnx_binding, dec_onnx, dec_onnx_binding\n",
    "\n",
    "pytorch_model = pytorch_model.cuda()\n",
    "model_decoder = model_decoder.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:31:44.651402Z",
     "iopub.status.busy": "2022-06-25T23:31:44.651208Z",
     "iopub.status.idle": "2022-06-25T23:31:47.566756Z",
     "shell.execute_reply": "2022-06-25T23:31:47.566232Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text generated by Pytorch:\n",
      "L'apprentissage par transfert, où un modèle est d'abord préentraîné sur une tâche riche en données avant sa mise au point à l'égard de tâches ultérieures, est devenue une technique puissante dans le domaine du traitement de la langue naturelle (PLN).\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "torch.cuda.synchronize()\n",
    "with torch.inference_mode():\n",
    "    print(\"text generated by Pytorch:\")\n",
    "    pytorch_tokens = pytorch_model.generate(\n",
    "        inputs=input_ids,\n",
    "        min_length=10,\n",
    "        max_length=128,\n",
    "        num_beams=2,\n",
    "        no_repeat_ngram_size=2,\n",
    "    )[0]\n",
    "    print(tokenizer.decode(pytorch_tokens, skip_special_tokens=True, clean_up_tokenization_spaces=True))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:31:47.571595Z",
     "iopub.status.busy": "2022-06-25T23:31:47.571410Z",
     "iopub.status.idle": "2022-06-25T23:32:07.257300Z",
     "shell.execute_reply": "2022-06-25T23:32:07.256672Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pytorch: 4.1, including inference: 4.0 (98.6%)\n",
      "Pytorch + cache: 3.5, including inference: 3.4 (98.3%)\n",
      "Pytorch: 4.3, including inference: 4.2 (97.8%)\n",
      "Pytorch + cache: 3.7, including inference: 3.6 (95.7%)\n"
     ]
    }
   ],
   "source": [
    "for seq_len, num_beam in benchmark_tasks:\n",
    "    timings_key = f\"{seq_len} / {num_beam}\"\n",
    "\n",
    "    # monckey patching of forward function to add a timer per generated token\n",
    "    old_fw = pytorch_model.forward\n",
    "    timing_pytorch = list()\n",
    "\n",
    "    def new_fw(self, *args, **kwargs):\n",
    "        timer_start = time.monotonic()\n",
    "        res = old_fw(self, *args, **kwargs)\n",
    "        torch.cuda.synchronize()  # makes timings correct without having significant impact on e2e latency\n",
    "        total_time = time.monotonic() - timer_start\n",
    "        timing_pytorch.append(total_time)\n",
    "        return res\n",
    "\n",
    "    task = \"Pytorch\"\n",
    "    with nvtx.annotate(task, color=\"orange\"):\n",
    "        pytorch_model.config.use_cache = False\n",
    "        with torch.inference_mode():\n",
    "            with torch.cuda.amp.autocast():\n",
    "                # warmup\n",
    "                pytorch_model.generate(inputs=input_ids, max_length=10, num_beams=num_beam, min_length=10)\n",
    "                pytorch_model.forward = new_fw.__get__(pytorch_model)\n",
    "                start = time.monotonic()\n",
    "                pytorch_model.generate(inputs=input_ids, max_length=seq_len, num_beams=num_beam, min_length=seq_len)\n",
    "                total_time = time.monotonic() - start\n",
    "                pytorch_model.forward = old_fw\n",
    "                inference_time = np.sum(timing_pytorch)\n",
    "                print_timings(name=\"Pytorch\", total=total_time, inference=inference_time)\n",
    "        timing_pytorch_no_cache = copy(timing_pytorch)\n",
    "        all_timings[timings_key][f\"{task}\"] = copy(timing_pytorch)\n",
    "        timing_pytorch.clear()\n",
    "    torch.cuda.empty_cache()\n",
    "\n",
    "    task = \"Pytorch + cache\"\n",
    "    with nvtx.annotate(\"Pytorch + cache\", color=\"green\"):\n",
    "        pytorch_model.config.use_cache = True\n",
    "        with torch.inference_mode():\n",
    "            with torch.cuda.amp.autocast():\n",
    "                # warmup\n",
    "                pytorch_model.generate(inputs=input_ids, max_length=10, num_beams=num_beam, min_length=10)\n",
    "                pytorch_model.forward = new_fw.__get__(pytorch_model)\n",
    "                start = time.monotonic()\n",
    "                pytorch_model.generate(inputs=input_ids, max_length=seq_len, num_beams=num_beam, min_length=seq_len)\n",
    "                total_time = time.monotonic() - start\n",
    "                pytorch_model.forward = old_fw\n",
    "                print_timings(name=\"Pytorch + cache\", total=total_time, inference=sum(timing_pytorch))\n",
    "        all_timings[timings_key][f\"{task}\"] = copy(timing_pytorch)\n",
    "        timing_pytorch.clear()\n",
    "    torch.cuda.empty_cache()\n",
    "\n",
    "pytorch_model = pytorch_model.cpu()\n",
    "model_decoder = model_decoder.cpu()\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Benchmark analysis\n",
    "\n",
    "Below, we plot for each setup (short and long sequence):\n",
    "\n",
    "* the time spent on each token generation\n",
    "* the full time to generate the sequence (for each length)\n",
    "\n",
    "We can see that for short sequence and batch size of 1, cache or not, latency appears to be stable.\n",
    "However, for longer sequences, we can see that the no cache approach (being `Pytorch` or `ONNX` based) doesn't scale well, and at some point, `ONNX` is even slower than `Hugging Face` code with cache support.\n",
    "\n",
    "On the other side, `ONNX` timings are mostly stable whatever the sequence length which is quite remarkable.\n",
    "It's because we are working one token at a time and converted a quadratic complexity in the attention layer into a linear one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:32:07.262946Z",
     "iopub.status.busy": "2022-06-25T23:32:07.262462Z",
     "iopub.status.idle": "2022-06-25T23:32:08.269837Z",
     "shell.execute_reply": "2022-06-25T23:32:08.269184Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABCQAAAJICAYAAAC5cDUSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAD7T0lEQVR4nOzdd5gURf7H8fdm0sISlhxVKAFFEDGCqKceior5Z0CCWc9w6hnu9ExnTmfOARTMOWLGdEZUUNQSVJLksLCkzb8/umdphp60OztpP6/n4WF2uqe7qid017e/VZVVU1ODiIiIiIiIiEgiZSe7ACIiIiIiIiLS+CggISIiIiIiIiIJp4CEiIiIiIiIiCScAhIiIiIiIiIiknAKSIiIiIiIiIhIwikgISIiIiIiIiIJp4CESIIZY3oaY2qMMbnu31ONMScnsTx7GWMWJGv/0TLGTDDGXJPscvhx389tkl2OWAV/FjNJfT8vxpg5xph941mmhuCtZ6zf5WR8940xVxpjJtXxtfV9T58yxhzqPj7eGPNODK+9xhiz3BizuK77TyRjTAdjzM/GmIJkl0VERCScjLsIFakrY8xQ4CagP1AF/Az83Vr7dVILJpKBjDFzgJOtte95nqsB1gM17lNPW2t9g3XGmH7A48DW7lPTgHOstT81WKHrwBgzAZhqrZ1gjBmHU+ehYdafCkyy1j7cAGWpAXpba2fHe9upzhgzANgBOA7AWjsZmBzla7sDFwA9rLVLG6yQcWStXWKM+RA4FbgrltcaY/bC+Qx2bYiyiYiIeClDQgQwxrQEXse5cGsDdAGuAsqSWS6JP2NMljGm0fz2pWH2ww7W2hbuv3CZQwuBI3G+r+2AV4GnE1FASUunAZOttTUR19xSd2BFXYIRSf69mYxT74RKw98cERFJIp00RBx9AKy1T7l/bwBq03ndO5unAF8B44GVwGj3df8BCoALrbUT3fVHAtfg3L1dDTxirb0y1kIZY3YG7nX3swHngvp8d9muwG1AP2AucK61dqq7rBcwAdgR+AKwQJG1dnQU++yME5jZE1gL/Ndae6e77Ep3fxuBw4B5wFhr7TchtnUHcDjQCpiFk3HySTTbMsYMAh4BegNvsumuud9+cnCyW8YCpcCtbh3yrLWV7l3nz4C93GOyvXvRfBcwGFgG/Nta+6y7vQLgWuBonPf2JeA8a+0Gd/mFwPlumS7zlGMITmCrs7W2yn3ucOAKa+0OPuU+ELgF6AascY/1Le6yg3A+Qz2Bn4DTrbUz3GWX4Hwe2wPzgUuttS+5y8ax6bM6BrjPGHOtu60jgSLgB2A/T1GON8b8B2jmluHaEMe5lXvMDsDJZHgIuM5aWx24+4/zeTsJKAHOtNa+5bOdJ3Aaea8ZY6qAq621N/ntMxRrbYm7D4wxWThZTcHdZtoZY94FdgW+BcZYa+eGqNsJOMeoBc73yrusALgR5/MA8CxwsbW2LHA3GfgvcLFbjn9Zax8L2kZf4H4gzxizFqi01hYFrXMtMAzY1RhzOzDBWnuWMWZbQnxWo2WM+dh9ON3NlDgJWOIuu8Cv7JG+B0Hbnwscbq2dZow53j0m21lrZxpjTgIOttYe6q6eb4x5HP/vfV/gPmAg8CfwT2vtqyHqFPI74uMAnO9D4LXj8GSruMfkDJxMiGKcxvxZwF+A14AC93173lo7LsLv71Ri+72ZAKxz67GnW5fjrLW/ucv7A7e7r60A7rDWXucGOi7C+b4XAe+7x2ClW80vga2MMT38Pvd+vz84x/4tT33BOf+sIPJ34C7gPOBd9/f7n9ba19x95QGLgP2std/5vUEiItI4KSAh4vgVqDLGTMS5y/qFtXZV0Dq7AA8DbXGyJ57GuVDdBhgOvGCMecFauxbn4nIMMBPYDucC7Xtr7csxlusOnIvPJ4wxLdxtYYzpArwBnABMwblofsEYs621dhnwJPA5sL9b7jeAVyLtzL3Afc1d91igK/CeMcZaa992VzsEJ8gwHqcxcDdOg8/P18DVOEGZc4HnjDE9rbUbw23LGJMPvIxzEX43MAp4CueC2M8pOA2OgTjH/jmfdU5w17FAc+BH4HL3ue1x3qMf3ZT/G3CCSQNxGgBPuuv+0xgzAvgHzjH/A6dRDoC19mtjzAqc4x5oiJ+A07XAzyPA0dbaT4wxrYFeUBuMeRQ4GPgGJ/j1qnHeiDLgN5yG62LgKGCSMWYba+0id7u74Hw+OwB5OI2O/sDu7mt2Aao95RgKGJyGx1fGmBettT/7lPcunODSVjjfg3dwGhmPePY7ESdj4VTgEWNMl+C70tbaE4wxwwjqsuH62P0c/g8431o7J8Sxwz1WJThBhGyc98jreGAkTsPsJpxG5hbdJdzuH/cBB7rrXo/z2Q+4FOczPhAnCPUKTiDq3+7yjjjHpQtOoOd5Y8zL1tpV1tpxnv2cTpguG9baS40xe+DpsmGMaQ68S+jPalSstXu6je4dAl023IZkyLIT5nvgs4uPcBrg03B+D3/HaVzPdP/+yLNuqO99Hs7vz6M436GhwCvGmJ2stda7syi+I951m+N8tzbbho+DgCFAS7cer1lrpxhjDsDThSGK31+I7fcG4Bh32bc436FrgWOMMYXAezjf4YNxvs/93NecDRzqHt9lwJ3APTi/3bjB2Nk4XVX8AnFb/P5Ya9cF19et89VE/g60AXrgfBfPxnlPXnOXHwgsUjBCpO6mTZv215yc3Ctqamo6oix3SQ/VWVlZi6uqKq8aPHjw26FWUkBCBLDWrjHOGBIX4zQwOxpj3gROsdYucVf7w3Pn8BmcRsrV7sXvO8aYcpzgxPeBO2WuGcaYp3AuGl+OsWgVwDbGmHbW2uU4d5/BudB701r7pvv3u8aYb4AD3X7DQ4B93bJ9bIx5bYst+xsCFFtrr3b//t0Y8xDOxXLgh+TTwH7dO91/D7Uxa6138LpbjTGX4TR8p0fY1q44F963u43Z540x54cp99E4gZsF7rZuwGkkeE2w1s50l48A5njuYn9njHkBOMq98D4VGBC402iMuQ6nMfZPd1+PWWt/dJddidsAcE3EeX/eMsa0Af4KnBmi3BVAP2PMdLcBGAiCnQo8YK39MrBNY8y/3OPykbXWG3B5xhjzT2BnNgWdFlpr73LLVw2cCOxqrf3TXf4/d1lgG1e5d72nG2Om4zRgNgtIGCcL5RhgoLW2FCg1xtyK0/AKBCTmWmsfctefiJPd0wEnCBKN4Tif8WY4DdXXjTEDrbWVoV5grS1yG5xj2bLR9Ya19mO3PJcCq40x3ay184PWOxJ43bPuv3HujgccD5wdSNk3xlwFPMCmxlgFzm9BJfCme2fZsOn7Wh8HEeKzihMYrS/fshtjviT89yDYRziBw1txgmXXA/viBHqG4wQXA8J971sAN1hrq4EPjDGv43y/rgzaX9jvSNC6Re7/pRGOxQ2BzBv3d3QgTsAhWMjfX5zvP0T5e8Om9/Ala+1X7vqT2ZSlcxCw2Fp7q/v3RpygGcDpwFme370rgXnGmBM835lST/2Dhfr98RPpO1CNkwlW5i6fBPzbGNPSWrsG53fiiTDbF5Ewpk2b9tfc3Py727btUJ6f32RVVlZWXbqfiSRUTU1NVnn5xlYrViy5e9q0aWeFCkooICHicu8IjwMwTor0JJyL6EBjc4ln9Q3ua4Kfa+G+fhecu4vbAfk46c5+d+0jOQknw+AXY8wfOA3H13HuQh1ljDnYs24e8CHQGVhlrV3nWTYXJy03kh5AZ/euc0AO8Innb2/jcj3QxBiT69doNMb8w61DZ5y7ai1x7p6H3Za7/p9Bd9Z9U+1dnXG6LgQENziDn+sB7BJUz1ycC+ZinAbxNE+DPQvnOAT2NS1MuSYBP7uN5KOBTzyZC8GOwLnLeIMxZgZwibX2c7d8Y40xZ3vWzXf3jTFmDE6XkZ7ushZsfly9dW0HNMHJqggl+H1o4bNOO5zPmLe+c3HurG+xHWvtevf4+W3LVyAgAJQbY87FSSPva4xZjZPGHlivRdDr1hlj7geWGWP62k19/ed71llrjFnJlp8Vgp9zt7ciaHlwvTt7/l4R9PkPdQzrItxnNR5ClT3S9yDYR8AtxphO7jrPAlcYY3riZGB871k33Pd+vhuMCAj+jAWE/Y4EKXH/L8Rp0IcSzfcgsO9Qv78B0f7eRNp3N0J/d3sAL7lBx4AqnCBgIPhYyKb6Bwv1++Mn0ndgmSfzDWvtQmPMZ8ARxpiXcLI/zg2xbRGJICcn94q2bTuUFxQ03aLLnEiqysrKqikoaLqhbdsOLF++6Ao23dzcjAISIj6stb8Yp19vXQcEexInDfkAa+1G4/QHbxf+Jb7lmAUc66awH46TKdAW52L3CWvtKcGvMcb0AFobY5p7ghLdCTMGg8d8nEyQ3rGW1accw3D6N/8FmGmdcQZW4TRqIlkEdDHGZHmCEt0JfWG+iM1T7P2CL976z8fJNNgveCX3WG8A+nsyCoL35d1+d+9Ca+2fxpjPcd6vE3DuEPuyzgwuo9xU9bNwGnHd3PJda33GcnDf34dwjuvn1toqY8z3bH5cvXVdjtMI25pNmSl1sRznjmoPNgUHurOp4ROraD6PNUCWtXYekRv42TgN6C5AICBR+z4Zp8tTG5zBMIMtAvp61m2G0yUlYCFOvWe6f3cPsZ1Ioq2zV8jPagNbTvjvwWastbONMetxUvU/drPOFuNkMnwaFGQIZSHQzRiT7Vm/O06XumAhvyM+ZVtnjPkNp0vSskjrRyHk769HVL83Ue7rmDDLTrTWfua30A3ybEOI732Y3x+/z2mk74DfaybijCuTi/NbVdffCpFGr6ampmN+fpNwWUwiKSs/v8lGt6uRLwUkRKjNiBgJPGOtXWCM6YaTGVHXlOtCYKUbjNgZZ6q5qOe895RrNPC2tXaZ5+5aNc5d+K+NMX/F6V+ch5OqPNtaO9dNH77KTWHeGafvse/AcEG+wknFvxinP3I5TkOtqY19+tNCoBKnAZBrnIEYW0b52s/d155jjLnXLf/ObH4H0utZ4FxjzBs4Y0hcHGH7r+PcFTyBTTMzDATWWmt/drup/NcYc5a1dqlx+oxvZ51xNJ4FHjPOoHxzgCt8tv84cAnOBfyLfgUwzjgZR+F0FVhtjFnDpnEdHsK58/keznvSDKd//sc4/dFrcBtWxpjxuGOL+HEDQY8Ct7n1XYJzLL8Nf4i22E6VMeZZ4Fo3Q6MNTpbGLbFsx2MJzlgUQO3AfXk4A242xemy8SdBXUc86++H02iegXNMrsFJOfeuf6BxumJ9hTP47Bc+3TUAnge+9Kx7NZv3z30KuMwY8zXOsb8c5zsYqyVAV2NMvrW2PMw6W3n+DvtZrcP+twIiTvvpfm7CfQ/8fITTsP2b+/dU9+//RFm+L3GyAy5yuwPtgfPdH+KzbsjviNulKNibOF1HfBvvMQr3+7vAZ/36vIev43x3/44T3MwH+rldVe7H+T6OdX/3i4HdrbWBrls743QV8RvQMtzvzxKgrTGmlbV2tftcXb4DL7Op21ZMg9aKyBay1U1D0pX72Q057okGRBFxlOIMyPelMWYdTiDiR5wR1+viTOBqY0wpzoVbTCPie4wAZrr9uu8AjrHWbnAbVaOAf+E0TOcDF7LpO32cW5+VOA3mUIMqbsY6M0MchHOx/AdOg+9hnJTrWL2N0//6V5z03o34d6XwK0c5TobBOJw6/B8hGvauh3ACPjOA73AaH5U46ct+2y/FGTTvGJy7fItxBswscFe5GKfR9oV7of4ezpgAWGfWiNuBD9x1PvDZxUu46dTW2vVhyn0CMMfdx+k4/bSxzowDp+Bk2axy9zPOXfYTTj/9z3EaDtsTuZH1D5yG/tc4x/NG6vb7fzZOwOd34FOcTKBH67AdcMYYuMwYU+J27ekAPIPTTeN3nO4oB1lrK0K8vginkbQaJ3Nma2CEN23cLd8VOHUejNP3fwtuX/+/uesvwjnm3oblNTgDJ87AOY7fus/F6gOcO8yLjTHLQ6xzB3CkMWaVMebOKD6rsbgSZ6yFEmPM0ZFWJsz3IISPcAKRH4f4Oyz3e38wTnr/cpzG7Bhr7S8+64b8joTwIM5sMtFkaEUqZ6Tf3+D16/weuq/dD+e4LMaZrWhvd/EdOIHmd9xzzRc4v/sBx+MELUIJ9fvzC85363f3s9KZOnwHrDMuzQs4A4q+CGCMecsNlOP+vdbNpsMYM8xsmtlDREQaiayaGgXbRDKdcQY728ZGMe1nJjDOKPH3W2t7JLEMvwGn2S1nkRCRJDDGPAk8a2Of7SjtGGPa4wSEBgUF6RJdjsuBPo3l3CPSUL777vs5XbtuHSqQLZLyFiz4rd2gQQN7+i1ThoSIpD1jTFNjzIHGmFw3rfwKnCyFZJXnCJy0Zr/sCRFJAmvtcY0hGAFgrV1qre2b5GBEG5xBjR9MVhlEJPXU1NTw3HNPtx079pht//KXPQbtu+/QQSefPMa8885bm2XjXnrphT2HDt1p8JVXXrrFzaUTTji676WXXtgz8Pdzzz3ddujQnQYfddQh/aurNx+y6MYbr+166KEjtg/8/cILz7YdNmzI4GnTvm7uXa+0tDT7oIP2G3Dxxef3ilddJToKSIhIJsjCmT5vFU6XjZ9xusoknDFmKk5f779FOZCfiEhGMcacgtOV5S3P7DkiIlx99b+733XXbT379Nl23eWXX/Pbv/51xW/t23cou/rqf2/zwAP3bDHw4dSp77dduPDPvGi2vWjRwiZvvvla63DrHHHE0Sv69Nl27W233dS9qmpTz9677vpvl7Kysuzzz78oqu7FEj8a1FKkEbDWXpnsMjQkd5wGv4HvEs5au1eyyyAikkzW2odwxvYREak1ZcobRe++O6X4jDPOnnf88WNrZz3aZ5/91txyy/UVkyZN6LLbbnusGTBg4HqALl26biwtXZM7YcIjHf/1r8sjBgr69u1f+tRTT3Q66KBRYWckufDCf8497bTx/Z544rH248advPSHH2Y0nTLl9fYnnXTa/A4dOm4xjb00LGVIiIiIiIiINAKlGyuzJ3+zoN0N783qMvmbBe1KN1YmrD34wgvPtu/QoWPZMceM3mIK5lNOOXNxkyZNq5599qn2gefy8/OrR406Ysl7773dbvnyZRFvpI8Zc+KiuXPnNH3vvbfDDsa+7bb9No4cecjSyZMndl648M+8W265vkf37j3Wjx49bmm410nDUEBCREREREQkw30+Z2WLAx/4Yof7PpvT7YXpizre99mcbgc+8MUOn89Z2aKh911ZWcmsWb+2GDJkl5KcnJwtlrdq1aqqf//tSn/66cfNynLccWOW5uXl1kyY8EiHSPsYNGjwuu22G7Bm0qQJnSKte+aZ5y5s2rRp9Zlnnrzt77/Pbn7eeRfN8yuXNDwFJERERERERDJY6cbK7Ite+an3xsrq7LLK6myAssrq7I2V1dkXvfJT77VlDZspsWLF8tzKyoqsDh06lYdap337juWrVq3M9z5XWFhYPXLkqKVTprzeftWqVREjBmPGjF88e/as5p988lFhuPVatGhRfeSRxyxavnxZ/rBhe60YPHjIuuhrI/GkgISIiIiIiEgGe/XHxW1qQiyrcZcnsjyxOOGE8UsAJk2a0D7SurvvPqy0T59t1z7++KNhsySqqqp4++232mVlZWHtzy3Ky8uz4lVeiY0CEiIiIiIiIhlsfsmGgkBmRLCyyurs+as2FDTk/tu2bVeZm5tXs2TJovxQ6yxduji/des2W2RQtG7dumrEiJFLX3/9lQ5r166N2H4dPXrsop9/nln49ddfNg+1zhNPPNZ+wYJ5TS+99MrfVqxYkf/ww/dH7BIiDUMBCRERSTvGmH7GmG+MMQm7o2GMmWCMuSZR+/Ps90pjzKRE7zcdGGPmGGP2DbFsgDHmf4kuk4hIKupW1LSsIDfbdzrygtzs6m6tm5Y15P5zc3Pp3bvP2m+++aqounrLYqxZsyb7p59+LOzXb7u1fq8fO/akJRUV5VlPPvl4caR97bPPfmt69dpq/YQJD/tmSSxZsjh38uSJnQ86aNTSESNGlowadfji559/ptP8+XNDBkuk4SggISISI2NMT2NMjTGmwadOdhve/3MfX22MOcezLN8Y87zbKKsxxuwV47aPNsb8zxiz3hgzNWhZH2PMK8aYZcaYlcaYt40xxrM8yxhzjTHmT2PMamPMVGNM/wj762yMWRBi2X+MMT8YYyqNMVdGUfz/ALdYa0NloKYdY8wDxphTk12OTGGtnQGUGGMOTnZZRESS7ZDtOq4MFcHPcpc3dBmOOOLopYsXLyp45pnJ7YKXPfzwfZ02bNiQc/TRx/rOdFFc3L5y333/uvzll1/oWF5eHrENe9xxYxdNn/5dq99+m9UseNltt93YrWnTptVnnHHOQoDTTz9rUcuWhZW33npjt7rUS+pHAQkRkdQ2GPjG8/jboOWfAqOBxXXY9krgduAGn2VFwKuAAToAXwGveJYfBZwIDAPaAJ8DT0TY34HAlBDLZgMXAW9EKrQxphOwN/BypHXTzAHAm8kuRCqJQ9BvMnBaPMoiIpLOCpvkVt80qt+sJrnZ1YFMiYLc7OomudnVN43qN6tFQa5v9kQ8jRgxsmS//UYsu//+u7tfd93V3T7+eGrhhx++1/Kyyy7q+eKLz3UcPXrsnwMGDFwf6vVjx560eP36dTl//rmgSeR9HVjSpUvXjT/99ONmg1t++unHhZ999kmb0047a36LFi2qAZo2bVpz+ulnz//mm6+KPvzwvZb1r6nEosHv7omIpDJjzMXAOUBLYCFwprX2fWNMNk4D+RScxvn7wOnW2pXAx+7LS9ykgf2AvwLbWGtHu9vtCfwB5FlrK90MhM+BvwDbAh8C493thbMTMM19PAj4PrDAWluOE1DAGFMVa92tte+5rz3ZZ9lXOEEI3HX+C1xmjGlrrV0B9AI+tdb+7i6fBJwXYZcHAr5dD6y1E93tHB9F0fcDvrXWbvSUry7vI8aYE4BrgBbAbe56JweOTTjGmIPc1/YEfnK3O8NdNge4GxgD9MAJxIz1ljloWwOAEmutbwYJ0MQY8wzOMZyF89mZ7r62M3AXsCewFvivtfZOd9nOwB1AX2AD8AJwvvvZwRhTA/wN573riPN5moATXNrOLffowPpBZd4GeAQYCFQA71tr/89dtq1bpsHAMuDf1tpn3WUj3eO2NbAaeMRae6W7rCfO9+Zk4ApgDrCnMeYU4HygKzDfLVMgODfQGHNbiOM8FXjYGFNgrW3QdGQRkVS3W882a986fdfpr/64uM38VRsKurVuWnbIdh1XJiIYEXD55f+Z16/fdutef/3l4vfee7tddnYWPXtutf7yy/8ze//9D1gd7rVdunStGD58nxXvv//OFhkWwbKysjj22BMW3XLL9b0Cz5WXl2fdccct3bfffoc1I0cessq7/ogRI0tef/2V1XfffXv33XcfNrOgoCBjMjBTnQISItJouV0QzgKGWGsXuo2hwJRSZwOHAsNxGlR3AvcAx+I0/P4Aiqy1le62/hrFLsfgBC7+AB53tzk6RNneBYbgNJTXG2PuApoDC4wxn1trD4iifscBl1hrB0RRtkj2BBa7wQiAp4GjjTF9cOozltDZDxhj8txtjI1DWbYHrGfbdXofjTH9gPtwGvlfAtfjNHgjMsYMAh4FDsbJYBkNvGqMMZ6G79HACGAj8BkwDrg/xCYPJHx2yCicz95o4FzgZffYVwGv4WSvHOuW/z2nGPZtd/l5bhm7Am8BZ+IGslx/xQkcdMPJwNnd3c8KnCDascBEnzL9B3gHJ1slHyd4hjGmOfAucDlO1sf2wLvGmB+ttT8B63C+CzNxgh7vGmO+t9a+7Nn2cJwgSrUx5ijgSpz38RucQEaFZ92Qx9la+6cxpgIn02dGiGMrItJotCjIrT5ucNflydp/VlYWRx11zIqjjjpmRbj1rr325jl+z1911XVzr7rqurne50Jt79BDj1h56KFH1N74yc/Pr3nuuVdnhtrn3Xc/ODtiBSTuFJAQkcasCigA+hljlllr53iWnQ6cFbhj7Y5rMM+9o15XT1hrf3S392/ge2PMWGvtFtkN1tr93Abnk9banYwx/wIqrLU3R7sza+2TwJP1KC9uWbviNOLP9zy9CKe7iMU5jvOBfcJsZk9gurW2tL7lwcl08F541PV9PBJ43Vr7sbvs3ziBjWicCjxgrf3S/Xui+x7tCnzkPnentXahu+3XcDIJQhkJ/CvM8mnW2ufdbd0GXODuqxwottZe7a73uzHmIeAY4G1r7TTPNuYYYx7Aaezf7nn+JmvtGmCmMeZH4B1P5stbOJk5fgGJCpyshM7u8f3Uff4gYI619jH37++MMS/gdPO5ylo71bONGcaYp9wyvex5/kpr7Tq3DCe7ZfzaXRZ8wRjpOJfifGZEREQkxSggISKNlrV2tjHm7zh3X/sbY97GSWdfiNPQeskY401jrMIZT6Gu5nsezwXygHbAEu9KxpizcFLaC9y/S4BCYK0x5lKgj7XWd9CneDPGFOPcBb/XWvuUZ9HlOBkc3XDGrxgNfGCM6W+t9ev/eSDxGx9hFc7xAOr1PnbG855Ya9cZY8LesfHoAYw1xpzteS7f3WaAd1yP9UHLahljinC68YSbEcJbzmp3cNDOONPHd3Y/IwE5wCfutvvgdEXZCWiGc973Bilg88/fBp+/O4Yo00U4WRJfGWNWAbdaax/FOTa7BJUpF3eMEWPMLjjjlmyHc8wKgOdC1RfnM/ZbiDJA5ONcCJQgIiIiKUcBCRFp1AJZBMaYlsADwI3ACTgNohOttZ8Fv8YY08NnU+twGnwBfo047+jN3XHuMG+RNmmtvRu42xgzBbgKZ3yCH6y13aOqVJwYY1rjBCNetdZeG7R4IPCMZ8yDCcaY24F+bBqE0+tA4PA4FW0GQV0/6vg+LsLpFhD4uxnQNsoyzAeu9TkudfFX4AO/TBmP2s+OOy5GV5yxMiqBP6y1vUO87j7gO+BYa22pG7g5Mg5lxlq7GGfMDYwxQ3G6inyMc2w+stbuF+KlT+KMr3GAtXaj+7kJ7g/s7bs7H6ebRsyMMV1wgh420roiIiKSeApIiEij5Y490AWn3/lGnLvBgbEH7geudbtUzHUzBXa31r6CMxZBNbAV8Ku7/vfAxcaY7jgD9f3TZ5ejjTGP4wzUdzXwfIRG6EBgOrALW86uEahDAc6MXQD5xpgmQFk002EaY3JwsjRygWz3tVXW2gq3Yf828Jm19hKfl38NHGWMeRrneBzvbmuL/pfGmF5AgbX25zBlycM59tlArluWihDH513gDmNME7dBW9f38XngS7cx/RXOexLt7FMP4WRevOe+thmwF/BxHbqlRBo/AmCwMeZwnJlPzgHKgC9wPoel7qCed+J04egLNHW7OBQCa3Cya7YFzsB5v+rNHdvhczcotQoniFANvA7c4HaLedpdfSCw1v0MFAIr3fduZ+A4nMBXKA8DtxljPsX5HmyN89mYG+Y1AcNxgj0a0FJERCQFadpPEWnMCnBSx5fjpH23Z1Mg4Q6cxt87xphSnMbfLgBul4Rrgc+MMSXGmF2tte8Cz+DcvZ+G0ygL9gTODAaLgSY4DUtfbmBjhbuvHdkyzT7A4jTAu+AEEDbgpMxjjDneGBNy8CacDIINOHfRh7mPH3KXHYbTJWO8MWat518gS+NGnGDJ9zjp8OcBR1hrS3z2M5LI3TUecvd/LHCp+9h3vA5r7RLgA5yBHqHu7+NMnBkmnsQZE2MVEGqWi+AyfIOTHXC3+7rZOIMpxsQYk4WTIRFyQFDXK8D/ufs6ATjcWhsI2ByE0+D/A+cYPAy0cl/3D5wGfynOMX4m1jKGMQQnoLMW5xifa6393Q3I7I8zjsVCnPfkRtwuSDiDal7tvh+XA8+G24m19jmc79uTbj1explqNhrHE3ogUREREUmyrJoazWgiItLQjDPt5yRr7cPJLkuiGWPeBO621sZrDAncGTImAjtHkw0Sw3bnEOW0n3Ha3844x2bnROyvMTHOVKoPWGt3S3ZZRETq47vvvp/TtevWSZsZQ6S+Fiz4rd2gQQN7+i1Tlw0REWloU4EP47lBd/rIIfHcZhJdkewCZCJr7QxAwQgREZEUpoCEiIg0KGvtTckuQ6qy1n6V7DKIiIiIJIsCEiIiCWCt3SvZZZDIrLU9k10GERERkcZCg1qKiIiIiIhIg6qpqeG5555uO3bsMdv+5S97DNp336GDTj55jHnnnbdaBa976aUX9hw6dKfBV1556RZTrZ9wwtF9L730wp6Bv5977um2Q4fuNPioow7pX11dvdm6N954bddDDx2xfeDvF154tu2wYUMGT5v2dXPveqWlpdkHHbTfgIsvPr9XPOqabPPmzckfOnSnwe+99/YWxzbVKCAhIiIiIiIiDerqq//d/a67buvZp8+26y6//Jrf/vWvK35r375D2dVX/3ubBx64p6Pfa6ZOfb/twoV/5kWz/UWLFjZ5883XWodb54gjjl7Rp8+2a2+77abuVVWbZha/667/dikrK8s+//yL5sdUKak3BSRERERERESkwUyZ8kbRu+9OKT7ttL/Nu/TSK+cPH773mn322W/NddfdPOfQQ49YPGnShC4zZnzfzPuaLl26bmzWrFnVhAmP+AYrgvXt27/0qaee6BRpvQsv/OfcBQvmNX3iicfaA/zww4ymU6a83n706LELO3ToWBlLvQ49dMT2zz33dNtYXiObU0BCREREREREGswLLzzbvkOHjmXHHDN6WfCyU045c3GTJk2rnn32qfbe5/Pz86tHjTpiyXvvvd1u+fJlEcc+HDPmxEVz585pGqmbwrbb9ts4cuQhSydPnth54cI/82655foe3bv3WD969Lilsdesbp56alK7Y445rN9ee+2244EH7rPD+eefvdXq1atzAL755qvm55xz+jYjR+47YJ999hh03HFH9HvppefbBG9j/vy5+Rdd9PdeI0bstcM+++w+6NhjD+/38ssvbLbehg0bsq+88tIe++03bODBB+8/4M47b+0c3K3l559nNjnrrFO32XffoYP23XfooAsuOHurJUsWJ2ysSQUkREREREREGoGsstXZTb9/sF2Lqf/s0vT7B9tlla1u8PZgZWUls2b92mLIkF1KcnJytljeqlWrqv79tyv96acfWwQvO+64MUvz8nJrJkx4pEOk/QwaNHjddtsNWDNp0oSIWRJnnnnuwqZNm1afeebJ2/7+++zm55130Ty/sjWEe+65o9O9997RY7vtBqy94or/zD777PPnNm/evGrdurXZAAsX/pnfv//2a88//+K5V199/ew99hi26vbbb+75yisv1gYbli1bmnvGGSdtO3v2rOYnnXT6gquuun72/vsfuHzJksX53n098sj9XZs2bVp12WVX/z58+D4rnn32qU5vvPFqbbeW33//reCcc07ftqKiPPvCC//1x/nnX/zH/Pnzmv7jH+f2rqmpScjx0CwbklGMMROA7ay1O8XwmvbAmcAEa+2cBipagzHGnAL8C+gGfJrOszkYY3oCfwAHW2tfj/G1pwJLrbUv12G/E4jxcyMiqUnngYY5DxhjLgK+stZODXq+BjjbWnt3vPfpbn8UcAOwNbAw2plwjDFXAmdZa9u5f+8FfAhsb639sSHKKpLq8uZNbdHqrVN6U1NDVtXG7JqcJtXNv7y52+oDHppV0X2vtQ213xUrludWVlZkdejQqTzUOu3bdyyfPv27LTIbCgsLq0eOHLX01Vdf7HDSSactbt26dZXf6wPGjBm/+KKLzuvzyScfFQ4bNrw01HotWrSoPvLIYxY9+OC93ffcc+8VgwcPWRdNXSort+zRUVNTvdnzOTk5ZGVl+b6+pKQk54UXnuk4cuQhSy655N8LAs8fcMBBJYHHhxxy2KpN265h1113L122bFn+G2+82m7UqMNXAjz++KMd1q/fkPPww0/83LFjpwoAv/r27btd6cUXX7YAYPjwvdd8++3XrT7+eGrrgw8+dBXAQw/d27lVq6LKu+56cFZ+fn4NgDF9N4wbd+x277//bqt9991/dTTHpT4UkBCB9sAVwFRgTlJLEiNjTEfgPuBu4DlgVfhXZLRTgR+Bl5NcDhFJPzoPRHaRu4+pDbT9LRhjcoDHgbeAU4CoGgwisqWsstXZrd46pXdW5YbajIisqo3ZAK3eOqX3inHTptcUtKwOvYXkOeGE8UteffXFDpMmTWh/9tnnLQq37u67Dyvt02fbtY8//mincAGJqqoq3n77rXZZWVlY+3OL8vLyrECDPJy99tp1cPBzd955W88777ytZ+Dvc8/9x5yjjjpmhd/rv/326+bl5eXZhxxymO9ygFWrVuXce+8dnb/66ouiVatW5ge6WLRu3aYisM6MGd8XDhw4aE0gGBHKzjvvssb7d7du3TcsW7asNotixozphXvvve+K7OzsmkBQpUePnmXt2hWX/fzzzGYKSIhIJNsAOcCj1toZ9dmQMaaptXZDfIolIiIJksnngU5AS+BJa+2nyS6MSDpr8vMzbQiVgl9TQ5Ofn2mzYeApyxti323btqvMzc2rWbJkUX6odZYuXZzfunUb3wyK1q1bV40YMXLp66+/0mH8+FOWRNrf6NFjF11++T97f/31l81DrfPEE4+1X7BgXtNLL73ytxtuuGarhx++v8OZZ56zONK27777wZ+9f1922UXb7L//gcv33HOvksBz3bv3KAv1+tWrS3IBOnToGDKQcOWV/+o5a5ZtccwxoxdutdU2G1u0aFH1wgvPFn/99RdFgXVKS0tze/c26yOVt7Cw5WYZJbm5uTUVFeW1Qam1a0tzX375+Y4vv/z8FgOHLlu2JOT7FU8KSEhGM8Z0Aq4F9sK5sJkPPAtcba0td7sI/OCu/qExBgBrbZb7+jY4qaKjgFbAt8B51tovPfuoAf4OdMC5g1ODc5fqfGttmWe9Hu629gOaAbOBG6y1TxpjvgJ+staOCyr/BGAHa+0gn7pdiXNHD2C6W/bx1toJxph2wK3AQUBT4CvgH9babzyvnwO8AJQAp7nlDzmtkjHmZOA8nIvfxcA91tqbPMt3A/4JDMG5gJwF3GytnRy0nZDHwbNaM2PMA8AxQCnwCHCVtdY3cm+MmQoMBgYbY8a6TweORQ7wb+BEt46zgWuD9he8vXzgaWAnYB9r7WxjTHfgJmB/oAnwCXCOtda6r+mJ093k/4C/RFt2EWlYOg/U/zzgrtcWuMIYE9jf3p7uGznGmOsi1D3sb6jPPscBj7l/vuLW7SpgAj5d++LR9c4YkwdcDxyNcyxWAF8C/2etLY+2HsaYbsADwN7AEuAaYATQLtCdxq+8ft0WjTHZONkpJ+N0yZmLcw6b6HndVGA58CJwNU7Gz2fAKdbaBZ71muIcw/8DOgILgaettf/0rBP2XC/pK6fkj4JARkSwrKqN2Tmr/yhoqH3n5ubSu3eftd9881VRdXX1guzszYuxZs2a7J9++rFw5513Kwm1jbFjT1ry5puvtX/yyceLI+1vn332W/PYYw+tnzDh4U7du/fcGLx8yZLFuZMnT+x80EGjlo4YMbLk559/Wvz88890OvjgUSu7desRslsJwMCBO24WBMjNza3p1KlzWfDzobRqVVTpliGvbdt2W/T/2LhxY9a3335TdPrpZ807/vixtQOAPv/805v1ASksLKxcuXJFVNOhhtO8efOqXXbZbdWoUUdsEYxq06ZNTDOO1JUGtZRM1w5YCZyPczFwMzAeuMtdvgg43n38N2A39x/GmALgPWBf4ELgUGAZ8J6bIut1AdAZGO3u4zTg3MBCt3/y5ziN9X8AB+M0VLu5qzwCHGmMaeF5TQvgSODREHV72C0zbh12A95w/34Z+Ku7r//D+a5/aIzZJmgbxwHDcfpO/1+I/WCMuRAnJfhlnIvb+4D/GGPO8qzWA+cC6CS3fi8Ajxljjo3hOATcBKx16z8JuNx9HMqZwC/Am2x6DwPH4mrgUuBB4BC3jJO95QqqaxPgJWAHYJgbjGgDfAoY4HSci9XmOJ+FpvUsu4g0LJ0H6n8eOAxY7ZYxcHy+jaHusfyGBrwBHO4+/oe7z4dDrBsv/8Q5jv/GCRr9HafeORBdPYwxWcArwHY458PzcY7FbnUs013AZTjnsJE456dHjTEHBa23C3AWzntxKrCj+xqCynUGcA9wIE4wq51nnWjO9ZKmqop6ldXkNPG9OVKT06S6qlWvkHf14+GII45eunjxooJnnpncLnjZww/f12nDhg05Rx99bMhZLoqL21fuu+9fl7/88gsdy8vLI7Zhjztu7KLp079r9dtvs5oFL7vtthu7NW3atPqMM85ZCHD66WctatmysPLWW28Mvh6Nux13HLIuPz+/+rXXXvGdKrSsrCy7pqaG/Pz82veqtLQ0++uvvyryrrfDDoNKp0//ruXSpUvqlWCw/fY7rJk3b27THXYYtH7gwB03+9e9e8+wwZl4UYaEZDRr7Q84FzIAGGM+w+mD+qgx5mxrbZkxJpDi+pO19gvPy0fjXFD0t9bOcl//HmBxTvgXetad47mr9bYxZg+cC6nAXYXzcO6sDbbWBvq+ve95/VPAbcBRbLojdDTOnSrfO/nW2gXGmJ/cP2cEBukyxowA9gD2stZ+5D73AU6/6AtxLhS9DrLWbhE9DjDGtMS5aLnGWnuV+/S7xphmwGXGmPustVXW2qc9r8kCPga64twxeyrK4xDwsbX2As++RuAcz2dDHIufjDHrgGXe99C9ePy7W/Zr3KffNsZ0Ba70lCuwfjPgVbfce1pr//SUuzkw0Fq70l33M5xjeiLOxV2dyi4iDUvngfqfB6y13xljKoEFQccnlrpH+xsa2OcyY8x3m/509utmETSUnXG6h0z0POf97Y6mHgcAg4BdA1k0xphpwG84mYNRc4NHZ+BkvQTK9J6b9XMF4B38uSUw0lq7yn1tR+C/nm44++MEWUZZa1/1vO5xd/2ozvWxlF9Sy8a+/7ey+Zc3+ze4s7LY2Pf/Vjbk/keMGFny5ZefL7v//ru7//HHH02HDt2zpKqqMuv9999pM3XqB21POGHcnwMGDAybZTB27EmL33nnreI1a1bnbrNN77BjyowYcWDJhAkPbfzppx8L27VrV9uw/vTTjws/++yTNv/85+W/t2jRohqgadOmNaeffvb8a665YusPP3yv5d5777sm9Jbrp6ioqOroo49bNHnyxC6VlRVZu+8+dHV5eXn2//73aavTTvvbws6du1RstdXW6ydPfrxz8+YtqrKysnn66Sc6NmvWrGrDhvW1gZgxY05c8uGH77c944yTtj322BMWdezYqfyPP35rsmHDhuxTTz0zYreWgFNOOXPhGWec1Pess07d5sADD17RunXryiVLluR9882XLUeOPGTF7rsPCzkOR7woICEZzW0Yn4tzt6AXTnplQHecdNlQ9gWmAX8YY7zflY9wUvm93gn6+6egdfYBpnguQjdjrV1jjHkeGMemC9FxwKvW2pCD3oSwM85sEx95tr/OGPM6MDRo3ffDXYS6dsO5AHsu6Dh8gHMXqSsw1xjTGicVdBTQBfeOEvCn5zVhj4OH3/HsHuE1frbDSYt+Luj5Z4AJxphia20gHa45MAUoAoZba70/5vsC7wJrPMegFOfzEc1noS5lF5E40Hmgdvv1OQ9EEqnusfyGJtP3wBnGmCU454MfrLXeTvfR1GNnYIm3S4+1dq4blIjVX4Bq4KWgz9/7wLHGmBxPkODrQDDCFQhUdcH5jO8DrAwKRnhFda6vQx0kRdQUtKpefcBDs4Jn2SAri9UHPDQrEQNaXn75f+b167fdutdff7n4vffebpednUXPnlutv/zy/8zef/8DIg6e2KVL14rhw/dZ8f7772yRZREsKyuLY489YdEtt1zfK/BceXl51h133NJ9++13WDNy5CGbDQA8YsTIktdff2X13Xff3n333YfNLCgoaLA5L08//azFLVu2rHz55Rc6vPvulOJmzZpX9evXv7RFi8IqgCuvvO73G2/8T49bbrmhV4sWLSoPOmjU0o0bN+a8+eartd1V2rUrrrz33od+ufPO27o++OA93SorK7M6dOhUduyxoyNdY29mm216l91770O/3Hff3Z3vvPPWHuXl5dmtW7cpHzBgYGmPHr3qe26IigISkun+jpM+eiPOBeQqnHTZe9j8otRPO2BXwG/Qmd+C/i4J+rs8aPttga8j7O8RYKoxZisgCxiGk1IZq06AX8rbEqCNz3ORBH70Z4ZYHujTOgHneP0H50JoDc6dnVGedaM5DhD5eEYrMA91cD0Df7fBSb8GJ924N07f3OD1A58Fv3Tm4AyPkqC/61p2EYmPv6PzQEBdzwORlAT9HVz3WH5Dk+kanADAmTiflz+NMTdba+9wl0dTj474H/ulQGGM5WmHE9wP1VDrBATGiCgJWha4Ixx4H9ridE8Kty+IfK6XNFbRfa+1K8ZNm97k52fa5Kz+o6CqVa+yjX3/b2WiZtfIysriqKOOWRFqBgqva6+9eY7f81dddd3cq666brPPYqhtHnroESsPPfSI2syP/Pz8mueeezXUZ5y7734wXIDa18svT/kh8lpbOu64McuPO26M7yCiW221ddkDD0z4Nfj5c845f6H3727depTffPMdv/tto3v3nuWffvrNFoFQv+Pau7fZeNttd/luJxEUkJBMdxTwvLX20sATxph+Ub52JfANTqM6WKz97FawqXHsy1r7sTFmFs4dsSycwaaC7zpFYxHOgFbBOuDUySua6G/gNQfhf+Fq3XEXDgL+Zq29P7DAHYzLK+JxiLPAxVd7d98BHdz/vcdjFnAHTubEYmvtfZ5lK3G6cvzHZx8NnsomIvWi88AmdT0P1Fc8f0MDd+yCR39vHWuhgrmZIpcDlxtjeuOME3G7McZaa6cQXT0W43/s2wPeGUw2ErkOK4FKnO43fg3GkP3tfUT6/EU818ewL0lhNQUtqxtqNg2RulBAQjJdU7a8aDw+6O/guwgB7+P0uZxnrY3lpO/nfeAcY0wHn7vvXo/i3JkBeLyO/TW/BK4yxuxprf0YasdGCAyGFavPcS6iOltr3/BbwRjTCmfANO+I6oU4g0h6L3ajPQ514ZeJ8COwHqdBcrXn+aOBXz3dNQCw1j7hDiJ3tzGm1Fo7yVPuo4GZNrWmxBORyHQeoN7nAahftlc8f0OX4mSs9A084f5u704c7+Bba2cZY/6BM2hoP5wuHNHU42uc2Uh28Ywh0R1nkMnPPOstAHoaY5p4uszsH7StD3AyJFpZa9+tZ5XeBy4yxhzknZ3EI+K5XkSkISggIZnuXZwLwC9x0muPx5nKymsezkl4rDFmNVDhTov2OM7dkanGmFuA33FSHncGFltr/xtDOf4LjAE+McZcizPtXF+gedB0WhNxUkZz2dSHOCbW2reNMf8DnjHGXIJzV+QfOBflN9dheyXGmVruDuNMWfcxTvChD860b4dZa1cbY77Guau0BudOziU4aaYtPZuL9jjUxS/AX40xf8Wp8x/W2hXGmNtxBuSqxLnTeThOCrTvLBvW2vvci9vHjDFrrbUv4ww0Nxr4wBhzF864GB1wRqb/1Fr7lN+2RCQl6DxQz/OA6xdgpDFmCs5MQtZaG212Q9x+Q6211caYV4DzjDFzcboqXMDm2Qd1Yox5CWc8iO/c7R2J8z58HEM93gSm44zFcDFOMOwqtsxmeBknUP6wcaYAHYQzMKa3rtYYcz/wtDHmJpxzWBOgP9DHWntyDNV7F3gbeNIYczXOLCmdcAZwPi2ac30M+xIRiZqm/ZRMdzXOyOXXuP+XA+d4V3DvTJwCDMbpX/y15/m9cU7iV+Gkzd6BM87AV7EUwr0TvwfORc7tOCNjn4pzEexdbzHOna3PrLVb9B2LwaFuuW/HGdAxC9jHWhtz3zi3XDe55T0AZ9qwp3Au6j/xrHYczsX64zjH6QX3sXc7UR2HOroG+BlnRPSvcabUAyf99nqclOvXgT2B0d5ZQYJZa292X/O0MWY/a+1ynH7Dv+A0Kt7BGT2+FTAj1HZEJCXoPBCH8wDO7BzrcKbj/BrnWEWlAX5Dz8LJNrgXZyyQp3CyCerrfzjH7Umcc91g4Ag3OBVVPdxBMA/BGUvpUXe9u3EyEGq5M6KciDOY5Ks4QY3xPmX6G04XkTE4wY4JOJkuH/usG5JbrsNwpgL9O/AWzndiuWedaM71IiJxlVVTk4iugyISDXeayj+Bs6y1jyS7PCIiklg6D2QmdwaVdtbavZJdFkk/3333/ZyuXbfWuA+SthYs+K3doEEDe/otU5cNkRTgjrfQD2dqulKcuxIiItJI6DwgIiKNkQISIqlhMPAhzoBcY6y165NcHhERSSydB0REpNFRQEIkBVhrp+L07xURkUZI54HMZq09MtllEBFJRRrUUkREREREREQSTgEJEREREREREUk4ddkIobq6uqaqqv4zkOTkZBGP7aSqTK5fJtcNMrt+mVw3UP0SJS8vZzlQ3FDbj8d5JlWOVUNR/dJXJtcNMrt+mVw3SJ36NfQ5JtXceedtnZ999slOgb+Liooq+vTZdt3ZZ5+/oFevrcqi2cZTTz3RrnXrNpUjRowsabCCBrnzzts6v/nmq8VTpkydnqh9NjYKSIRQVVVDSUn9x5MqKmoWl+2kqkyuXybXDTK7fplcN1D9EqW4uHBuQ24/HueZVDlWDUX1S1+ZXDfI7Pplct0gderX0OeYVNS0abOqG264ZRbA/PnzCyZOfKTzeeed2Wfy5BdmNm/evDrS699887Xi7t17bEhkQEIangISIiIiIiIi0qBycrJrBg/eeR3A4ME7r+vUqXP5BRecbT788L1WBx00alUiy1JVVUVVVVVWfn5+8tNlGrmEBySMMTnADcA4oAnwDnCatXZ5iPVHALcCWwG/Aedba99xl7UBXga2dbe1DHgMuNZaW+OuMxXYDajwbPYYa+3rca6aiIiIiIhIylpbUZr91oLX2/y5bkFBl+Zdyw7oetDKFnmFEbMTGsKAAQPXAfzxx+9Nhg0bMnjSpOd+6NGjZ3lg+dy5c/JHjz5q+8suu+q3559/usMff/ze7I8/fm82dOhObQHOPfcfc4466pgVVVVV3HPPHZ3fe+/ttmvWrM7r0KFj2bHHnrDo0EOPWBnY1qWXXthz3ry5TY8/fuyixx57qMvixYsKbrzxv7/uuuvua996642ip556vNP8+fOa5ucXVG+zTe+1l1xy2bxu3XrUluWHH6Y3veWWG3rMnz+3aadOncvOPvv8ebvuuvvaRB6vTJWMDIlLgFHALsAK4FHgCeCA4BWNMVsBLwKnAs8CRwEvGWP6W2vnAOuAM4BfrbUVxphewJvAUuBBz6b+Y629psFqJCIiIiIiksK+XvZliyu+/Wfv6poayqvLsvOzC6of+/WhblfteP2sIcW7JLxxPX/+3AKATp06lxcVta545ZUX251zzvkLA8tfeeWFdoWFLSv/8pf9S3r27LXxsssu3rpDh45l48efvAigR49eZQB33nlrl5dffqHD0Ucft6h//+3Wffjh+61vueX6XllZWYwadXhtUGLZsqX5Dz10X9fRo8cubNu2uKJbt+5lL774bJvbbrup1+67D115wgnjF9bU1PDNN1+3XLFiRW4gIFFeXp593XVX9Tr88KOWtG3brmLixEc7X3XVpVu/8MIbPzRr1iwpwZxMkoyAxKnA1dba3wGMMRcBs40xPay1wX2pxgLTrLWT3L8nG2NOd5+/ylpbBswMek01YBqu+CIiIiIiIuljbUVp9hXf/rP3xqqNtbMslleXZQNc8e0/ez+7z6vTW+S1aPDGdWVlJQBz5vxRcPPN13dv0qRJ9dChe66ZP39ewQcfvNv27LPPW5iVlUVNTQ0ffPBu2+HD91mRm5uLMX03NmnSpLpVq1aVgW4fAKtWrcx57bVX2h955DGL/va3cxcB7LXXX9YsX74s74knHuvsDUisXbs29+ab7/x1++0HbACorq7m0Ucf6jpkyC4lN910+x+B9fbf/4DV3jKXl5dn/+1vf58/dOiepQDFxR0qzjjjxH5ffvm/Fnvvve+aBj1gjUBCAxLGmCKgOzAt8Jy19jdjzBpgByA4ILGDd13Xt+7z3u2+DvwFp9vGfOCBoNf83RhzPrAImATcYq2tIIycnCyKippFUavwcnKy47KdVJXJ9cvkukFm1y+T6waqn4iIiMTmrQWvt6mu8R8uobqmhikLXm9zZK9jfLvQx8vatWtz99pr18GBv9u2bVf+r39d8VvHjp0qDjvsyOUvvvhsp//975PCPfbYs/Szzz4pXL58ef6oUYeFLdMvv/zctLy8LHv//UdsNgbF3nvvu+r222/uuXz5stx27YorAVq3blMRCEYAzJ79a5OSklV5Bx54cNh95Obm1uy++9DSwN/GbLsRYMmSJfmxHQHxk+gMiUL3/9VBz5cALUOs77duf+8T1tqD3LEphgAHA94P1T+BX4A17vLJ7r7+Ga6gmmUjOplcv0yuG2R2/TK5bqD6JUpxcWHklURERNLAn+sWFAQyIoKVV5dlL1i3oKChy9C0abOqm2++/desrCyKi9tXdOrUuSIrKwuAnj17lfft27/09ddfbbvHHnuWvvHGK+223nqbddtu229juG0uX740D6C4uP1mN5vbtm1bAbBq1aqcQECiVatWm62zatWqXL/XBmvSpElVdvamQxcYCLO8vCwruppLOL4fygYUiCy1Cnq+CCdg4Ld+VOtaa6ustV/gBDDu8Tz/ubV2lWf55cDoOpVe6m3mqh+4+KvzmLnqh6RuQ0Qk3owx2caY/xljaowxXZNdHhERib90vQ7t0rxrWX52gW+XjPzsguquzbuWNXQZcnKyawYO3HH9DjsMWt+5c5faYETAgQcevPzLLz9vvXDhn3lffvlF0f77H7Ai0jbbtXOCCcuXL8vzPr9ixYo8gNatW1cFngveX+vWrSsBli1butlrJbESGpCw1pYA84AdA8+5A1e2BGb4vGS6d13XIPf5UHKB3mGWVwOKZiXJ47Me5evlX/L4rEeTug0RkQZwHpD81A4REYkrbxAiXa9DD+h60MrsLP8mUHZWFiO6HrTSd2ECjRgxclVubk7Nv/998VY1NdVZI0eO2qxMubm5NeXlFZu1X7fdtu+G/PyC6nfemdLa+/zUqe+37tixU1kgO8LPNtv02di6dZuKt956o218ayKxSMaglg8CFxtjPsSZZeNG4G131oxgjwMXGmOOBZ4HjgQGA2MAjDG7As2B/wHlwB7Aue4+AmNWDAWm4szIMRC4EnimAeolURjT+8TN/k/WNiR6gZPvmN4n0r/19skujkhKMsb0Ac4EjgC+S3JxREQkjgJBCEjf69AWeYXVV+14/azgWTays7K4asfrZyViQMtImjRpUrPnnnuvnDLljeI99hi2sqioqMq7vEuXbhu///7bllOnvt+yqKh1ZffuPcratGlbdfDBo5Y+99xTnXJycmr69eu/furU94u+//7bVhdddOnv4faXnZ3NSSedtuCWW67vdfHF5/fab7+/roQspk37qvCvfz1w5cCBO+omQwIkIyBxA9Aa+BooAN7F7UJhjDkeeMBa2wJqB7w8HLgVZ3rQ34HDPMGLfOAmoA9QA/wJ3OnuAyAPuAxn3IhsnEEtJwPXN2gNJaT+rbfnxp3/m/RtSPS8J2Edd5EtGWOycc5R/8AZ5ygq8Rg8OdMHAFX90lcm1w0yu36ZXDeIvn7Tl03nwR8f4NTtTuPMQWfy4I85nLrdaexQvAN79NolASWNvyHFu6x9dp9Xp09Z8HqbBesWFHRt3rVsRNeDVqZCMCJgzz33XjVlyhvFI0eO2mKgyZNPPn3h9ddfnX/ttVdutWHDhpxzz/3HnKOOOmbFOedc8GdOTk7NW2+91v7ppyfldujQseyCCy7545BDDlvltw+vQw89YmVBQUH15MkTO11zzZVbFxQUVPfu3Wdt27ZtQ2ZWSHxl1YQYbbWxq6ioqtGglpFlcv0yuW4Qff3SMUNC7116S5X6FRcXTgN2irSeMeY8YA9r7ZHGmJ7AH0A3a+2CcK+Lx3kmVY5VQ1H90lcm1w0yu36ZXDcIXz/vNU/ghsyQdrs0yA2ZaM8xAN999/2crl23btAZMFLFzTdf1/Wzzz5u/eKLb/7gHUhS0tuCBb+1GzRoYE+/ZcnIkBCRNKKMFJHQjDHbABcQ5UWliIikrkzompGuZs36tWD27FlN3377zeJjjhm9UMGIxkMBCfGVjnfFRUSSYChQDPxojIFNg0XPMMZcZq29N2klExGRqASue4d1HA5Qe/2rGzKJc9NN1/ScPXtW85122rlkzJgTlya7PJI4CkiIL40bICISlWeB9zx/dwU+B/YHfklKiUREJCK/7hmg695keeihx22yyyDJoYCE+FKamohIZNba9Xim+jTGBM6ri621a5NTKhERiUTdM0RSgwIS4ktpaiIisXNngfKf6F1ERJJq5qofePK7iRzXc+xmQQhd94okj0YLEYnBzFU/cPFX5zFz1Q/JLoqIiIiIROC9dnt81qN8vuh/PD7r0doghMZKE0kuBSQkadKxcR9I73t81qPJLkrM0vF4B3jLnin1EBERkYbnvXYb0/tEduu0u7pmiKQQddmQpEnHAYS86X3pNhNJOh7vAG/ZgYyoR7qVHTT7joiIpAfv+Sq4a8Y9ve6lpGR9hC2ISKIoICFJk44DCHn7GF781Xlp1bhMx+Md4Ff2dKqH33Ri6SjdAyoiIpLZAufbtRWl/Lz6J8A5X+mcJZK6FJCQpEmXAYRC3RVOtwa+93h7B3VKhzvdwZ+VdPjceGVKQz7dPvMiIpL5/Kbv7NuqH0Pa7aLzVQq5887bOj/77JOdAn8XFRVV9Omz7bqzzz5/Qa9eW5VFu52nnnqiXevWbSpHjBhZ0iAF9XHnnbd1fvPNV4unTJk6PVH7TIahQ3cafOqpZ84bM+bEZYncrwISIhGEakymS0DFT6BOlRVVaVuHdJIpDfnAZz4wFoa6boiISLKFmr5T56fU07Rps6obbrhlFsD8+fMLJk58pPN5553ZZ/LkF2Y2b968OpptvPnma8Xdu/fYkMiAhDQsDWopCRFpML9UHuxvTO8Tw0bZU7nsoaTDoE7RHNd0OfaZNpJ3Og/uKiIi6c97/vdep2Xa+TbT5ORk1wwevPO6wYN3XnfooUesvOSSf89Zvnx5/ocfvtcqGeWpqqqivLw8oVN1n3LKGHPnnbd1TuQ+U50yJCQhIqWsB0e3U2ngvEiZEKlc9lDdTfwGdUq1AQuj6eaQKV0hUpnf5yJTMj5ERCR9+HXNAI0Rkc4GDBi4DmDRooX577//bqsrr/zXNpMmPfdDjx49ywPrzJ07J3/06KO2v+yyq357/vmnO/zxx+/N/vjj92ZDh+7UFuDcc/8x56ijjllRVVXFPffc0fm9995uu2bN6rwOHTqWHXvsCYsOPfSIlYFtXXrphT3nzZvb9Pjjxy567LGHuixevKjgxhv/++uuu+6+9q233ih66qnHO82fP69pfn5B9Tbb9F57ySWXzevWrUdtWX74YXrTW265ocf8+XObdurUuezss8+ft+uuu69NxLEKV75Zs2yTBx+8r7O1P7VYu3ZtTnFx+/IRI0YuGzv2pKXZ2ZvyD1auXJFz5523df366y9brVu3Lrddu3blI0cesnT8+FOWBtapqqrOuu22G7u8++6UdpDF7rsPXXXRRZfOLygoqAmss2DBvPw77rit64wZ37WsqKjIMqbv2vPPv3he7959ou5646WAhEQUj4ZqpAaMd7lfIzPVGsteocqeCsGJWBrsqda4j6bRm84NY+9nGkj6ZyUUv89F8HgkqVp2ERHJHKG6ZkhsqktLs8vefLVN1YIFBTldu5YVHHjIyuzCwqi6S8Tb/PlzCwDatm1bsdde+6wuKmpd8corL7Y755zzFwbWeeWVF9oVFras/Mtf9i/p2bPXxssuu3jrDh06lo0ff/IigB49epUB3HnnrV1efvmFDkcffdyi/v23W/fhh++3vuWW63tlZWUxatThtUGJZcuW5j/00H1dR48eu7Bt2+KKbt26l7344rNtbrvtpl677z505QknjF9YU1PDN9983XLFihW5gYBEeXl59nXXXdXr8MOPWtK2bbuKiRMf7XzVVZdu/cILb/zQrFmzBj1+kcq3ePHivG7dum3cf/8RK5o3b15t7S9Nn3zy8c5lZWXZp59+1mKADRs2ZJ155slmzZrVecceO2Zhr15bbZw/f27BggXzm3j39dJLz3XYfvsBpRdffNkfs2b92vSJJx7r2qFDx7JTTz1zCcCqVStzzjzzFFNYWFh11lnnzW3atGn1k08+0fGCC84yzzzzyg9Nmzat8atDOApISETxaGRHyjLwLvebWjN4tGQ/yWoYhSp7KjTwozlhp8IMEKEa55GOWyaM4xGQ7M9KKJE+Q6kWhBMRkczid52Szuf/ZCr/8vMWay67uDc1NVBWlk1BQfW6hx/o1vKaG2fl77JbQu70V1ZWAjBnzh8FN998ffcmTZpU7777sNKcnBz22We/5R988G7bs88+b2FWVhY1NTV88MG7bYcP32dFbm4uxvTd2KRJk+pWrVpVDh6887rANletWpnz2muvtD/yyGMW/e1v5y4C2Guvv6xZvnxZ3hNPPNbZG5BYu3Zt7s033/nr9tsP2ABQXV3No48+1HXIkF1Kbrrp9j8C6+2//wGrveUuLy/P/tvf/j5/6NA9SwGKiztUnHHGif2+/PJ/Lfbee981oepbVVVFTc2mNnpNDdTUVNceB4Ds7Gy8mQxe0ZRv2LDhpcOGDS91tl/DzjvvVrpx48bst99+szgQkHjppefa/vnngqb33vvIT4G6A6XB+ysubl9+7bU3zwkcw5kzf2jx2WeftA4EJCZOfLRDWVlZzsSJT/3UunWbKoDBg3dee/TRh2z//PNPtzvhhPExD4ipgIREFEsjOx5BAb+pNUONlhwqfS9Sw8ivATys43A+WfxR7XN1mYUiVHAi3H7DHatIxzPS8mhO2IkMnIQqbyIa5/HMSIjHttJlKtNIn6FUC8KJiEj6C9c9Q+qmurQ0e81lF/dm48ZNLd+ysmyANZdd3LvNS29Oz27RokHv9K9duzZ3r712HRz4u23bduX/+tcVv3Xs2KkC4LDDjlz+4ovPdvrf/z4p3GOPPUs/++yTwuXLl+ePGnXY8nDb/eWXn5uWl5dl77//iFXe5/fee99Vt99+c8/ly5fltmtXXAnQunWbCk+DnNmzf21SUrIq78ADDw67j9zc3Jrddx9a24A3ZtuNAEuWLMkP97ojjhi5/fLlyzdb55dffmrx3HNP1844cvTRxy3yZoV4RVO+jRs3Zj344D2dpk79oM2KFcvzq6qqasfFqKysJDc3l+++m9ayR4+e671197PjjjttFlzp3r3Hxt9+m9088Pf3309rOWDAwDWFhS2rAkGVwsLCql69tl5v7c/NAQUkGoOGygQIN95AqOyF4EZ7vE8akUZLDpW+F3h+bUUpLfIKt2g4+jWA7epfWFOxerPn6jMLhV9DLlTD2xtA8StnNPWo693pRKY9hipvPBvn3s9lYJ+h3vPgckQ6bn4ZO95ted/vWINFwZ+VUK9PtSlbIwXhGpK6i4iIZCZ1z4i/sjdfbYPnTv1mamooe+PVNk3/77iwjfL6atq0WdXNN9/+a1ZWFsXF7Ss6depckZW1aUzJnj17lfft27/09ddfbbvHHnuWvvHGK+223nqbddtu229juO0uX740D6C4uH2F9/m2bdtWAKxatSonEJBo1arVZuusWrUq1++1wZo0aVLlzWLIz8+vASgvLws7KOZ1190y2ztw5i23XN9j6617rz/ssCNrG+4dOnQMue9oynfbbTd2fe+9d9odc8zxC7fdtt/6li1bVk2d+kHR888/3Wnjxo3ZLVq0qF6zZk1u69ZtwtYRoLCwsNL7d25uXk1Fxabyl5aW5s6ePau5N7AUkJW1/RYZF9FQQCINNVSkOJrt+mUvlKwvqW0s+3W3iFe2hJ/ggEVww2htRWnIRqj3f2CLDIncvJzaRm28hGp4h2ssR1uPun4uYkl7rO97Gq684RrnsZQtECyorKgCwr/n0QR9/DI5/DJ2gi+U6vp+ROqmFI8pWxuqIZ/osSXUXUREJHN4zxuhru+k7qoWLCgIZERsoawsu+rPBQUNXYacnOyagQN3XB9unQMPPHj5nXfe1mPhwj///PLLL4pOOunUBZG2266d01hfvnxZXps2basCz69YsSIPoHXr1rXPeQMg7rJKgGXLnKBGvPXrt91mGQlNmzatbtu2XUWk4xBL+f73v09aH3jgwUsD3SoAPvnko81mLmnZsmXlokUL6/0et2jRorJr1yEbxo8/ZVHwsubNW1T5vSYSBSTSUKhIcX3TyGONQI/pfSK5eTmUrC/xbVgGAhaB5xtCqJNU4PngYxIod6i70wd1P7T2ueBZKOrDWw6//YZqLMdSj8CyYR2Hc/FX5zVIw6y+wbCGvJvuDRbs1mn3zYJJ4d7zSEEf72tCZez4NcLrWr9wQY/A9vyCZbEEABLR9SqWQE8kocoTKStKgQkRkdSmmTMSJ6dr1zIKCqp9gxIFBdU5XbrWaYaEeBsxYuSqe++9o/u//33xVjU11VkjR45a6V2em5tbU15esVkdtt2274b8/ILqd96Z0rpPn21rG8pTp77fumPHTmWB7Ag/22zTZ2Pr1m0q3nrrjbb77Tdidaj1kiWa8pWXl2fn5+fVpr9UVVXx8ccftvGus+OOO6259947e8yc+WPT/v23C9ttI5wddhhU+umnH7c2pu+Gugxg6UcBiTQUqhFe3374sUagA1NHfvbHl1s0liE1UuwipcUnSqQGYKRyRlOPwDrxCARF0wCsr3jf8fCWbY9eu9QGk6IdGDNU0CeWMsfjYipSNyXvlK2R+thGeh+9watA+ePVXzeWQI9XpK5gwZkQ4QJLupgVEUlt6pqROAUHHrJy3cMPdPNdmJVFwchDVvouS7AmTZrU7Lnn3iunTHmjeI89hq0sKira7K57ly7dNn7//bctp059v2VRUevK7t17lLVp07bq4INHLX3uuac65eTk1PTr13/91KnvF33//betLrro0t/D7S87O5uTTjptwS23XN/r4ovP77Xffn9dCVlMm/ZV4V//euDKaDMZGko05RswYOCaN998rbhr124bW7YsqnrppeeKKysrNwvaHHbYUSteffXl9hdeeG6f448fs7Bnz14b//xzQf68efOa/OMfl/wZbXnGjj1pydSpH7Q544wTzaGHHrm0Q4cO5cuXL8/7/vtvCwcMGLjWO4BotBSQSBPR3LGMZz/8WETKUpDEnmTjsa9QDdJUfk/rW7ZQQZ9YsgXicez96hHNYKB+QYZIDXlv8ArCd2+paz1CBXpCZXQFyhyqK1ikz6bf/kREJHWoa0ZyZBcWVre85sZZwbNskJVFy2tunNXQA1rGYs899141ZcobxSNHjtpiTIuTTz594fXXX51/7bVXbrVhw4acc8/9x5yjjjpmxTnnXPBnTk5OzVtvvdb+6acn5Xbo0LHsggsu+eOQQw5b5bcPr0MPPWJlQUFB9eTJEztdc82VWxcUFFT37t1nbdu2bUNmViRSpPJddNGl86677qoe9957Z4/8/Pzqvffed8XQocNL7rnn9h6BbTRp0qTmnnsetLfffkvXSZMmdt6wYUNOcXFx2ciRh8Q0CGXbtu0qH3jgsV/uuef2Lg8+eE+39evX5xQVFVVsu23/tcZsW6fgTVZNqMFNGrmKiqqaeKTrFxU1i0vaf6DhMKTdLjENnOfVEH2641W/VJTJdYPw9avrZyXcQIyJ7N8fz/cu1HcvkYLLEKif33H1rusNQgQa8t56xHPmkfrUCdiszE/OmUjJ+hJ+Xv1TvX7z6qu4uHAasFNDbT8e55nG/DuVCTK5fplcN8js+jVU3YLHSErWeTVV3rtYzjHffff9nK5dt47LgJPVa9dml73xapuqPxcU5HTpWlYw8pCVqRSMALj55uu6fvbZx61ffPHNH0JNhynpZcGC39oNGjSwp98yZUikiVB3LGPpC64B4CRake5SRLq7Dfg2hiH90uhTIYU0VBn83qdQd5v8tpHMLk3hxk0J1xVMd9BERNJTpDGSJDGyW7SobujZNOpq1qxfC2bPntX07bffLD7mmNELFYxoHBSQSBOhLsIjNZYiTYvp/VvBCYlWpFlB/D5j3v/TSSo0gGMpQ7p0oYp13BQREUk/4bpnpKuKH2ew/rGHaTb+ZPK2G5Ds4mSUm266pufs2bOa77TTziVjxpy4NNnlkcRQQCLN1XdazHS/ey3JEe7utt9nTI1LERGRxiETZ87wBiHWP/YwFV99wXqg1a13JrtoGeWhhx63yS6DJJ4CEhkumrul6Xz3WpIjllk/REREpPHIlAzJUEGIZuNPrv1fROpPAQlRw1FERERE6iwTZ84IFYTI226AMiNE4ijhAQljTA5wAzAOaAK8A5xmrfUdXMUYMwK4FdgK+A0431r7jrusDfAysK27rWXAY8C11toad51mwN3A4e4mXwDOstZuaIDqhRRpZPhQgwR6H6dzfzsRERERySzBM2dA5nTNSLEgRHVNTU1WVlaWpkeUtFNTU5MFhJzJJRkZEpcAo4BdgBXAo8ATwAHBKxpjtgJeBE4FngWOAl4yxvS31s4B1gFnAL9aayuMMb2AN4GlwIPuZu7ACVgYoAYngHGb+7qE8RtEcljH4Xyy+KMt+tgBvo/T9cddRERERDKD3xgR6T5zRiAQUb22lKqfZtaOD5EqmRBZWVmLy8s3tiooaJrQG6oi8VBevrFJVlbW4lDLkxGQOBW42lr7O4Ax5iJgtjGmh7V2btC6Y4Fp1tpJ7t+TjTGnu89fZa0tA2YGvaYaJ/iAMaYpMBo4yFq7xH3u38BrxpjzrLUbG6B+vvwGkbSrf2FNxeotlge/JvixiIiIiEgyhBojIt0yef3GiMjp25+8nXdNufEhqqoqr1qxYsndbdt2ID+/yUZlSkg6qKmpySov39hkxYol+VVVlVeFWi+hAQljTBHQHZgWeM5a+5sxZg2wAxAckNjBu67rW/d573ZfB/6C021jPvBAYJH73LSg1zcF+gAz6l6b2PgNIunNkAg3SKAyI0REREQkWTJljIiKH2fw56THyB89PuwYEalm8ODBb0+bNu2s5csXXVFTU9MRyE52mUSiUJ2VlbW4qqryqsGDB78daqVEZ0gUuv+vDnq+BGgZYn2/dft7n7DWHuSOTTEEOBgIjEfht7/AY7/91crJyaKoqFm4VaKSk5O9xXb2KNqFPXrtAsDoAcfVex/J5Fe/TJHJdYPMrl8m1w1UPxERaXiZOH1nIAhRWVGVamNEROQ26EI26kTSVaIDEqXu/62Cni8C1oRYP6p1rbVVwBfGmD2Be4Bjg/ZXErRvv/3VqqqqoaRkfbhVolJU1Cwu20lVmVy/TK4bZHb96lI3b+omUPs4Fe+UZPJ7B6lTv+LiwsgriYhkqEycvrPZ+JMpz8shf/T4tAhCiDQGCQ1IWGtLjDHzgB2B76F24MqW+HefmA7sHfTcIOD9MLvJBXoHdglsdPf3gef1G4BfY6+BiGQqb+omUPtYFysiItJYzFz1A09+N5Hjeo5N+64ZweNDBM7pxfc/kBJBbxFxJGNQyweBi40xH+LMsnEj8LY7a0awx4ELjTHHAs8DRwKDgTEAxphdgebA/4ByYA/gXHcfWGs3GGMmAVcbY350t3k18HgiB7QUkdTnTd0ENnssIiKSyYKn76ysqErLrhl+s2UEn99FJLUkY0CUG4DXgK+BP4EcnJkwMMYcb4xZG1jRWvsbcDhwGU4Xi8uAwzzBi3zgJpxpPlfhDGZ5J3ClZ39/x8mGCPyzwHkNUTERSV+B1M287QZs9lhERCTTebtn7NZp97TqmlHx4wxWX3BObTCi4qsvoIba2TJ0ThdJbQnPkHDHeviH+y942WRgctBzU4ApIbb1MU7GRLj9rQNOdP+JiIiIiDR6oWbO2KPXLinfpSFUl4xUny1DRLaUjC4bIiIiIiKSYJkyc0a4IITGfhJJLwpIiIiIiIg0Auk8c0bwbBkKQohkBgUkREREREQyVKiuGekyc4bfQJWtbr1TQQiRDKGAhIiISD0ZY64FjgPa4kw3/TFwvrV2XlILJiKNUrp3zfAbIyKnb//agSpFJHMoINGAKn6cwZ+THiN/9HgNrCMZwXuBoM+0yGaeAG6y1q42xjQDrgGeBnZPbrFEpDFKx64ZGqhSpHFSQKIBBX5MKyuqlFYmGcF7gaDPtMgm1tpfPH9mAdWASVJxRKSRCmRGDOs4HEivrhkaqFKkcVJAogE1G38y5Xk55I8en+yiJJ3urKevUINIicjmjDHHAfcBLYFK4PzklkhEGoNw3TNSnQaqFBEFJBrQr12yePL/cjiuSxb9I6yb6Q32TL6z3tjeu0x7/xqLTP+cBuqXP3xvyj/6MCn1tNY+CTxpjOkInAT8EG79nJwsioqa1WufOTnZ9d5GKlP90lcm1w1Sq35PfjeRr5d/SW5eDmcOOpMHf8zh1O1Oq3P5GrpuG77/npX330eb089g3aTHqPjqC8rzcuhy/wMwdNcG229AKr13IqKARIMKRKkrK6oiRqkzucEOZPSddb136S3TG+oBW6TCZlida7vI2Z+pWb06qd9Ha+1iY8xDwO/GmO7W2pV+61VV1VBSsr5e+yoqalbvbaQy1S99ZXLdIPn182ZFHNdzLJUVVRzXcyw98npz7aBbAOpcvoaqW/BsGUsrqmg2/mQqK6rIHz0+Yccz2e9dQHFxYbKLIJISFJBoQGN6n0huXg7H9Rwbcd1MbPQFN/QysbEOm793mdi4zcT3LtTAWZlWTy/v5zRT6uyX6uvNkEiyXKA50BnwDUiIiMQiHWfOiDRbRiZeY4hIbBSQaED9W2/PPb3uDRmFzfQGezwaPenQwPe+d6svOCcjGnqZLtTAWZkm1G9MptQ5VHeipoccltByGGOygTOBZ621S40xXYG7gDnAL+FeKyISSSAQsbailJ9X/wSkz8wZmi1DRCLJTnYBGrPaH+nHHk52URpEs/EnbzFfdMWPM5xG+48zotpGKh8jv7r41TkTxfo+Jkuocnrfp0BDPRMvjkJ9f2Kpcyq/1yn2fTsQ+NEYsw74ElgP7GutrUxusUQk3XmzIYa022WzmTP6t94+yaXbkve80VjOtyJSd8qQSKJE3KVMZoaBX9ZHrFkTqXwn168umZjp4iddUv7DjZsQbbn9vkPpkLkD8fn+xPO9rutxm7FwDQ99PpdTdusBUPt4QIp836y11TgBCRGRuPB2z/BmQ6RiAALCd4VMhd9pEUldCkgkUUM2XoMHDkqVhmOs4y001DGKR4MyUmOvrvtIh8ZuKgeKvOIxboJfUCPW71Ui39O6dgULVcZ4fs7r+h489PlcvpizqvbvwONTduuxKTjRuWXU2xMRSUXpPEaE97yYLtcIIpIaFJBIQfFovPgNHJQKGnK8hUQ0jLwiNfbqOqtBOsyG4K2797gDKVXeeIyb4BfUiPV7lYiMkroEIaMZ3DOWz3mk/UXzHlT8OIM/Jz1G9u571g5OecpuPQFqMyQCj72BiruOSM27hiIi0fIGIVJ5jIjGPlClN2tPwXCR+lNAIgX5NUi9I8dH09DzGzgoEXdpY9lHvGeniHfDqL7qenc+nrMhJOI995YRSNmuHHW9QAoV1IjleNYlyyDW964uwZJQg4011Pc41Hvgd3FbNeMHckrXsKSskgF337tZwCHwOBCg8AYqdKEoIukkXNeMVM2KaIwDVXrPLQqGi8SXAhIpyK9BWml/pmb16rANvUip2vG+Sxu4k5k/enztSSiWfcSSLRFNAymWIEMiovex3J2Px2wIfscoEdkWwWWsb6CnvkEUv89lvMQjqOEnUrcQ73sH1NYv8Npm40+u00Vh8GsifR/93ptQr/MLpv7cpmftBV3erz+x+uEHaHXyaRQ+/zhNpn/DkrJKSo8cw5qla/m2+w70+uUbppv9uDhE+Qd0bll7MRi4WCzdWMnMxaWALhRFJDWlc9cMv/NNqt2AaAjeIIRfMFxE6k4BiSSI5S5i4Effe1EfSqRgQKjGbaTyhFpeGyypqIprWnxd6gb1DzIEdz2IZ6M2ULbAyNOhjmWs6fJeftuIJtsiUO9Ys3BCldGvK0ekz1Wg/N5yri0tJbuwMGx5/LYRaMh7P5cNKVQ9I2U9BMobfHHnl+ngl4VSWVFV+7iuA4eF+nyF+j5Gem8iBVMf2v2U2gu64196gG3mzmT2ww/w6i6HsMOiUqab/ViwojlfDBxH/46F/LTjPpyyW4+ougUFLhb7dyxk156tdaEoIikrnbtmNKaBKr1ZEd4ghDcYLiL1p4BEEtQ1i6DpIYcBdR98LlS/f79GRqCc4RqyzcafTHleDtm777lZI7u+d5D96lfXdPJoGjJ+fe+BLYIt8RDuWNZl4EC/Oxb5w/f2fT8iNTK9Dcd4ZFP41TXUxQ2wRfpn9dpS3+yOwLZDbSOnb3+a7rFHbQZBQwuVheJX/3DdW8J1Cwl+78rzcjZlSBB7ADCSSIEK73sTKoDmF0w9pc2mC7q8tqcx282QOKBPPx7qvE3tBV9ebjbjhnSr7XKx+tbI3YKCLxZFRFJNIDNiWMfhQGp2zfBmGYbqmpHJwnXNUBBCpGEoIJEgoVLd6iJcGn5d7qb7NTJgywaiXzCg+P4HmHvSyXHtCuLXkIuUFh5LkCG4nKH63gcaffEctDHcsazLAJl+dyxCpdsHB30CFxx+Dcd4dO/xq2u4UbiD0z9jCV4Eb6N46K6UlKwPLlKDCJWF4vc9D1VeL7/PQvBzxfc/UFu/wPOJGDvB772JtC5sCqYOwNONovOusNeutet7L/QeHTtks/cvmuOmO1YikorCdc9IJcHn6cqKqkbVNcOv25+6ZogkRlZNTU2yy5CSKiqqauLRoCkqakZJyfraRmLezrvW+wfdr6EW63ZjSSf3NryD61FU1Ixln34R17EJYkn1D1X/QDlz+vb3zfqIlFoP/u8dELf3MZbPRKg6+wVkoskgibTveA+G6fd+1HUqVIgcFAq8d4mWqOk9/ep39gs/8MWcVezas3XaN8yT9f4FKy4unAbs1FDbj8d5JlWOVUNR/dJXJtcNoq/fxV+dx9fLv2RIu11qgxKBzIhU4j1P57cpapBxmFJF4L3zy4bo37GQwia5Ccm2a+hzjEi6UEAihHgHJBqqoZKoBlCo/SX7giOW/vux8nvvIH7TWta1jPGoW8WPMyhvoIEfQ+0vkZ/TZH8uG5pf/QIXVnv3bseHs5anddeFVHn/FJBIPtUvfWVy3SB8/bxZEUDKBiFCXd8kMsswGQLvnTeQHwhKJPLcqYCEiEMBiRDiHZDIVJlcv0yuG2R2/TK5bhC+fvHIlIhH94/6BEhS5f1TQCL5VL/0lcl1gy3r59c1Y0i7XVK2a0a4LNNMfu9mLFzDhK/nM25IN4CkThOtgISIQ2NIiEjG8jasIbkXHokSjz6voeZY9wsyBNYPfhzYxs+LS1m9sXKLbYmIZJJUnjkj1JhMjXmgyorKag1UKZIiFJAQkYzlbVgDvo1sP4kYIDIe/MoZj8EdvUENvws5b5AB8D3GgW0EBy9ERDJJqs6cEU0QIpMHqgw1W8Ypu/WonclJRFKDAhIiklFmLFzDhFdmMm5IN99sgWgaxqEyBBpKXQMgkcoZKUMk3PLA9gJdQCB8kCH4sXcbhw/oFHWdRERS3cxVP/DkdxM5rufYlJ05ozEGISDybBkDOrfcYiYnEUkuBSREJKOES8cMF1zwNs4TPdVXrIGFcAGXUNuFzbMXgi/YvMu9ZQi+kPMLMkR7jEVE0pXfGBGVFVUp2z0jf/jeQOMKQgTPlhEYrFLTQouktoQHJIwxOcANwDigCfAOcJq1dnmI9UcAtwJbAb8B51tr33GX9QGuA3YDWgLzgP9aax/2vH6qu7zCs9ljrLWvx7ViIpIS6pqOGRwUqO/FSyxZD35dJLxZCH6BhWj6v4bKEPG7YAt+TYAu5EREthwjIjcvh+N6jk3Z7hlARgchvIK7ZAT+T+UulyKySTIyJC4BRgG7ACuAR4EngAOCVzTGbAW8CJwKPAscBbxkjOlvrZ0DtAY+BM4BFgF7AK8bY1Zaa1/0bOo/1tprGqxGIpIy6pqOGU1WRF0GdQxsM9zr/LpIeMdpCC5btAGX4GBC4HGoCzYFHkRENvFmRXgzIfq33p57et2bMtOeN/aBKkNl8olIekhGQOJU4Gpr7e8AxpiLgNnGmB7W2rlB644FpllrJ7l/TzbGnO4+f5W19kvgS8/6nxpjpgB74QQyRESiEuoipr6DOkbzOr8uEsHTZXrXqW//V12wiYj48+uaAc74EKmYCdGYxogINVClZssQSW8JDUgYY4qA7sC0wHPW2t+MMWuAHYDggMQO3nVd37rP+22/GbAr8J+gRX83xpyPk0UxCbjFWlsR/HoRaRximQ7ULxU01kEdo30doMEgRUSSKBWn7wwEIqrXllL108xGFYSAyANVikh6S3SGRKH7/+qg50twxoDwW99v3f7BK7pjUzwBzAce9yz6J/ALsAYYAkx29/XPcAXNycmiqKhZuFWikpOTHZftpKpMrl8m1w0yu36R6jbhlZl8MWcVebnZALWPHx07ZIt1z9+vD3d9OJuz996GQd1bs2e/jgCcuOfWtesEngt+XPtcUbOIr4tFJr93kPn1E5HUk6rTd8KmGTNy+vYnb+ddG1UQQgNVimS+RAckAsO5twp6vggnYOC3fsR1jTF5OIGGTsAB3uwHa+3nnlW/MMZcjjOoZtiARFVVTVz6BhYVNcvoqYUyuX6ZXDfI7PpFqtu4Id2oqKyuHYch8NjvNb1aFnDbKCcGmirHK5PfO0id+hUXF0ZeSUTSVrjuGcnm7Z4RnA2RqUJ1ydBAlSKZLaEBCWttiTFmHrAj8D3UDlzZEpjh85LpwN5Bzw0C3g/8YYxpAjwPtAD2t9aujVCMaiCrLuUXkcwQarBHERFpPFKte0aoMSJa3XpnRmdDBIQLQug8LZK5kjGo5YPAxcaYD3Fm2bgReNudNSPY48CFxphjcYIORwKDgTEAxpgWwGs4U3oeYK3d4H2xO2bFUGAqsA4YCFwJPBPfKomIiIhIqgs3c0YyMiM0W4ZmyxBp7JIRkLgBZ7rOr4EC4F1gNIAx5njgAWttC6gd8PJw4Fac6UF/Bw7zBC+OwJlRYwOwzBgT2Mcka+3pQB5wGU53jmycQS0nA9c3aA1FREREJCWk2swZmi1Ds2WIyCYJD0hYa6uAf7j/gpdNxgkYeJ+bAkwJsa2JwMQw+1qGM+uGiGSgULNl7KkBEUVExJVqXTMaYxACNFuGiPhLRoaEiKSpwMVE8PSV4abNrOs+Qm0v1J0VoPZxXWevEBGRzJFqM2cEMiPyhzvDozWmIIRmyxCRUBSQEJGoBS4mfl5cyuqNlbXPeweh8stYCDwOFcjwPg7so3RjJYVNckMuD+zP+3/wYxERaVxSbeYMv+4ZQKMLQoBmyxARfwpIiEhYfgNOBQcWgLAZC4HHoQIZ3seBbZZurAy73O/Oiu6yiIg0bqnQPaMxDlQZbRBC52kRCaaAhIiEFWrAqcMHdKpdJ/BcpIyFUIEM7+PABUvw+BDBy0VERCD1Zs5ojGNEKAghInWVVVNTk+wypKSKiqqakpL19d5OUVEz4rGdVJXJ9cvkukH09Ys0pkMq0nuX3lKlfsXFhdOAnRpq+/E4z6TKsWooql/6aui6+XXNGNJul4QFIILr582KAGof5203ICHliadY3ju/saVS/VohVb53DX2OEUkXypAQkbB0d0NERFJNKnTNgE2BiOq1pVT9NJP1OONDZGomBISftlNEJFYKSIiIiIhIykuVrhkVP87gz0mPkT96fG33jJy+/cnbedeMHB8CohsjQkSkLhSQEBEREZGUFQhErK0o5efVPwHOrBnJmjkjEISorKjaYoyITKKBKkUkERSQEBEREZGU4jdGRN9W/RjSbpekds1oNv5kmo0/mfK8HPJHj8+4gSpnLFzDhFdmMm5INwUhRCQhFJAQERERkZQSaoyI/q0T1xAONX1nq1vvpPj+B1JiYMR4CwQhKiqrFYQQkYRQQEJEREREki4VxogIFYTwds3IRN7ZMvJysxk3pJuCECKSEApIiIiIiEhS+HXNgMSPEeE3W0bw+BCZ1DUDQo8R8cRJu2Rk9oeIpCYFJEREROrBGHMjcBDQDVgLvAFcbK1dmdSCiaSBZE7f6ZcN4Z0tozEFITRbhogkiwISIiIi9VMFjAZ+BIqAx4EJwCHJK5JIagtkRgzrOBxIzvSdobpkaLYMEZHEUUBCRESkHqy1//L8ucwYcwfwbLLKI5KqwnXPSJTg2TIytUuGghAiki4UkBAREYmvvwDTk10IkVSTrO4Z4WbLyKQgBGwKRJRurGTm4lJAQQgRSW0KSEgtbzR9QOeWyS6OiEjaMcYcAZwODI+0bk5OFkVFzeq1v5yc7HpvI5WpfukrULfpy6bz4I8PcOp2p3HmoDN58MccTt3uNHYo3oE9eu3SoGXY8P33rLz/PqrXrKHihx8oz8uh/dlnsfL+HNqcfgZN63HsU/W9m/DKTL6Ys4odurRk2DZtOXvvbRjUvTV79usY9TZStW7xkun1E0k3CkhILW9Kn6LnIiKxMcYcBTwAHGKt/TbS+lVVNfUeyb6oqFlGj4av+qWnmat+4Mk5Ezmu59jarIjKiipu3Pm/XDvoFoAGq3e4gSrzR4+nrGcfmt/wX8qAsnqUIZXeO+8NpXFDulFRWb3ZzaVYy5lKdWsIqVK/4uLCZBdBJCUoICG1NMKyiEjdGGPGA7cCB1trP0t2eUSSyRuESPTMGY19oMq7jtheN5VEJK0oICG11K9QRCR2xphzgCuAv1prv052eUSSwTtg5ZjeJ5Kbl8NxPccmZOYMDVSpG0oikr4UkBAREamfO4BK4ENjTO2T1toWSSuRSIIEAhFrK0r5efVPgDNrxj297m3QtHgNVKmBKkUkMyggISIiUg/W2qxkl0EkWQLdM/q26seQdrs0aNeMUEEIb1ZEpvDLhujfsZBde7ZWEEJEMooCEiIiIiISteDuGeCMEdG/dcM0kAOBiOq1pVT9NNN3fIhMyIqIpkuGZkETkUyjgISIiIiIhOUNQgSyIsDpntEQY0SEmy0jk4IQXuGCEMqGEJFMpYCEiIiIiITlDUIkYuaMxjhbhoIQItIYKSAhkmG8FzdK7RQRkboK1zWjobMiMrFLRoCm7BQR2UQBCZEME5zyqeCEiIhEKxW6ZjTW2TJERBojBSREMoz34ib4zouIiEg4ieqaEWmgykyh2TJERMJLeEDCGJMD3ACMA5oA7wCnWWuXh1h/BHArsBXwG3C+tfYdd1kf4DpgN6AlMA/4r7X2Yc/rmwF3A4e7T70AnGWt3RD3yomkAO/FjTc4oa4cIiISSiAzYljH4UDDds0AMnqgSs2WISISvWRkSFwCjAJ2AVYAjwJPAAcEr2iM2Qp4ETgVeBY4CnjJGNPfWjsHaA18CJwDLAL2AF43xqy01r7obuYOYFvAADXAy8BtwBkNUz2R1OENTpz9wg/KlhARkVrhumc0hHBjRKS7aIMQOv+KiGwuGQGJU4GrrbW/AxhjLgJmG2N6WGvnBq07FphmrZ3k/j3ZGHO6+/xV1tovgS89639qjJkC7AW8aIxpCowGDrLWLnH392/gNWPMedbajQ1UR5GEi5QBoX6qIiLilYjuGZk+RkSkcSEUhBARCS+hAQljTBHQHZgWeM5a+5sxZg2wAxAckNjBu67rW/d5v+03A3YF/hN4CqdbiHcb3wJNgT7AjFBlzcnJoqioWfgKRSEnJzsu20lVDV2/7+at4q4PZ3P23tswqHvrBtuPn3R77ya8MpMv5qwiLzebR8cO2WL5nkXN2LNfR8A5rne/+hNn7bU1AHd9OJu/9uvA2z8t4ey9t6l9LhnHPR7S7b2LleonInWViJkzKn6cwZ+THiN/9PiQ03dmCo0LISJSP4nOkCh0/18d9HwJzhgQfuv7rds/eEV3bIongPnA42H2F3gctvNeVVUNJSXrw60SlaKiZnHZTqpq6Prd9u6vfDFnFSvXllPYJDeh/S7T7b0bN6QbFZXVjBvSLWK5A8e1vKIKgC/mrGLGgtWs3lhJRWV17XMVldVpeVGVbu9drFS/xCguLoy8kkgaSMTMGX6ZEJUVVRk5feeMhWuY8MpMxg3ppnEhRETqKdEBiVL3/1ZBzxcBa0KsH3FdY0weMBnoBBxgra3w2V9J0L799icpJnCiL91YmRHjH3i7VQARB5kMrL9373Z8OGv5Fq8L3ka0x+aU3XqQl5vNuCHdap8L3kdgPRERSU+BQMTailJ+Xv0TEP+uGaFmyyjPyyF/9PiMCkIEjxERCNqn83WJiEiyJTQgYa0tMcbMA3YEvofagStb4t99Yjqwd9Bzg4D3A38YY5oAzwMtgP2ttWu9uwQ2uvv7wPP6DcCv9ayOJECgkR3ckE8loYIMfo+9fUyBzQa98ntd4KLn58WlrN5YucXrvI9juSAa0Lklj44dUnsXOvDawwd0ql1HF1giIuktkA3Rt1U/hrTbpUFmzgg1W0bx/Q+kRKZTfYQbqDI4qC8iInWTjEEtHwQuNsZ8iDPLxo3A2+6sGcEeBy40xhyLE3Q4EhgMjAEwxrQAXgMqcDIjNpvK01q7wRgzCbjaGPOj+/TVwOMa0DK9eO/+J3P6Sr/gQ6ggg99jbx/TgOALHe/rAutFyl5IxUCNiIgkXrgxIuJFs2VsHtQXEZG6S0ZA4gac6Tq/BgqAd3FmwsAYczzwgLW2BdQOeHk4cCvO9KC/A4d5ghdH4MyosQFYZowJ7GOStfZ09/HfgbvYlBHxAnBew1RN4iFSwCH4AqEuwYlI+/D2Dw3sM1TgIFSQIdRj7/4CQRa/GTCCu2GEyl5QJoOISOOWrDEiNFuGiIjUV8IDEtbaKuAf7r/gZZNxxoLwPjcFmBJiWxOBiRH2tw440f0nacDb6Pc7+XsvEEIFJwLbCTX2gt/rvOt6+4fClhkL0QQZwj0OFnyho4seERGJVkNN3xkqCJEps2X4ZUNotgwRkcRKRoaENGLRjLfg1+j38l4ghApOAGHHXvB7nXddv/6hfhcnulAREZFkaKjpO6MJQqTzQJXRdskQEZHEUEBCfEUzG0SgW8PQXm0izgAReBzNeAuxjFgdKjgREGrsBb/XedcNNeijiIhIsiSia0ZjDkLoXC8ikngKSIivcIMsBp/UZyxYHXEGiMDjWMZbiFWo7IVIM0eEGqdBREQklTRU1wzYlBmRP9yZ3CzdgxBeCkKIiKQuBSQSJJkzQ9RFqLES/Ka9Cs6Q8Hud93E04y2IiIiII5AZMazjcCA+XTPAv3sGkBFBCNh07bV373aAghAiIqlIAYkEiTRQY6oJlW0QatqraGaASId6i4iIpIJw3TPqozEOVAm6BhERSVUKSCRIpIEa04XuLIiIiDS8eHbP0ECV6X3tJSKSyRSQSBA15EVERCSceM+cEQhEVK8tpeqnmY0uCKFrLxGR1KeAhIiIiEiSNOTMGYFsiJy+/cnbeVcFIUTqKHfxNJp9dRvrdz4foPZxZcfBSS6ZSPpTQCIJEjnAZah9RTOtp4iIiDSseM+c4e2eEZwNka4C1yzeqcMVhJCGECrw0Oyr2yiY/1HteoHHaw6ZnJRyimQSBSSSIBDVL91YSWGT3LgFAvyCDMEnb787C4AGfRIREUmgmat+4MnvJsZl5oxQY0S0uvXOjMqG8E4driCE1Jdf8CG7fA15S76rXScQeAisE/g/+LGI1J0CEkkQCBiUbqz0DQTEkkERKn0R2OLkHWmgJw36JCIi0nD8umdUVlTFNQiRzrNlRNslQyQWwYGHnLfuIHfQub5ZD+XtB1LWbfgWgYfKjoM3y4ZQZoRI/Cgg0YBmLFzDhFdmMm5It81OoIGovvfEG+4kHJhD+8NZy7foYhFNkCGw73DpjbrLICIi0rCCu2fk5uVwXM+xUb8+E2fL0LgQEk+B4EPZ1gdS8Nubvt0tsud/RLOKqpBZD95xIRR4EGl4Ckg0oMCJtaKy2vdE6j3Bnv3CD74n4cA2fl5cyuqNlbWvjeaEHbxPndBFREQSK9zMGff0upeSkvVRbytTghCgcSGkfiKN9ZC7dAY5Zc61cnDgIS8vh/WDzlXWg0iKUECiAZ2yWw/ycrMZN6RbVOsG/veehAPPB2dI+K0rIiIiyRfvmTMCmRH5w/cG0j8IoXEhJBbRZD0Ej/XgXTc48FB17PNUxhAIFJGGpYBEAxrQuSWPjh0S1d2PUCdh7/OHD+hU+7xO2CIiIqkpHjNn+HXPADIiCAEaF0K2VJ+sh8Bjb/ChrP/xCa6BiNSFAhIiIiIi9RSua0a0WRGZMlCldwwtjQshwWKdWjOarAd1txBJXwpIiIiIiNRRIBCxtqKUn1f/BMTWNSNTBqr0y4SoqKxWEKKRi8fUmsp6EMlsCkiIiIiI1FGge0bfVv0Y0m6XmLtmpHMQAkIPThkYQ0tBiMYhlqwHTa0pIl4KSIiIiIjEIFz3jGhV/DiDPyc9lpYDVUYzOGW0Y2hJ+olX1kOAAg8ijZsCEiIiIiIRxGPmDL/uGXkVVWkbhAANTplpchdPI+etO8gddC6Ash5EpMEpICEiIlIPxphjgL8BOwDNrLU6t2agus6cEW6MiPK8HPJHj2/IYtdLtEEIdclIT6EyHbKXfEeziiqAsINMKutBROJBF00iIiL1swq4F2gKPJjkskgc1XXmjGgHqiy+/4GU69agIETmiXV8h+qt9mG9myEB4QeZBAUfRKR+FJAQERGpB2vt2wDGmL2SXBSJk/rOnJGOA1WGGpwy8L+CEOkhEHzwTpEZaTrN4IBDi22HUekGyhR4EJGGpoCEiIiIiEddZs7wZkWkchDCmwEBRBycUkGI1BQp6yF36Qxyypzslmim0wQFHEQkORSQEBERSYKcnCyKiprVcxvZ9d5GKktk/aYvm86DPz7AqdudxpmDzuTBH3M4dbvT2KF4h7Cv2/D996y8/z6q16yh4ocfKM/Locv9D8DQXSPuM1H1+27eKu76cDZn770NE76ezxdzVpGXmw1Q+/j8/frUrjOoe+t671OfzfjJWvAV2Z/cRLU5iGz7OtXDLiL7uzvInv8ReXk5ALWPq/f5J9Wf5FBjDqLavk7OsIto0XVn2HYYLQIb9D72ofdORBJJAQkREZEkqKqqqff4AUVFzVJuDIJ4auj6+c2cUVlRxY07/5drB90C4Lt/vzEicvr2J2/nXckfPT7qMjd0/YK7YVRUVnPKbj2oqKxm3JBuTl3cx71aFnDbqP6Af51jpc9m7MJlPeTO/4iaP78ju2wVFRVVlO58Ps0qqmrHegg8rmyxHRzwuLPBrY52/o+xnHrvEqO4uDDZRRBJCQpIiIiISKMUy8wZ0Q5UmWx+g1KG64ahLhmJF8sgk7Cpq4V3XAh1txCRTJHwgIQxJge4ARgHNAHeAU6z1i4Psf4I4FZgK+A34Hxr7Tue5Q8DuwEGmGCtPTno9ROA44Eyz9MXWWvvjVOVRESkEXPPa3lAvvt3E3dRmbW2JmkFE1+xzJwRbRAi2WNERDszhiReqKk185Z8V7tONNNpBgIOZf2PT1TRRUQSIhkZEpcAo4BdgBXAo8ATwAHBKxpjtgJeBE4FngWOAl4yxvS31s5xV5sBPAecFmafE4MDFSIiInFyAvCY5+8N7v+9gDkJL41swa9rBkSeOSMTghDKgEiMWKfWLOs2XINMioiQnIDEqcDV1trfAYwxFwGzjTE9rLVzg9YdC0yz1k5y/55sjDndff4qAGvtne52jk1I6UVERDystROACUkuhoQRS9cM2JQZkT98b0BBCNlc7uJp5Lx1B7mB8RvqmPUQoMCDiDRmCQ1IGGOKgO7AtMBz1trfjDFrgB2A4IDEDt51Xd+6z8fiCGPM4cBy4BXgKmvt2hi3ISIiImkkkBkxrONwIHTXDPDvngEkPQgBWw5OCQpCJEK4rIfs+R/RrKIKUNaDiEh9JDpDIjCc7Oqg50sAv86NhSHW7R/DPu8CLgaWAX1x0mofAsJmVMRjOrasBV+RM+VmWg+9kJquO9drW6kqk6dOimfdaqfsGnZRynwW9N6lL9VPJLRw3TOCRTNGRLLEOjil1E8g+OAdODLcIJN5eTm1M1wEnlPWg4hI7BIdkCh1/28V9HwRsCbE+tGu68ta682wmGmMOQ+YaowZZ60tC/W6eEzH1vKD68md/xFV5ZUZe2JKlamTGkI86xb4LFRUVKXMZ0HvXfpS/RJDU7Klp0jdM1JtoEpv4AHQ4JQNLNJYD7lLZ5BT5hz3cN0tqo59nkr3dypVzusiIukooQEJa22JMWYesCPwPdQOXNkSZ3DKYNOBvYOeGwS8X49iVLv/Z9VjG1Hxi6BL4+R3USMiIvERy8wZqRCEACcQMeGVmaxcW17bDQPQuBBxFGvWA2hqTRGRREvGoJYPAhcbYz7EmWXjRuBtz6wZXo8DF7oDVj4PHAkMBsYEVjDG5APZQA5Q4063Vm2tLXeXHwNMcYMhvXGmEH3VWruxoSoYEBxBl8Yr+KJGRETqJ5aZM7xZEckOQoTrhhGgIERs4pX1oKk1RUQSLxkBiRuA1sDXQAHwLjAawBhzPPCAtbYF1A54eThOEOFR4HfgsKDgxTvAcM/f44CPgL3cv08H7jXGFABLgZeAK+NfLREREUmUaGbOCAQiqteWUvXTTNbjDFKZ7CAEOEGHvNxsxg3ptlk3DAUhQotlak1Q1oOISDpIeEDCWlsF/MP9F7xsMjA56LkpwJQw29srwv7CLhcREZH0EWnmDL8xInL69idv510TNkhltNNzPjp2SEqMl5IqQgUcAo/rMrWmsh5ERFJbMjIkRERERKIWaeaMaAeqbEjRBiGUAeGIZXyHwGNNrSkiknkUkBAREZGUM3PVDzz53USO6zk2YveMZA5UGQhElG6srB2cUkGITeIxvoP3sabWFBHJLApIiIiISMoJBCEqK6p8Z86o+HEGq685J2EDVUaantM7OGVjDEIEBx5y3rqD3EHnxm18BwUfREQykwISIiIikhKCp+/MzcvhuJ5jfceISMRAlaG6YUDo6TkbA7+sh+DxHbLnf0SziiqN7yAiImEpICEiIiJJE258iHt63UtJyfqED1QZqRtGQKZnQ8Qyq0Xw+A55eTmsH3SuxncQEZGwFJAQERGRpAk1PkTFjzP4c9Jj5I8en5CBKv2yIcJ1w8i0IEQsg0yGy3oIqDr2eSo1g4iIiESggISIiIgkXKTpO1dfcw4VX31BZUVVg40REe3MGJkkHoNMKutBRETiRQEJERERSYhYpu9sNv5kyvNyyB89PilBiEzIgKhL1kO0g0yKiIjEgwISIiIikhB+3TNOqtqT1Recs9n4EIGBKovvf4CSOKT9Z3oQIt5ZDxpkUkREEkUBCREREWkwwTNngBOM6PNnDf96porqta9S4c6W4e2aEQ+RBqdMtyBELINMgrIeREQk9SkgISIiInEVqmvGNc3G869nqmg2vsZ3toy6ds3wZkAAMQ1OmaqimVozmkEmlfUgIiKpTAEJERERiatQM2esv+ahuM2WEaobBpBWg1PWZ2pN0CCTIiKS3hSQEBERkXqL1DWj2fgaiMNsGZG6YQSkSjZEqIADQM5bd9Bi3aqYsx4CFHgQEZF0p4CEiIiI1FkgELG2opSfV/8E+HfNCAxUWd8uGdF0w0hGECLW8R0Asud/RKWyHkREpBFTQEJERETqLNA9Y7+SHoz9oAVFJ+25WRCirgNVzli4hgmvzGTckG5hZ8ZIhniM7wCQl5fDukHnKutBREQaLQUkREREJCbe7hknVe3JES/OpGtNNfm/lpD3/IdbjA8RbVaEXyZERWV10mbGaOjxHaqOfZ7KOExrKiIikq4UkJCU4L3o894pEhGR1OANQrz37u3s+/oPvHfQGsZ92Zw2toScvl3IjmG2jEDwYe/e7fhw1vItBqc8Zbce5OVmM25It4QEIeI1q0WAMh1EREQiU0BCUoL3bpMu4kREUkOo6TvP+bSagj+g36fVNDs1+tky/DIgfl5cyuqNlcCW3TEeHTuEkjhnEGhWCxERkdShgIQkjfei0O9uk0hAuFHqlVkj0nC8QYhA14yik/ak/albs/6xh2kVRTZEqOk5A8EHb4ZEvDMhAr8dZVsfSMFvb4YdZFJZDyIiIomngIQklLdhGZwVoQs98Qr1WQF8H+vzI6DuX/ESyIw4oHTr2iBEh+c/pI11xojIu/WwegUhgoMPhw/oVK/yRsp6yF06g5wypwzhAg/KehAREUksBSQkobwNS2VFSDiRPiuhHkvjpu5fdec3RkSbqul0nrd2i4Eq/cQahKirumQ9eNdV4EFERCR1KCAhCeVtWAZfFAboDqdA+M+KGhMSzNtIBQWp6sI7UOWR7hgR5X2Kydt5u7ADVQYCEaUbK5m5uBSITxAid/E0ct66g9xB5wL1z3oo6398zGUQERGRhqWAhCRUqCCEl+5wCkT3WREJ0O9G3XizIo70DFTZ/tTzNhsjAjbPgAC2yIbo37GQXXu2jjkIEa67Rfb8j2hWUQUo60FERCQTKSAhCRFL1oO6cohIrPS7Eb1Q03eeGRSEaHXrnU4Q4oUftuiGAYTskhFOrFNr5uXlsN7NkAg8p6wHERGRzKGAhCRELHcvdWe8cVOXHakL/W5Ez69rRr9Pq8k7elOXjEjdMAJCZUPEa2rNqmOfp9Kd9lPvr4iISOZRQEISQncvJVpKvU8vyQwgKXgVvV8/f4mSR+6j6KQzQnbN8BuUMlw3jMBjTa0pIiIidaWAhCREfe9ehrrbFqoRooZK4kRzrCOt411e1+CVt1GUM+9tcgedu9m+Yv0MxZNfgy1TPpeJDiCFmzpYthQIROStL2Or+Rv4/ZH7qBp9NatXPkCrw09jRZuePLT7KZzSJvzMGKCpNUVERCT+FJBIkHRrIIcqb/AFaWAE9MA68W70BbYXro9xpLTgwAVvqr0H8WjIx7Mc3sYyEPX7GDw9Z13ej+DldWmkeBtF2WWraFZRtdl2wt2xrU/9vWJtsIXbR6htAb4zDwQeR1MPv3Wj+Xz5fVa8DU+/MsdSHm/9Crr/1fd1mjo4Mu8YEeseuY+tbAlzujThh14tqDliPE+saM4Xg8az64rm4AlCXNBnFXkrb6WizwVsk13OxLzbKFt1IAXfaGpNERERaRgJD0gYY3KAG4BxQBPgHeA0a+3yEOuPAG4FtgJ+A8631r7jWf4wsBtggAnW2pODXh/T/hpKKt/Ji5Ru6204BV+QZs//iOZrV1JT0CrsBWtd6xzYXqg+xtFcILd89fiwdQrsJ7jhFBxs8R6rWBtcfo3MWBryWWWra49xqH3EUh5v/fway95j6X3sF0zwNgrr8354/68L7z5azHub9YPOjZh5ESpYEKr+kYIXsTTYIn1XQi0HfGceCDyOph5+64b6DPrVz/s6bwCp5avH16s83vo1//M739d530eNG+HPO0bE9keM54enHuKdHQ/lvaqB7Lq+9WaBB6D28YA/HqCgfBplfzwAf/h/Rrz/Bx5rkEkRERGpq2RkSFwCjAJ2AVYAjwJPAAcEr2iM2Qp4ETgVeBY4CnjJGNPfWjvHXW0G8BxwWn3315BC3UX0a6jW9S5sqMeRthcp3TZcwzEvL4esdavIj9A/uK6CGx8BgYvfSBfI3gZSpIZzcMMp0OgLF5CJtSEf6lhGashml6+pPcah9hFLeYLrB2zRyPY7rn4ZK94GaV3fj3g0LL3baLrHKVSWrN9sf8GZF8Fljqb+kYIXsTTYQq3rDRD6LQd8Zx4IiKYefuuGC4CEC6x4+dUplvJ467cuKEMisDwVgxCpEPj2jhFx4IcVtP4Den5YwVt75vCXnZowoMU6zqryCTxA7eNI75+yHkRERCTesmpqahK6Q2PMXOBqa+0j7t9bA7OBntbauUHrXgXsY60d5nnuE+A9a+1VQetOACp9MiSi3p9XRUVVTYk7snd9FBU1I3g7gUZSefuBW2QWBLIAwt319q4LhH3s3UekbgGh0rfDBUuKipqx9pdPUqorhFe0XU/87nq3+u4OVnsyCKJ5bwLPRZMhEcv4BrEEoaLNkAjUL5b3zO+zG8vrE9VtJvC9i/f+InVvicc+Ase4rNvwkA0+v9+V+ormM5ao73dD1K8uiosLpwE7RVrPGHMpMBYYwabAdzNrbdjAd33PM79+/hKrH72fVieezsoH7mSb39Yxe+vm/L7DKIZ/9hwf7XEUo7t+Ttuln1GeV0R+RUnI37FU/P2G1PksNJRMrl8m1w0yu36ZXDdInfpFe44RyXQJDUgYY4qAVcAga+33nudXAydYa18NWv9lYI619u+e5+4AullrDw9adwJBAYlY9+dVXV1dU1VV/2OTk5NNVVX1Zs9lLfiK7E9ugo2ryV44jeqt9qF62EVkf3JT7f/Zv39A9Vb7AJD9+wfUNG1D1oaVW6wLhH0cah/edWu67hzX+mWKQN0C71d9j1Wqqct7ly7HIp0/l9Ec43SuXzRSpX55eTnRBiSSEvj+YtzetUGIrH655MxYRdWA1uzWqw1tl37GivZ7kD3sojqNF5IqUqXh0FAyuX6ZXDfI7Pplct0gdeqngISII9FdNgrd/1cHPV8CtAyxvt+6/Rtof7Wqqmri8mPl+6PXYjs44PFNdyQHnUul+xzg9OuvqKpNyW5WUbXpYjJoXSDsY+8+mn1wPbnzP6LC7XseeFyftNtU+VFvCLV18x7vDKprnd67NDkWaf25jOIYp3X9opAq9SsuLoy4jhv47g5MCzxnrf3NGLMG2AEIGZDIycmiqKhZncuX3S+XOZXVZPfLpdcRV1PR5gby9r6EVh0Kqf7kJloFglrbDqMpwB6n0KLOe0uOnJzseh2jVJfJ9cvkukFm1y+T6waZXz+RdJPogESp+3+roOeLgDUh1o923XjsL6FC9YUO1U+3LoOFebcV7/EdREQkeYHv9gf8m/w2t1K+ywUUb70bbP0K4KQFpkPgMBqpEpxqKJlcv0yuG2R2/TK5bpA69Ysm6C3SGCQ0IGGtLTHGzAN2BL6H2oErW+IMThlsOrB30HODgPcbaH8ZTQOSiYjEXdIC3z2335OiYSNS4sJaREREpC6SMcvGg8DFxpgPcQb/uhF42zNrhtfjwIXGmGOB54EjgcHAmMAKxph8IBvIAWqMMU2AamtteR32JyIiEjUFvkVERETqLjsJ+7wBeA34GvgTJ5AwGsAYc7wxZm1gRWvtb8DhwGU4d5ouAw4LCia8A2xwtzHOffxONPsTERGJg0Dgu5cxpiUKfIuIiIhEJWKGhDFmR2AkzuBcRTj9YqcDb1lrv4l1h9baKuAf7r/gZZOByUHPTQGmhNneXnXdn4iISBzcALTGCXwXAO+iwLeIiIhIRCEDEsaYvwLX4gzY9RHwGU5f2UKgLzDZGFMKXOYGDURERBodBb5FRERE6iZchsQpwBnW2q9DrWCMGQJcTJgMBhERERERERGRYCEDEtbaIyO92A1WRFxPRERERERERMQrq6amJuJKxphiYIO1dq0xJgdnlotq4AlrbXUDlzFZlgFzk10IERFJmh5AcQNuX+cZEZHGq6HPMSJpIdppP18HTge+wxlX4mCgAhgInNcgJUs+/UCIiEhD0nlGREREGrVoAxJ9cOdXxxk5fHdgLTCTzA1IiIiIiIiIiEgDyY5yvSog3xizPbDaWjsPZ/rPFg1VMBERERERERHJXNFmSLwFPAu0BZ52n+sH/NkQhRIRERERERGRzBZtQOJkYCzOuBGPu8+1A65sgDKJiIiIiIiISIaLapYNEREREREREZF4CjmGhDHmNmNMx3AvNsZ0NMbcFv9iiYiIiIiIiEgmC9dlwwJfGWN+Bj5y/y4FCnFm3dgLMMA1DVzGtGSMyQFuAMYBTYB3gNOstcuTWa5YGWNuBA4CuuHMrPIGcLG1dqVnnTHAFUAn4AfgTGvttCQUt16MMdnAp8BuQDdr7QL3+bSunzFmX5zv6XbARuBZa+2Z7rJ0r1tH4A5gH5zfs++A86y1093laVM/Y8wxwN+AHYBm1trcoOUjgFuBrYDfgPOtte94lm8D3I/z+V0F/Ndae2uCih9RuPoZYw4E/gEMAHKAH4F/WWs/8ayT0vVLtEw5x0DjOc9k6jkGMvc8o3OMzjGJq4FI4xUyQ8Ja+wCwNfCQ+//FwN3ARTg/VvcD21hrH05AOdPRJcAoYBegq/vcE8krTp1V4Uz12hbnR74rMCGw0BgzFLgPOANoDbwAvGmMaZnwktbfecB67xPpXj9jzF7A88AtOO9hV+Bhd1la1811L9AGJ0jaAfgGeN0Yk5WG9VuFU5+/By8wxmwFvAhcD7Ry/3/JGNPTXZ4DvAb8DBQDhwAXG2P+LxEFj1LI+uG8P3cB2+CU/0ngLWNMN0ib+iVappxjoPGcZzLuHAMZf57ROYa0+Q3WOUYkTWkMiQZijJkLXG2tfcT9e2tgNtDTWjs3qYWrBzeC/qy1tqX790Qg21p7gvt3FjAX+Le1dmLyShobY0wfnNlkjsC5A9LNWrsg3etnjPkc+Mhae4nPsrSuG4AxZgZwt7X2QfdvA/yCc0FxK2lYP/fi/r2guztXAftYa4d5nvvEXe8qY8zeOHeV21tr17rL/wMMtdbundAKROBXvxDrLca52/hiOtUvUTL1HAOZeZ7J1HMMZPZ5RucYnWNSrX4imShkhoTUnTGmCOgO1KbtWWt/A9bg3P1JZ38Bpnv+3oHN61mDc7GVNvV002gfxUnnKwlanLb1M8Y0B3YGco0x3xpjlhtjphpjdnJXSdu6edwMHGGMKTbGNAFOBT5109YzoX4Bm9XF9S2b6rID8P/t3XmYHVW19/Fv0ukEAkk6mEBAQgICSwgyiCEEZXKWiyIg+jJIAAkgMqOgXq8DwhVU9CKDTIYwOoEIDsgYBpUhApIQYMkURoME0kmHQNLd6fePXdWpnFSdofvM/fs8T54+OadO1d51TndVrVp77X/FJ1IprzcUM3sfYSanudFTTdW//mryYww02XGmWY8xMCCOMzrGrHq9af4G6xgjUl8UkKiMEdHPxTnPtwP1mspXkJntDxwDnJh4egSN388TgQXufmPKa43cv9GE3/EDCePMNyKMM/9zdEHTyH2L/Y0wHvQ/hLHn+wHTo9eaoX+xQn1pmr6a2fqE1Ocfu/vT0dNN078yacpjDDTtcaZZjzHQ/McZHWOKe71h6BgjUn8UkKiMjujnqJzn2wh3sBqOmR1AqCfyGXd/JPFSBw3cz6iI0anAcRmLNHL/4u/hFe4+x91XEMaFtgK70Nh9i+863gH8i9CP4cBZwH1mtgEN3r8chfrSFH01s42AWYQLmm8kXmqK/pVR0x1joDmPM01+jIEmPs7oGKNjDA3WP5FGpYBEBbh7O/Ai8P74uahg0EhgTo2a1WdmdjhwCfBpd5+V8/JjrN7PQcD2rJ5uW88+RBgL+riZLSSk6AHMMbNjaeD+uftiYD6QWyimJ/rXsH2LrAdsCpzv7kvcfUVUZHcwoUp2o/cvabW+RHZgVV8eA7aM0qfTXq97UfG0+4Bb3P24KP051vD9K6dmO8ZAUx9nmvYYA01/nNExRseYhumfSCPLW/AlZmY3AlcCf3L3zso2qWlcSqjQOwt4AzgHuNXd59e0VSUysxMIU1p9wt1npyxyGfCXqHDVfcAJhCno0lJT69FvCHdAYhsD9wMfJxSumkNj9+8i4EQz+yXhLs8pwHLg74Q7Ag3bN3dfaGb/Ao41s68T+nUoIfVyDrCQBupfVOW7FRga/X+t6KXlwFXA18zsQEI1+88BOxL6C3AvoZja/0b7woCjWT3tvaYK9M8Iv4cz3f1bKW+v+/7VQFMcY6DpjzPNfoyBJj3O6BijYwx11D+RZlZUQILwh/bbwC/M7DfA1e7+98o1qymcTRhbORsYBtxOmNas0ZwHdAGzQnHpwN3XjX7+NbrLcxmr5uHey90bIs3N3ZeRmIbNzOLfiQVRcaOG7h9hGrYRwF2EE6VHgU9Fd7UavW8AnyUUHXuBcCLyDHCAuz8HPNdg/fsicEXi/29HPzd192fNbD9CVfcZwHPAvvHFp7t3m9mnCXeY3yCMff2Ru/+qSm0vRmb/CNNKvxs4ycxOSixztLtf2yD9q7ZmOcZAEx9nBsAxBpr7OPNZdIzRMUZEKqqkaT/NbBLhhOcgYAVhzvNro+reIiIiIiIiIiJFKSkgETOzXYELgG0IVYdnA6e6u8ZaiYiIiIiIiEhBxQ7ZwEIeZW52xN7A68CxwO8JaVEiIiIiIiIiInkVlSFhZv8AJgK/Bq5y9wdTlnne3RWQEBEREREREZGCig1IfA64OZpfWkRERERERESkXwYXudwSQoZELws+VvYWiYiIiIiIiEjTKzYgcSFhLumkjuh5EREREREREZGSFBuQWN/d/53z3L+BcWVuj4iIiIiIiIgMAMUGJJ4zsw/nPLcH8Hx5myMiIiIiIiIiA0Gx035+F/idmf0CeBZ4D3B49E9EREREREREpCRFzbIBYGY7AUcA44GXgF+4++wKtk1EREREREREmlTRAQkRERERERERkXIpasiGmQ0FDgO2B9ZNvubuh5a9VSIiIiIiIiLS1IqtIXElsB3wB+C1yjVHRERERERERAaCYgMSnwQ2dff2CrZFRERERERERAaIYqf9fBEYVsmGiIiIFMvMtjazf5jZoCpuc6aZnVmt7SW2+10zu6ba220EZjbfzD6a8dq2Zvb3ardJREREildshsRVwE1mdh45Qzbc/a6yt0pEpI6Z2UTgeaDV3bsqvK2tgcvdfRczOwNY6O4/i17bGfg+sCPQDdwNnODu/y5y3Z8HTiLUB3rI3fdIvLYl8CNgF6AFmB2t26PXB0XbPpxQW+hR4CvuPi/P9jaKtrNxzvPrA+cBuwPrAI8Dp7j7g3ma/33gx+7eNJWZzewS4GF3v7TWbWkG7j7HzNrN7NPu/odat0dERETWVGyGxHHABsD/Ar9I/Lu8Qu0SEZFgR+AficePJF4bDVwKTAQmAB3AFSWs+03g/4CzU15rA24GjPD3/yHgpsTrBxCmgt4VWA+4H7i6wPb2Av6S8vy6hIDHjtG6rgT+ZGbrpiyLmW0I7An8vsD2Gs2ngD/XuhH1xMyKvXGS5Vrg6HK0RURERMqvqAO9u29a6YaIiNSCmZ0OnACMBF4FjnX3O81sMHAaMJ1wcX4ncIy7vwncG7293cwAPgZ8Atjc3Q+J1juRRBaFmd1NuGj/CPBeYBZweLS+fD4APBw93gH4Z/yCu9+S05cLgHuK7bu73xG978iU1x4iBCHidf8U+JaZvcvd3wA2Bf7q7s9Fr18DnFxgk3sBaww9iNbxk8RTl5rZjwnBkIdzlyfs70fc/Z1E+/ryOWJmXwTOJARFfhItd2S8b/Ixs72j904EnojWOyd6bT5wAXAoIVj0F2Bass0569oWaHf3lzM2t5aZ/ZqwD58mfHcei967EXA+sBuwFPhpIotmJ0L2yVbA28ANhOyTFdHrPcBXCJ/dOEKAaiYhuLRN1O5D4uVz2rw54ebE9kAncKe7fyF67b1Rm3YEXgf+x91/E732X9F+ew+wGPiFu383em0i4ffmSOA7wHxgNzObDpwCbAy8FLUpDs5tb2Y/ydjPdwOXm9kwd1+esW9FRESkRorNkMDMWs1sVzOLTzbWMbN1Ktc0EZHKshBNOA6Y7O4jCEGF+dHLxwOfJQwj2AhYBFwYvbZb9LPN3dd19/uL3OShhKyCDYEu4Gd52na7mbUTLhbPN7MlhEyFl83sloy37Qb0Dpkws4PMbE6RbStkN2BBFIwA+BXwHjPb0sxagWmkZz/EbWmN1nF7oQ2Z2fbAUOCZjEXeB3hi+T59jtFwmJ8DX4xeexfhgrcgM9sBmEG4+/4u4BLgZjNL1lv6PFFRaGBbwvTZWfYC/pTn9X2A3xIySK4Dfh8dlwcTZsB6DHg3IeB1kpl9InpfNyHYMAaYGr1+bM66P0EIHOxMCN5cChwCjCcEJQ7MaNP3gdsImTobEwIQROcGt0ftXB/4f8BF0f4GeIvwu9AG/BfwZTP7bM66dycEUT5hZgcA343eMxL4DPBGYtnM/ezurxCCJZbRBxEREamhojIkzOx9hNTd5YSTjl8TThamAV+oWOtERCqrm1Cwd2sze93d5ydeOwY4Lr5jbWbfBV6M7qj31dXu/ni0vv8B/mlm09y9O3dBd/9YVMfhOnf/gJl9E+h09x+lrTi6w/5twoVrvI7rCBeF/WJmGxMu4k9JPP1v4K+EwEA34a71h/OsZjfgMXfvKLCtkYS7899z98UZi7Wx+gVpXz/HzwF/dPd7o9f+hxDYKMZRwCWJOhdXRp/RzqzKUvmZu78arfsPhEyCLP8FfDPP6w+7+/XRun4CnBptawUw1t3PiJZ7zswuIwQBbnX3ZIbJ/KhOxe6ETIjYD919CTDPzB4HbktkvtxCyMy5MqVNnYSshI2i/fvX6Pm9gfnuHg8fetTMbiAM8/meu9+dWMccM/tl1KbfJ57/rru/FbXhyKiNs6PXcgNVhfZzB+E7IyIiInWm2LGZPwe+7e5Xm9mi6Ll7gMsq0ywRkcpz92fM7CTC3ddJZnYrIZ39VcKF1o1mtjLxlm5ClkJfvZR4/ALQSrhzvVqxYDM7jpDSPiz6fzswAlhqZv8NbOnu/0ksvzlwC3Ciu9/Xj/atwczGEu6CX+Tuv0y89G1gMuEu+gLCHfW7zGySuy9LWdVeFKiPYGZrE+72P+DuP8iz6CLC/gD69TluROIzcfe3zCwZ6MhnAjDNzI5PPDc0WmdsQeLxspzXeplZG2EYT74ZIZLtXGlmL0fr6wE2ir4jsRbgvmjdWxKGonwAGE447ucOg0l+/95O+f+4jDadRsiSeCg6NzjX3WcQ9s2UnDYNIaoxYmZTCHVLtiHss2GE7I/U/hK+Y89mtAEK7+cRQDsiIiJSd4oNSExi1bjfHug9cVu7Iq0SEamSOIsgujN/CXAOIYX/JeAId/9b7nvMbELKqt4iXPDF0i7ixiceb0K4w7wwpU0XABeY2V+A7xHqE8x1900y2nIH8H13L1RUsiRmNpoQjLjZ3c/KeXl74NeJmgczzez/gK1ZVYQzaS9gvzzbGka4Q/4yhYsQziFk6PXq4+f4b8KwgPj/wwnDL4rxEnBWyn7pi08Ad6VlyiT0fneiYRobE2pldAHPu/sWGe/7OWEGlAPdvSMK3HyuDG3G3RcQam5gZh8C7jCzewn75h53/1jGW68j1Nf4lLu/E31vxuQsk5w95SVCvYmSmdm7CUEPL7SsiIiIVF+xAYn5rF7pPS6UlTW+V0Sk7kW1B94N/A14h3A3uCV6+WLgrGhIxQtRpsAu7n4ToUjfSmAz4F/R8v8ETjezTQiF+r6RsslDzOwqwt/UM4DrC1yEbk+oDTCF1WfXiNv/buAu4AJ3v7jIbiff30LI0hgCDDaztYBud++MLuxvBf7m7l9Pefts4AAz+xVhfxwcrWuN44KZbQoMc/cnM9rRClxP2P/T3H1l2nIJtwPnmdla0QVtXz/H64EHo4vphwifSbG1lS4jZF7cEb13OLAHcG+hYSkpCtWPANjRzPYjDJ88gTCE8gHC97DDQlHPnxGGcGwFrB0NcRgBLCFk17wX+DLh8+q3qLbD/VFQahEhiLAS+CNwdjQs5lfR4tsDS6PvwAjgzeiz2wk4iBD4ynI58BMz+yvh9+A9hOFLLxTRzN0JwR4VtBQREalDxZ54/Q9hCrbvAUPN7BuE9MpvVaxlIiKVN4yQOr6QkPa9PqsCCecRLv5uM7MOwsXfFIBoSMJZwN/MrN3Mdnb32wn1deYQUuL/mLK9qwkzGCwA1iJcWKaKAhtvRNt6P+mzTRxJCIp818yWxv8S6zjYzOalvC/2RcLF+88J03e+zaqhePsShmQcnlx31C4IGQiPEQIx7YTCifu7e3vKdv6L/MM1diHUHfg4YeaSeFu7pi3s7q8RAjFxvYy+fo7zCEVDryPUxFhEyNAoyN3/QcgOuCB63zPkL1qZyswGETIkMguCRm4i1GxaRPjc9nP3ziigtTfhgv95wj64HBgVve+rhAv+DsJn++tS25jHZEJAZylhH5/o7s9FAZmPE+pYvEr4TM4hGoJEKKp5RvR5fBv4Tb6NuPtvCb9v10X9+D2huGcxDiYEpURERKQODerp6Sm8FL0VxacTxoa+BFyWUyxLREQyWJj28xp3v7zWbak2M/szIYsjbw2JEte5NaHQ4k7uXtyBrLj1zqfIaT/LtL2dCPtmp2psbyCJCr1e4u5Ta90WERERSVfsLBsHRHcojs15/nNx1W8REZEMdwOzyrlCd3+CcIe+GXyn1g1oRu4+hzDVqYiIiNSpYmtI/II1K2BDmKtcAQkREcnk7j+sdRvqlbs/VOs2iIiIiNRK3iEbZrZZ9HAO8D5gUOLlzYCr3D11GjMRERERERERkSyFMiSeIVTNHsSac4AvIMz5LiIiIiIiIiJSkqKKWprZPe6+exXaIyIiIiIiIiIDQNGzbIiIiIiIiIiIlEuxs2wMIcywsTswhkQtCXffrTJNExEREREREZFmNbjI5X4KHA3cC+wI3ACsD9xVoXaJiIiIiIiISBMrNiCxH/Apdz8P6Ip+fhbYs1INE8liZjPN7B8lvmd9M/uumU2sULMqysymm9nzZtZlZnfXuj39YWYTzazHzPbuw3uPMrPP9nG7JX9vRKQx6LhQmeOCmZ1mZnukPN9jZsdVYpvR+vcxsyfNbIWZzS/hfd81s4WJ/+8RtXWbijRURET6rdiAxHDgpejx22Y23N2fAnaoTLNEym594DvAxBq3o2RmNg74OXATYdjUsbVtUU0dRQiGioj0l44LhZ0G7FGhdacysxbgKuAx4MPAvtXcvoiIVFdRNSSAJ4HJwEPAP4DvmtkS4JVKNUxEem0OtAAz3H1Of1ZkZmu7+9vlaZaIiNRIMx8XNgRGAte5+19r3RgREamsYgMSJwLd0eNTCFH5EYS7lSI1ZWYbAmcR7uJsSMjm+Q1whruviNJx50aLzzIzANx9UPT+9YCzgX2AUcAjwMnu/mBiGz3AScAGwHSgB/gtcIq7L08sNyFa18cImUXPAGe7+3Vm9hDwhLsfltP+mcB27r5GxpGZfZdwBw/gsajth7v7TDMbA5wL7A2sTQgYftXd/5F4/3xCzZd2Qh2YDYDWPPvySOBkwsnuAuBCd/9h4vWpwDcIAcqRwNPAj9z92pz1ZO6HxGLDzewS4P8BHcAvgO+5+8qMtt1NqGGzo5lNi56O90UL8D/AEVEfnwHOytle7vqGAr8CPgB82N2fMbNNgB8CHwfWAu4DTnB3j94zEXge+ALwkWLbLiLVpeNC/48L0XLvAr5jZvH29nT3u6PHLWb2vwX6nvdvaso2DwOuiP57U9S37wEzCX97P+3uf8zZT9u4+wfS1lcMM2sFfgB8nrAv3gAeBL7g7iuK7YeZjQcuIQxnfg04E/gkMMbd98hqb+K40ts3MxtMyE45EhgPvEA4pl2ZeN/dwELgd8AZhIyfvwHT3f3lxHJrE/bhF4BxwKvAr9z9G4ll8h77RUQqqaghG+4+290fiR4/7e4fdfcp7n5fZZsnUpQxwJuEYNkngR8BhwPnR6//Gzg4evwVYGr0DzMbBtwBfBT4GmE4wOvAHVFKbNKpwEbAIdE2jiYE64jWtT5wP+Fi/avApwkXquOjRX4BfM7M1k28Z13gc8CMjL5dHrWZqA9TgT9F//898IloW18g/D7PMrPNc9ZxEKtSer+QsR3M7GuEYOPvCSezPwe+nzNOeALhhOdLUf9uAK4wswNL2A+xHwJLo/5fA3w7epzlWOAp4M+s+gzjfXEG8N/ApcBnojZem2xXTl/XAm4EtgN2jYIR6wF/BQw4hnByug7hu7B2P9suItWl40L/jwv7AoujNsb755ES+l7K39TYnwh1y4j6MDXqbyV9g7Af/4cQNDqJ0O8WKK4fZjaIMHxmG8Lx8RTCvpjaxzadD3yLcEz7L8LxakZK7aUpwHGEz+Io4P3Re8hp15eBC4G9CMGsMYllijn2i4hUTGaGhJl9uJgVuLtm2pCacve5hBMXAMzsb8BbhIP38e6+3MzilNYn3P2BxNsPIZxATHL3p6P33wE44QD/tcSy8xN3sW41sw8STpziuwgnE+6k7eju/46euzPx/l8CPwEOYNUdoM8T7kyl3sl395fN7Inov3Pc/fGojZ8EPgjs4e73RM/dBcyP2nx0zqr2dvd30rYRvXck4STlTHf/XvT07WY2HPiWmf3c3bvd/VeJ9wwizLyzMeEO2S+L3A+xe9391MS2PknYn7/J2BdPmNlbwOvJzzA6WTwpavuZ0dO3mtnGwHcT7YqXHw7cHLV7N3ePh56dTDjJ3N7d34yW/Rthnx5BOJnrU9tFpLp0XOj/ccHdHzWzLuDlnP1TSt+L/Zsab/N1M3t01X/Ddq2yhUd3IgwPuTLxXPJveTH9+BShrtrOcRaNmT0MPEvIJCxaFDz6MiHrJW7THVHWz3eAPyYWHwn8l7svit47DvhpYhjOxwlBln3c/ebE+66Kli/q2F9K+0VESpVvyMYvinh/D7BZmdoi0ifRhfGJhLsDmxLSKWObENJjs3wUeBh43sySvw/3EFL5k27L+f8TOct8GPhL4qRzNe6+xMyuBw5j1YnnYcDN7v5Gnjam2Qn4T3zSGa3/LTP7I/ChnGXvzHfSGZlKOOH6bc5+uItw12hj4AUzG01I/dwHeDfRHSRWryeTdz8kpO3PTQq8J802hDTo3+Y8/2tgppmNdffXo+fWAf4CtAG7u/trieU/CtwOLEnsgw7C96OY70Jf2i4iFaDjQu/6+3NcKKRQ30v5m1pL/wS+bGavEY4Pc929J/F6Mf3YCXgtOaTH3V+IghKl+giwErgx5/t3J3CgmbUkggSz42BEJA5UvZvwHf8w8GZOMCKpqGN/H/ogIlK0zICEu29azYaI9MNJhHTRcwgnjIsI6bEXsvpJaJoxwM5AZ8prz+b8vz3n/yty1v8uYHaB7f0CuNvMNgMGAbsSUihLtSHwn5TnXwPWS3mukDh9c17G6/EY1pmE/fV9wonPEsKdnH0SyxazH6Dw/izWhtHP3H7G/1+PkG4NIb14C8JY3Nzl4+9CWvpyboZHe87/+9p2EamMk9BxIdbX40Ih7Tn/z+17KX9Ta+lMQgDgWML35RUz+5GHKe6huH6MI33f/4dQc60UYwjB/sUZr28IxDUi2nNeWxH9jD+HdxGGJ+XbFhQ+9ouIVEyxRS1F6tkBwPXu/t/xE2a2dZHvfZMwc8yXU15bnvJcPm+w6uI4lbvfa2ZPE+6ADSIUl8q9y1SMfxMKWOXagNCnpJ6U5XLF79mb9BNVj+ou7A18xd0vjl+Iim8lFdwPZRafbK0fbTu2QfQzuT+eBs4jZE4scPefJ157kzCU4/sp2+goU1tFpDp0XFilr8eF/irn39Q4m2NozvOjS21UrihT5NvAt81sC0KdiP8zM3f3v1BcPxaQvu/XB5IzmLxD4T68CXQRht+kFUpOC3xkKfT9K3jsL2FbIiJ9ooCENIO1WfMk8eCc/+feNYjdSRhj+aK7l3KQT3MncIKZbZBy9z1pBqvmjL+qj+MzHwS+Z2a7ufu90FsbIS5+Var7CSdNG7n7n9IWMLNRhAJpyQrqIwhFJJMnt8Xuh75Iy0R4HFhGuAA5I/H854F/JYZrAODuV0dF4y4wsw53vybR7s8D87y+psATkdLpuEC/jwvQv+yvcv5N/Q8hY2Wr+Ino7/gulPEOvrs/bWZfJRQN3ZowhKOYfswmzEYyJVFDYhNCkcm/JZZ7GZhoZmslhsx8PGdddxEyJEa5++397NKdwGlmtndydpKEgsd+EZFKU0BCmsHthBO+BwnptAcTpq5KepFw0J1mZouBTg/ToF1FuBtyt5n9GHiOkOK4E7DA3X9aQjt+ChwK3GdmZxGmmdsKWCdn+qwrCSmiQ1g1Zrgk7n6rmf0d+LWZfZ1wF+SrhJPwH/Vhfe0WppI7z8IUdfcSgg9bEqZ529fdF5vZbMJdpCWEOzdfJ6SVjkysrtj90BdPAZ8ws08Q+vy8u79hZv9HKMDVRbizuR8h5Tl1lg13/3l0MnuFmS11998TCssdAtxlZucT6mJsQKhE/1d3/2XaukSkLum40M/jQuQp4L/M7C+EmYXc3YvNbijb31R3X2lmNwEnm9kLhKEKp7J69kGfmNmNhHoQj0br+xzhc7i3hH78GXiMUIvhdEIw7Husmc3we0Lg/HILU4DuQCiMmeyrm9nFwK/M7IeEY9pawCRgS3c/soTu3Q7cClxnZmcQZknZkFDQ+ehijv0lbEtEpE+KmvZTpM6dQahUfmb0cwVwQnKB6E7EdGBHwnji2Ynn9yQctL9HSJM9j1Bn4KFSGhHdif8g4aTm/wiVsI8inPQml1tAuJP1N3f/VynbyPHZqN3/RyjoOAj4sLvnK9aWKTo5PopQLfwmwr48mDDfeuwgwsn5VYT9dEP0OLmeovZDH50JPEmogD6bMIUehHTbHxBSrP8I7AYckpwVJJe7/yh6z6/M7GPuvpAwTvgpwkXEbYRq8aOAOVnrEZG6pONCGY4LhNk53iJMxzmbsK+KUoG/qccRsg0uItQC+SUhm6C//k7Yb9cRjn07AvtHwami+hEVwfwMobbSjGi5CwgZCL2iGVGOIBSTvJkQ1Dg8pU1fIQwROZQQ7JhJyHS5N2XZTFG79iVMBXoScAvhd2JhYplijv0iIhUzqKenGsMIRSQWTVP5CnCcuxczm42IiDQxHReaUzSDyhh336PWbRERqVcasiFSJVG9ha0JU9F1EO5CiIjIAKXjgoiIDHQKSIhUz47ALEIBrkPdfVmN2yMiIrWl44KIiAxoGrIhIiIiIiIiIlWnopYiIiIiIiIiUnUKSIiIiIiIiIhI1Q3YGhIrV67s6e7u/3CVlpZBlGM99Ur9a3zN3kf1r7HVU/9aW1sWAmMrtf5yHHfqaX9VSrP3Uf1rbOpf46uXPlb6mCPSKAZsQKK7u4f29v7XjmprG16W9dQr9a/xNXsf1b/GVk/9Gzt2xAuVXH85jjv1tL8qpdn7qP41NvWv8dVLHyt9zBFpFBqyISIiIiIiIiJVp4CEiIiIiIiIiFSdAhIiIiIiIiIiUnUKSIiIiIiIiIhI1SkgISIiIiIiIiJVp4CEiIiIiIiIiFSdAhIi0lTmvLqE42+Yy5xXl9S6KQOG9rmIiAwEOt6JlJ8CEiLSVC67/wUemL+Iy+4fWNN71/IkaaDucxERGVh0vBMpvyG1boCISDlNnzphtZ8DRXySBHD+/u+r+PbmvLqEy+5/gelTJwzYfS4iIgNDfMzbc4sxgI53IuWkgISINJVtNxpZlQvyepAVFEg+v+1GI8u2nT23GMOspxcyfeqENQIg8T5PbhsoWzuy1lvObYiIiKSpdtBfZCBRQEJEpEFlBQWOv2FuWU+c4u08uaCDxe90AdmZKMk2Af1uRxyI6Hini3kLOtZYbzm2ISIikkuZgCLVoYCEiEiDyjpBKnTiVEwWQ9qJWDJDIisTJW3bycdZ2RtZbYoDHJPGjWDniaMz16uTRBERKae0oP+8RXM5/aEZHLrFEUwarSC4SDkoICEi0qCyggLJ59Mu9LOyDeIAQL4hGfttu2FJbcptR3Lb06dOYOZN8zhs8vjMzIpkgCMZwEjbhoiISH9kZUXMWzSXq56ewdLODp5c/AQA5+z001o2VaRpKCAhItJgSqkRkXahn5VtkFy23OmpaZkO8XOdXSszMysGUk0QERGpvuQxNSsYf/pDM5i98EG2GrU1k8dM4dAtjqhlk0WaigISIiI1VkyAId8JUz75LvRj8TpysxHKGQhIy3SYPnUCrUMGc9jk8ZmZFSIiIpWUFYyPsyIO3eKI3gCEhmqIlJ8CEiIiNZZ7MpQWnOhr9kIpF/qVzEZIW/e2G41kxrTJtLcvq8g2RURE0mQNzWhZ+wWGj59By9pHcNXTISsCwvAMDdEQqQwFJEREaiDrZCg38BDXWKhk9oKIiMhAUmhoBrBaVoSIVI4CEiIiVZRW3DF5MpQWnOjsWrnaMiIiIlK6Oa8uYeZN89hzizFA4aEZyooQqTwFJEREqijfNJaw+tCGZI0FqT9mNhj4KzAVGO/uL9e4SSIikkd8DF688mnGjL9HQzNE6oACEiIiVZQ1jWUa1VioeycD+nBEROpYcojkR7d9ixeHXc3KdbuYvdABDc0QqTUFJEREqkj1H5qDmW0JHAvsDzxa4+aIiEiGZL2I4eN/TcfgeWzcumr6Tg3NEKktBSRERERKEA3VmAF8FWivbWtERCRXblbE/KFX8dH3HsnmY45gSGsLB02cpuk7ReqEAhIiIiKlORFY4O43mtnEYt/U0jKItrbh/dpwS8vgfq+j3jV7H9W/xqb+NYYLb76ZJ7tv4MJ/7M+Y8ffyVssTPLT0eqZNvojdNp9Kd/fKWjdRRCIKSIiIiBTJzDYHTgU+UOp7u7t7+l0PpK1teNPXFGn2Pqp/jU39q1/JrIiW0bczpONpWkbczkETj6Krs5uDJk6jvX1Z3fRx7NgRtW6CSF0YXOsGiIg0sjmvLuH4G+Yy59UltW6KVMeHgLHA42a2EHgken6OmR1bu2aJiAxs5z14F491/5jzHryLr2xzFJPHTOEr2xzVWyNCQzRE6pMyJERE+iC+E9PxThfzFnQAqFjlwPAb4I7E/zcG7gc+DjxVkxaJiAxQNz39AJc/dTlHvvdIho25kyEdTzNsxJ1MGv1ZFaoUaRAKSIiI9EFctXvSuBHsPHE006dOWC1dtNCUntKY3H0Ziak+zSw+ji5w96W1aZWIyMAyb9Fcrnp6Bo+/9h/ebnmey5+6nLM/dCJXPT1D03eKNBgFJERE+mD61Am9P+Pgw/E3zO2dWkzZEgODu88HBtW6HSIiA8mFj1/KUx0Ps8FaWzD4na058r1HavpOkQalgISINKw5ry5h5k3zOGzy+KpnJGy70cg1gg7JIIWyJURERMonzoo4dIsjWL7wI3R1LmF46778ct/P1rppItIPCkiISMOKh010dq2si4yEZJAimS0xfeoEBSdERERKlAxCxFkRHcu7OHHKWVx2/3uYPmVCrZsoIv2kgISINKzpUyfQOmQwh00e3++MhOT7gYKPC20jmS0RB05AQzlERESKlQxCxFkRy5d/hG13XTNLUUQaU9UDEmbWApwNHAasBdwGHO3uCzOW/yRwLrAZ8CxwirvflrLctsA/gHvd/aOVab2I1JNtNxrJjGmTaW9fVlL9hrTgQ3K2DKB3XVmPC20jmS2RDE6IiIhItrShGcuXf4QTp3xYWREiTagWGRJfB/YBpgBvADOAq4FP5S5oZpsBvwOOIky1dgBwo5lNigqJxcsNidZzX6UbLyL1qdBFfzIIkcxYANaYLSN3nfkeFyOt3oSIiIgExQzN0LFUpDnVIiBxFHCGuz8HYGanAc+Y2QR3fyFn2WnAw+5+TfT/a83smOj57yWW+wYwG3gN+FBFWy8iNZU1NCPtRCUrCJEWvMhdX3JdWY9FRESk/656egazFz4IoKEZIgNMVQMSZtYGbAI8HD/n7s+a2RJgOyA3ILFdctnII9Hz8TrfRxj+sQNwSrFtaWkZRFvb8BJan7WewWVZT71S/xpfM/Tx0RcXcf6sZzh+z82ZOfslHpi/iNYhg5kxbXLe/s28aV7vsqd8bMvedeywyWh223pc73LJx/WmGT6/fJq9fyIiki3OjNh0rSk80d3BLqO/wKYbb62hGSIDSLUzJEZEPxfnPN8OpFWIG5Gx7CToHapxBXCiuy8xs6Ib0t3dQ3v7sqKXz9LWNrws66lX6l/jK1cfqz2NZVp2Q2fXSqZPnUBn10oOmzye9vZleft32OTxvctuOnIYP9lnEkBDfebN/h2tp/6NHTui8EIiItIvyeEZcWbEE90dLPjXodyxYh3O319ZESIDSbUDEnHFuFE5z7cBSzKWz7fsacDT7v7HMrVPRDLkDnkoNjhRTCAjXmbPLcYw6+mFeYdYlDKGVONNRUREai8tCAGwy+gv8MRrHXxy3CE8tWK0ij+LDEBVDUi4e7uZvQi8H/gn9BauHAnMSXnLY8CeOc/tANwZPf448H4zi2foGA4Mif6/pbu/Wd4eiAwsyWBC1jSWyeAEUDCwkDaFZrzMkws6WPxOV++y8U8FFkRERBpXMghx6BZH9P68+C5YMP9QnloxWsd5kQGqFkUtLwVON7NZhFk2zgFuTc6akXAV8DUzOxC4HvgcsCNwaPT6AcCwxPKnAJOBAwlDO0SkH5LBhPP3f1/qNJZpM1ZkBRbSlk0ukwxkKAghIiLSuJJZEckgRPfbE1j20hF0bzyB6VPDssqMEBm4ahGQOBsYTZgVYxhwO3AIgJkdDFzi7utCb8HL/YBzCdN6PgfsGwcv3P315Iqj4pjL3f3l6nRFpDklh1DAmicKyWBB2owVWYGFfLNbxMvst+2GFeqViIiIVFLW0Ixzdvop5+z0UwCOv2tu6s0OERmYBvX09NS6DTXR2dndo6KWhal/ja/YPqYVkdx5Yv2nUDb7Z6j+Vc/YsSMeBj5QqfWX47hTT/urUpq9j+pfY1P/8jv9oZOZvfBBJo+Z0huUiLMi0oZsVqNIdq7+9LHz8Tksu+Jyhh9+JK3bbNuvdlT6mCPSKGqRISEidSiriKSIiIhIlqyhGZNGv69psiLiQMTKpR10PzGPZcCoc39W62aJNAUFJEQEUBFJERERKV4ciFja2cGTi58AVh+akVUYu1EksyGWXXE5nQ89QMtWk2jdaWeGH35krZsn0jQUkBARQFNkioiISH5pNSK2GrV17xCNpKzC2I0iDkIsgxCUiH72d6iGiKxOAQkRERERESkoa/rOSaNDsKGZsiJygxAaoiFSGQpIiIiIiIhIqkI1Iua8uoTj75q7xvTejZIVkTY0I64RoSCESOUpICEiIiIiIr2Kmb4z1ohFsbOCEMmsCBGpDgUkRERERESkV9bQjKR4eMaeW4wBGqModufjc3jlmitYsai9d7YMDc0QqS0FJEREREREBrh5i+Zy3aNXctDEaZnTdyblDs+oV4Vmy1AQQqS2FJAQERERERmgcqfv7OrsTh2aAY1ZtDJ3SMaK1haGHnK4ZssQqRMKSIiIiIiIDFDJ6TunbrgLB02clrlsoxStzDdbxtiLL6G9fVmtmygiEQUkREREREQGkKyZMz646ZQ1LtYbJStCs2WINCYFJEREREREmlwpM2ck1XNWhGbLEGl8CkiIiIiIiDShrCBE1swZsXrPiogDESuXdmi2DJEGp4CEiIiIiEgTygpCZM2cEavHrAjNliHSnBSQEBHpg+Rdp0mjiz9R6+v7REREipFVH6JQEGLOq0uYedM8Dps8vm6yIooZkqHZMkQamwISIiJ9kDv+ttLvExERySd3+k4oXB8iKc6K6OxaWTdZEfmCEMqGEGkOCkiIiPRBofG35X6fiIhIPsnpOyePmVL0cSauF7HnFmNoHTKYwyaPr3BL88s3ZaeCECLNRwEJEZE+KJT6Wu73iYiI5Mo3PKNYyXoRV39pzWk/q0FTdooMXApIiIiIiIg0iL5O35lUL7NoFJotQ0SanwISItIUVCxSREQGglKm70xKBiFqOYuGZssQkSQFJERkDY14cV+NYpFp+yX5XNyORtpvUjozOws4CHgX8A5wL3CKu79Y04aJSNPq68wZSckgRLWzIjRbhohkUUBC6l4jXhw3onwpoI2gGsUi0/ZL8jmg4fZbKfS72Otq4IfuvtjMhgNnAr8Cdqlts0SkmVRyaMa2G42salaEZssQkSwKSAwwjXhB0YgXx42+nxtlJoh5i+Zy3aNXctDEab13ieYtmsvpD51ckX2ftl+yniuUOdHo35FG+V2sBHd/KvHfQcBKwGrUHBFpUuU4LtfL0AwFIUQkiwISA4DufFdfo+/nRpkJIt7PXZ3dqRkL5ehDbuAgd525z8WPT3/o5LyZE43+HRnozOwg4OfASKALOKW2LRKRZhEfd3YdtztQ+nG5lgUrNVuGiJRKAYkBoBHvfCc1ysVxUnI/N8qd8LT9XO9tP3SLIxjS2sJBE6et9lz8sxzt72vgIF82Rdbr9Sq5Hxvtd7FS3P064DozGwd8CZhb6D0tLYNoaxver+22tAzu9zrqXbP3Uf1rbJXq32OvP8alj1/CUdsczXXzr2T2wgcZ0trCJR+/pOR1zbxpHg/MX0TrkMHMmDaZ3bYeV/R7+9K/t//5T968+Oesd8yXeeuaK+h86AFWtLaw/vHH8ebFLax3zJdZu46+E83+HRVpNIN6enpq3Yaa6Ozs7inHPMttbcNrMl9zKfpzUdYI/euPavQvvlM+ecyUmlzM9aePtW57MfL1L9n+OEOomN+DeipUWQ+/g2nfg3IFq+qhf7GxY0c8DHyg1PeZ2frAc8Am7v5m1nLlOO7U0/6qlGbvo/rX2CrVv74er5LizIg9txjDrKcX9taKKEUp/cudsjOeJSPOkKjXIpX18h3t6zFHpNkoQ2IAyMowSKYE3rfgnrq6C17vd+ZLUcqd8HrrdyNmeiQl25+V6ZDWr74WD+uret+3ad/hRhxyUkFDgHWAjYDMgISISFI5Zs5Iyq0XUQmaslNEym1wrRsglREX9pu3KDuLOL6guNwvYfbCB7nq6RkV3XYxbcptWzFtKmW9lZTVjuSJRbGfSbk+i/6K2z5p9Puq0rZyf5bJ9h+6xRG9d56S20nrV3LZasjat/MWzeUrs46t+Xc7uR9j1d5H9cLMBpvZcVFWBGa2MXAhMB94Kt97RUSyjj9pf2eLMefVJRx/w1zmvLqE6VMnsPPE0WWvF9H5+BwWn3pCbzCi86EHVmVB7LQz655wMqPO/VndZkSISH1ThkSTKubuZXwhkcyQSMqdwaAv206mHZZyRzUrq6CYu9m1UqgdpXwm9XiRV41siUp+lrmBoXx1VapdtyRr36YV7SyHYj6/Qssk91HWsqV8T+o9SyTHXsC3zWwdoB24G/iou3fVslEiUv/KUdcrWbSyGrNoaMpOEakkBSSaSFbqX5bkBcXem3x2jXUkL4aSgQVgjcfJoEZWmnxam/KN1S/2or6YC+VCFzvluEArtM+LDbLku8ir5UVbvgv6fO2ENb8vpe7Dcvc7LTW2ktOFFpJv38ZFO0upa1Ho92BpZwdPLn4CyB6+UkpwqJggZNb3JA56FrNsf4Me5eDuKwkBCRGRolRyaEalZtHQlJ0iUi0KSDSBrIuMvhzkci8K4ouh5PPAGo998VMs6Vy8xrazDrxpbU6ut5isjuTFY6EL5bSLq2IuhuJ9klzH0s4O1m0dUXQQJZYVbMi68Et7vl6yQfIFnfJ9rsnHWfu50D7qa7GvpLTPqh737aTR7+PCTS+ivX1ZwWlEkwoFCLYatfVqw1ey9nPyZ7FtzlpHvu9zHPTMWkfud6qvmVciItWS7zjfl79VWVN5brvRyLJlRXQ+PodXrrmCoYccrik7RaRqFJBoIFl3SNMuMvoq62Io7eIk+Thr2EehTIe0NheT1ZG8OEuuv5gLo9xgSL6LIWCNdSzt7Cj6wjBLMRd+acGXrGyQeJ1pj6979EqmvutDZS1cmvxc0/Zdoc81335OC74UurAth1IuwCsp63cmrX3J7was+tyL+T2IvwdZw1dKuXOX9X1Iez73+xwHPUv5TpUj5VlEpJLK/XeqkkMzcmfL6OrsXi0rQkSkkjTtZz9Vc+qg5JRQQL+nhypGpfrX3zTr/gyxiPfjVqO2pm142xo1MgqlxpdjSshS+p819WbW9yHt8cjWUSzpXFzx70tfawZA/v2cL0W/XqbvqpRipzUFUr8n5RiuVE653+dCn181hy5Vego2TftZnGbvo/rX2Ir9m1WOGcySWRFA7+NSp/JMkzVbxtD12hh6yOFNXaCyXr6jmvZTJFBAop/S/qiV82S5r2Pxy6Ve/miXU3KffnDTKXXfv2IuzCFPhsT81TMk4rs2lQ5OVEszfkeT8vWvHMGxaqvngJICEvWh2fuo/jW2Qud9yWNsf7P4jr9hLg/MX8TOE0eXJSMiLQgRT9cZPz/2Qzs39ecH9fMdVUBCJFBAop/iP2rlPBhV6sDWF/XyR7tSmr1/sGYf6+n7VQ7N/hmqf9WjgER9aPY+qn+NrdB5XzkC/XFmxJ5bjGHW0wvLlhWx+NQTUoMQyWyIZv/8oH76qICESKAaEmVSzrGCGh8tlZQ1Vl9ERESKk68+T3+LVubWi+gPzZYhIvVOAYkyKeVgVCjNuhwHNpFi6PslIiJSnOTsXOU+VyvnVJ5pQzM0W4aI1CsFJMqklCBEvlkccqc+FBEREZHaSDt/6+rs7vP0nUnlnsozd7aMOBtCs2WISD1TQKKCsoIQWVNoVmo6Q5HkSU85xqGKiIgMBLnnb/FUxeVQjqk8s2bLiOtEaGiGiNQ7BSQqqNjxhRrPL/1VKOCQmwpaqWnEREREmkFy+k5Ydf524aYX9asgYlZWRCmyhmTk1ogQEWkECkhUUKnjCzWeX/oqX8Bh5k3z2HOLMau9Fi8LlK1wloiISCPLymwt59CMcmRF5AtCKBtCRBqNAhIVpACDZCn3EIrkXZa0gENn18rek560OzJ9LZwlIiLSLCo1y1k5ClZqtgwRaVYKSIjUQDmn9AJWK4CVe7LTOmQwh00en7psubYvIiLSiJJZEeWcOaMcBSs1W4aIDAQKSIjUQH+n9Mon92RnxrTJ/RrvKiIi0kzyDc0oV2Zrf4ZmaLYMERlIqh6QMLMW4GzgMGAt4DbgaHdfmLH8J4Fzgc2AZ4FT3P226LX1gN8D743W9TpwBXCWu/dUtCMiJcodphGfnGQN30h7PvkcqCCliIhIMUqZ+aw/4uN0snZTMTRbhogMVLXIkPg6sA8wBXgDmAFcDXwqd0Ez2wz4HXAU8BvgAOBGM5vk7vOBt4AvA/9y904z2xT4M/Af4NJKd2Teorlc9+iVHDRxGpNGK+29WSRPCoDex/2tWJ01TKPQ8x3vdDFirSEqSCkiItJHxc581l+lDMnUbBkiIrUJSBwFnOHuzwGY2WnAM2Y2wd1fyFl2GvCwu18T/f9aMzsmev577r4cmJfznpWAVa75q8QHt67ObhWvbCLJkwJgtTGbfVHobkly+EbamNOOd7ryFsNSQUoREZF0WdN3lvO8rZSpPIsNQigbQkQGiqoGJMysDdgEeDh+zt2fNbMlwHZAbkBiu+SykUei55Pr/SPwEcKwjZeASwq1paVlEG1tw0vsweqO3eFYLnu8henbHN3vddWrlpbBTds3SO/fC5+fxoL/LGXc56ex5QYjePPiFtY75susnbIfHn1xEefPeobj99ycHTYZnfr8zNkv8cD8RbQOGczVX5qyxjp2axvObluPA+CIK2f3Ljtj2mR223rcGtuIlwVWe1xKH5uJ+tfYmr1/IlJ9lZq+M0sp9SIUhBARWV21MyRGRD8X5zzfDqQNgh+Rseyk5BPuvndUm2Iy8GkgtR5FUnd3T78L/U1o3YIL9ryI9vZlTVs0sK1teNP2bc6rS5g5+6XeGSh65wh/eSgPbH8YO788lOnjx3HZ5COYPnQcPLGgN9Nh1tMLVxtC0dm1svf/ac93dq3ksMnjC+7LwyaPX2PZTUcO4yf7hK98Xz6LZv4MQf1rdPXUv7FjRxReSETqXiVrRMT6mhWhIISIyOqqHZDoiH6Oynm+DViSsXxRy7p7N/CAme0GXAgc2J+GFqPz8Tm8cs0VDD3kcI3xa0DJoAGQOiwirWbDkws6WPxOV95l+zrFVynLioiISFCp6TuT5ry6hJk3zeOwyeMLZkVoyk4RkeJUNSDh7u1m9iLwfuCf0Fu4ciQwJ+UtjwF75jy3A3Bnns0MAbbod2OLEB9gujq7dXCpkqwZKfpi+tQJtA4Z3JshET+XDAqk3fVIZkhkLavAgoiISGVVY/rOpNzsR1gzK0JTdoqIlKYWRS0vBU43s1mEWTbOAW6NZs3IdRXwNTM7ELge+BywI3AogJntDKwD/B1YAXwQOJEqzLAB4cCyorWFoYccXo3NCaVVry5k241GMmPa5N508bT15QYW4sf7bbthwWVFZE3Ju4bKLBORUlVr+s5Y7tCM+EZG8pivKTtFRPquFgGJs4HRwGxgGHA7cAiAmR0MXOLu60Jvwcv9gHMJ04M+B+ybCF4MBX4IbAn0AK8AP4u2UXGt22zL2IsvKXn8czOekFerT4XGaUphzfT9a/ZhU83Yv9zUZRGRUlRr+s5Y7o2Q+EaGpuwUESmPqgckoloPX43+5b52LXBtznN/Af6Ssa57CRkTdS/fWMJGlpaaWMk+xXck5ry6hONvmFuWoRsDQbN+/5LDpuK+NdNJYDMOC1Pqsoj0RTWm70wqNGW3ZssQESmPWmRIDEhZB65Gl5aaWO478Ml0ya3enM+yKy7nlol78sCKdwH9H7pRrEbOLCj3969e9kVy2FTcx6UdHQweMaJfbatl/3KrsTfLsLBkv+KT9WbMABGR8qn29J3J8420IaLJv1kKQoiIlIcCElWSdeCqlwu7vkpLTVx86gmrX/z2s3/Jk4Iz/x4uOg9e3sXLe36Z6VMnrHYCES+fmzmRtcxubcOLbkc9ZBb09ftS7hOnNQIcNfoOJ4dNxX1cubSj359TLT/r3G3H/Wv0vxVp+7QZM0BEpHyqUSMiKW22rGPf9RaLTz1htQzD+G+W/m6JiPSfAhJVknURWC8XdoUkL4aA1dqZ26/kxW85LuySdSOGW1j3qMOP5Pxtwh2L42+Yu8bUnPHyaXc6ksvstvW4otsR92vo7nv2npxU+3MqdX8m7+akLV+OAEc5PuNC7cj3/YvF38XcZfui2llMWXPUJ1U7SFLKdyPr80k+TutXM2WAiEh5VGP6zqTcopVAb0bmmX8PQ1I7E7Nl6G+WiEh5KSBRY+W+sKuUZNuAvO1MBinKcWEXnxQMt/QASFqhy9wgRNYypYi3ncwAKffnlHYRmHWxWsxFeqE70IUCYlkXpeX4jEupa9HX718x+yhNudNvC13cFzNHfdZnXyiAU0owIe27U8z3POvzST5O61cyw6XRM0BEpO+qPX1nUvJc4SfW03u+kTVbRl+KmYuISDYFJCqomBPsQhd2xayjvxcnWa9njZUEir4ALcfwlLQLo+S6toLVAhbxWM9kECJt+s7Ox+fwyjHf6b3TUWzbirkwTLsQHrr7nqy4Z1bebaQFCHILhsb7IBkYgfSL9EJ3cwoFxIrJ4Cnl4r2YquSF2gnFf/9KCWSU+4I4b19ztlNMUCe5n9OCYqUEeIpZNu17nvUdzvf59OWzSu6j+DUFKkSaV7WHZiSzIo5911scfOMVjNrxaJZdcZ1myxARqTIFJCqgrzNPJFPOc8cr5jtJL3TBXuhiKOv1fGMl+3L3uK9349Mu1oq50ExmVrDRmsGC+PPp6uzOXEeaQheGWe3r8ifpWbw472eZFiBI3p1JyroIzN2P+e7mFAqIlZrBk3bhmuxfoarkye9+8n25QY9iv3/FBDIK/b6WI5CWth9zi2+W8jtV6HeiUNCsmGBQ2vc86zuc7/Ppy2eV9vuT3F9xHxSwEGlctRyasXptqusY+8I8Wv9wXepxSUREKksBiTJJCwBkXUgWUuhCJvciPF4mWdugmHXE/x+6+56Q8Xo5x0oWuijLF7DIV6cC0i/GCwVaWraaxNof/OCqDAnWvCgrpU/56gDEn098kZ4voJIWIEi70Mq6COzrkJKsbJZShmb0BrESF67J/hU62Sslo6HUPkH+gF3W72uhoGBW8CWrr/Hz/Sm+mRbAydpeWpHZ5O98MVlMyb8xad/hcpy05wuO5e4vyB4WIiL1q16GZiSzIhSEEBGpLQUkyiTfnd9SFbqQyb0IT6ttUMw64jYDmRfB5RwrWeiirJS78cVcjOcNxEQ/x35o597+9eUkJF+2RFr71v7Mvqv1P9+d+76eGBUTOCgkbf8X0560C1dY87tabNv7248spfy+FgoKZgVfsmpBlLP4Zqm1J9J+5/OtL7fNkP0dLqe03+81hkFR+e+JiJRXtYdmwKrMiM+0LOTgR69i1I5Hs9EfVmVFtGq2DBGRmhrU09NT6zbURGdnd085LrTb2obXTUG2UttQzPJx/yqlmBoMpezPUtdXzv7Vw3cgTV/6WK99SVPN/hWqDRI/V879lq9/5f6dr8XnXum/MaUYO3bEw8AH8i1jZucAewPjgaXAn4DT3f3NQusvx3GnnvZXpTR7Hwdi/+LMiF3H7c59C+7pHZpRDeecfyPb3f07Nhi0gvGvPd+bCdfXv3UD8fNrNvXSx2KOOSIDgQIS/VQvf9QqRf1rfM3eR/WvsdVT/4oMSPwv8FvgcaANuArodPfPFFq/AhLFafY+DpT+pQ3PmDxmSsWHZgA8efcDLL78EkYdeTQjrr+KtR77B8s3fy/rrtfW74DrQPn8mlm99FEBCZFAQzZERESK5O7fTPz3dTM7D/hNrdojUk/mLZrLdY9eyUETp1V9eEYyCLH48kvY/IV5PHP5JWz+9ZNZdsXljGqAjD8RkYFIAQkREZG++wjwWK0bIVIP4iBEV2d3VWbOSEoGIUYdeXTvTxWqFBGpbwpIiIiI9IGZ7Q8cA+xezPItLYNoaxver222tAzu9zrqXbP3sdn699jrj3Hp45dw1DZHc+wOx3LZ4y1M3+Zothu7HR/cdEplt33rfSy44ELGHfcVNjz+OJ694EI2PO4rbPeJXeGzH67INpvt88vV7P2DgdFHkUaigISIiEiJzOwA4BLgM+7+SDHv6e7u6fe45XoZ+1xJzd7HZuhfWn2Irs5uztnpp1yw50W0ty+rWB+TxXf/ff4FISvi/AvY+ZormDBlBkBF928zfH75NHv/oH76OHbsiFo3QaQuKCAhIiJSAjM7HDgX+LS7/63W7RGpljgQsbSzgycXPwFUb/rOOBCx9M12hj3zFK8t71ptaIaIiDQmBSRERESKZGYnAN8BPuHus2vdHpFqirMhthq1NZPHTKl4fYhkNsRrF1/MWo/9g3+vP5GF6xuP2cc4fY+dYY+dK7JtERGpDgUkREREince0AXMMrPeJ9193Zq1SKSCksMzcgtVVkIyCLHsisvpfOgBlgHX2sfY7t8d3L/LPrw50Zg+dUJFti8iItWlgISIiEiR3H1QrdsgUmlpNSIAztnppxXJhsgKQrz66YNY/FoHoz59EJ/acmsu22hzpk+dwLYbjSx7G0REpDYUkBARERGRXskgRDVqRCSDEMMPP7L350U+iAd2OJyd31iH8zcayfn7VyYrQ0REakcBCREREZEBLt/QjEpnRSSDEE+uN5HLdpnO9PUmMH1qWFbDM0REmpcCEiIiIiIDUL0MzXjh1DN7gxCX3f8CD8xfBMD5+79PWREiIk1OAQkRERGRAahaQzPiQMTKpR10PzFvjaEZySBEnA2hrAgRkYFBAQmpG3NeXcJl97+gglUiIiIVFGdG7Dpud6AyQzPSsiFatppE60475x2asa1qRYiIDCgKSEhFJYMMwBqP99xiDLOeXsj0qRPWuEOi4ISIiEh55BueUS6dj8/hlWuuYOghh2cWqmzdZlsALrthroZmiIiIAhLSP3HAIRlYgFWBh2SQAVjj8ZMLOlj8Theweppm7hhSERERKU1WEKJSwzPiIERXZ/caQYhR5/4snDPcMJfpUydoaIaIiAAKSEiO3IyGmTfN47DJ44E1sxuSgYNkYAHIOxY0+TgZyEimaepERUREpH+yghDlHJ6RO1vGitYWhh5yeG8QAladW3S808W8BR2AsiJERCRQQGKASRtCkTVsAkJgobNrZe/j5POwKmCQmyERv5Y7FjTt8X7bbrhGOzWGVEREpHTVmL4za7aMUef+jLEXX0J7+7LVzjfic4tJ40aw88TRutkgIiK9FJAYYNICDlnDJgBahwzuzZBIPh8/TgYOkoEFBRNERESqo1rTdxaaLSMpa+YM1YUSEZEkBSQGmLShEFnDJgBmTJtMe/syID27QURERGqrkvUhCs2WkTs0Ix7qmRuE0HmDiIikUUCiQWXNXlHozkPWEIq0YRMiIiJSnyo5NCNrSEbabBlpQzM6u1aqRoSIiBRFAYkGlTV7hQ7+IiIizSsORCzt7ODJxU8A5R+akS8IkTtbRu7QjNyhniIiIvkoINGgsoZeHB+dIGiMpoiISPOJh2dsNWprJo+ZUpGhGWlBiKR89SGSQz1FREQKUUCiQaUNvTj+hrmrnSD0ZUiHiIiI1Jd8wzP6I99sGblBCFg1PGPPLcYAqg8hIiL9p4BEmSTHUFbqor/QNpJ3KTSkQ0REpHFVcuaMUmbLSKsRATqfEBGR8lBAoh+S1aT7epAupThloW0k71KkDenQvN8iIiL1KysIUY6ZM0qdLSMtCJF2biEiItIfCkj0Q7KadPIgnUxpjKfTjJfPfZyWydDxThcj1hqyxrKlnAhkzaYhIiIi9SkrCNHXmTNKmS0jKV+NCJ1PyEA0ZMHDDH/oJyx/z14Me/bPLNvpFLrG7VjrZok0BQUk+iFZTTp5kI5rOTy5oIPF73T1Lp82hCItyNDxTlfmcAudCIiIiDSPSk7fWWi2jKRkVoSCEDJQxYGHZTudAtD7ePhDP2HYS/cw5D9zaFkezsuXfObaWjZVpGkoINEPWdWk4wN5boZE8rX4cVomQ+4wjtz3iYiISGOr1PSdpcyWka8+hIIQ0swKBR5i8eN4uWSGhIiUhwISFZAMMuy37Ya9zxc7hELDLURERJpbOafv7OtsGR3vdDFvQQeg+hDSvHKDDy23nMe6by2i9bVHe5fJDTwkAw7x8Iw4I2L5pIOr1XSRAUEBCREREZEqmLdoLtc9eiUHTZzW7+k7i6kPkSXOhpg0bgQ7TxytoRnSFIrNehj80j10rb89y8fvnjfwABqWIVINCkiIiIiIVEjazBldnd19Hp5RaMrOtKEZkL9GhEijSSsyWWi4RfyztbWFt3Y4cbWilAo8iNSOAhIiIiIiZZRv+s4hrS0cNHFaSesrZcrOJNWIkEZXSpHJYoZbAHQfeD1dOfXfRKR2FJAQERERKaN803deuOlFaxTDTtPXKTuzghCqESH1rBxFJjXcQqQxVT0gYWYtwNnAYcBawG3A0e6+MGP5TwLnApsBzwKnuPtt0WtbAv8LTAVGAi8CP3X3yyvcDREREZFe5Z6+s69TdmYFIVQjQupFWvBh8IolKjIpMkDVIkPi68A+wBTgDWAGcDXwqdwFzWwz4HfAUcBvgAOAG81skrvPB0YDs4ATgH8DHwT+aGZvuvvvKt8VERERGcjKOX1nKVN2JikIIfWolKyHFSoyKTJg1SIgcRRwhrs/B2BmpwHPmNkEd38hZ9lpwMPufk30/2vN7Jjo+e+5+4PAg4nl/2pmfwH2IAQyRERERCqmv9N3ljplZyxfkUoFIaSa+jrcIi34EFPgQWTgqGpAwszagE2Ah+Pn3P1ZM1sCbAfkBiS2Sy4beSR6Pm39w4Gdge+XqckiIiIiq8k3PKNYhWbLyBIHIjre6WLegg5ARSqleso93AIUfBAZ6KqdITEi+rk45/l2Qg2ItOXTlp2Uu2BUm+Jq4CXgqkINaWkZRFvb8EKLFdTSMrgs66lX6l/ja/Y+qn+Nrdn7J80ja+aMUoZndD4+h1euuYKhhxze79kyJo0bwc4TR6tIpVSEhluISLVUOyDREf0clfN8G7AkY/mCy5pZK3AtsCHwKXfvLNSQ7u6eoqpcF9LWNrws66lX6l/ja/Y+qn+NrZ76N3bsiMILyYCSb/rO5M9ixUGIrs7ussyWse1GafdyREoTBx+Wv2cvWl68lSE7nKjhFiJSNVUNSLh7u5m9CLwf+Cf0Fq4cCcxJectjwJ45z+0A3Bn/x8zWAq4H1gU+7u5Ly99yERERGWjyTd9ZSlZEslDlitYWhh5yuApVStUVynoY8p85DF6+iOGd3RpuISJVU4uilpcCp5vZLMIsG+cAt0azZuS6CviamR1ICDp8DtgROBTAzNYF/gB0EjIj3q5880VERKRZlWP6znyFKsdefElmVpAKVUo59LXI5PL37MW6L97Ksh1OVOBBRKqmFgGJswnTdc4GhgG3A4cAmNnBwCXuvi70FrzcDziXMD3oc8C+ieDF/oQZNd4GXjezeBvXuPsx1eiMiIgMLGb2/4CvEAosD3f3WhxLpcz6O31nVhCiUKHKrKEZKlQpxSh3kcm1PzidrjoZRiciA0PVT6LcvRv4avQv97VrCbUgks/9BfhLxrquBK6sQDNFRESyLAIuAtYmZP1JE+jr9J2FZssoVKgyOVtGMitCJElFJkWkWemujoiISAnc/VYAM9ujxk2Rfurr9J1p2RDFzJaRlDZbhoZmCKxeZHLYs38uariFikyKSKNSQEJEREQGjL5O31nMkIy02TKS5ry6hJk3zeOwyeM1W4YUVWSyZXkYwqMikyLSrBSQEBERqYKWlkG0tQ3v5zoG93sd9a7Sfbzu0SuZvfBBhrS2cOwOx3Lp4y0ctc3RBbf5yjVX0PnQA6xobWH944/jzYtbWO+YL7P29tvDh3bOfN+jLy7i/FnPcPyemzNz9kvc98wbAMyYNpndth5Xzq7VhWb/jvalf4NefojB9/2QlbueBtD7ePCj5zH4pXtobW0Jz0ePV374G6y8r4Ue25uV/kdadj2NdTfeCd67K+vGK00+LqNm//xgYPRRpJEoICEiIlIF3d09mbMrFKutbXi/11HvKtHHZFbEQROn0dXZzUETpzGhdQvO2uHHAKnbTGZFDD3kcLo6uxl6yOEsn7gl65z9U5YDy1Pel1aosrNrZW9WxGGTxzft59js39FC/csabjHkpXvo7OwOy0SPO3Y6JUyxucOJAL2Pu9bdBj51VVjhZp8PP6u0T5v984P66ePYsSNq3QSRuqCAhIiIiDSdcg/NGHXuz/LWhciaLSN3aMaMaZPr4mJISjdkwcO03HIeQ+IAgoZbiIj0mwISIiIiJTCzFqAVGBr9f63opeXu3lOzhklmECJZsDJLodkyshSaLUOFKhtTVqbD4JfuYXiU6ZCvyGTyfQo8iIhkU0BCRESkNF8Erkj8/+3o56bA/Kq3RnplBSEmjX5falaEZsuQUgtLtra29A6xiJ9LPk4GH5ZPOriKPRERaUwKSIiIiJTA3WcCM2vcDEmIMyN2Hbc7UHoQoi+zZcTDMzRbRmMoFHiIFcp06D7werqiITfKehAR6T8FJERERKTh5KsRkU++IERfakScv//7lA1RZ9KCD4NXLKH1tUd7l8kNPCjTQUSkNhSQEBERkYZTSo2IZFZEOYIQyawIqZ1Ssh5WrL89y8fvrsKSIiJ1RgEJERERaQjJrIhCNSIqPVuGsiKqK6vIZL7hFmnBh5gCDyIi9UEBCREREalbpUzfWUx9iCyaLaM+lFpkMvkzfqysBxGRxqGAhIiIiNSVUqfvLDRlp2bLqD/lKjKpwIOISGNTQEJERETqSjHTd5Zjyk7NllEd5RpuoSKTIiLNRwEJERERqQvzFs3lukevLGr6znJM2anZMspryIKHabnlPIbscCKg4RYiIlKYAhIiIiJSM2nDM7o6u1ODEJotoz7kG24x+KV7GN7ZDWi4hYiIFKaAhIiIiNRM7vCMIa0tHDRxWu/rmi2jttKCD4NXLKH1tUd7l0kGHlpbW1gWZUjEz2m4hYiIZFFAQkRERKoq3/SdF256Ea//9QEWn3lCybNlKAjRd6UUmVyx/vYsH7976nCL7gOvp6t9GaCsBxERKUwBCREREam4Yqbv7Hx8Dq9ccwUrFrWXNFuGpuwsLC3g0N8ikzEFHkREpK8UkBAREZGKy5o5o6+zZaRlQ2jKztIyHVRkUkREak0BCREREamIfEMz4qyIxWeesNqQjBWtLQw95PDU2TKKHZIxEJQSeID0gIOKTIqISK0pICEiIiJlFQcilnZ28OTiJ4A1h2ZkzZYx9uJLaI9qEIDqQkDphSWTP+PHWQEHFZkUEZFaUkBCREREyioenrHVqK2ZPGZK5tCMrNkyBmoQolyFJZXpICIijUIBCREREem3tOEZX+rejQ1+PYvhh/cUNVvGnFeXMPOmeby5dEXTF6eMgw8qLCkiIgOZAhIiIiLSJ1kzZ5w5/HC++etuVi69mc4Cs2U0e3HK3KyHllvOY8gOJ/YGH1RYUkREBjIFJERERKRoWUGIL3Xvxv6/m0fbl3YrOFtGviEZrUMGc9jk8Q1XnLLY4RaDX7qH4Z3dvcupsKSIiAxkCkiIiIhI0bKCEBtcP4v1vJ3W62etkQ0BxdeFmDFt8mpFLetRf4pMtra2sGyHE1cLPqiwpIiIDFQKSIiIiEheWdN3bnDmZZlBiFHn/iwEIW6Y27DFKStRZLL7wOvpqvOAi4iISLUoICEiIiKp0qbvjOtDDD+8BwrUheh4p6thilOqyKSIiEj1KSAhIiIiqeLhGR9rn8C0u9ZdrT5E7pSdjVKcslDWg4pMioiIVI8CEiIiItIrOTwjrhGxcc9Khv5rzaEZxdaFqJasYEPycaGsBxWZFBERqR4FJERERAa4ZBDijtv/j4/+cS537L2Ewx5ch/W8nZat3s3gaLaMJ9ebyGW7TGf6erWtC1FKYcnk42KyHlRkUkREpDoUkBARERmAsoIQn/vrSoY9D1v/dSXDjwrZEK9++iAuemOdmgQhylFYMvlYWQ8iIiL1QwEJERGRASgrCLH+USez7IrLGZXIhuh4pYt5CyofhBiy4GFabjmPYZt8ouyFJRV4EBERqT8KSPRDfOI0JJpPXERkICpm3L7+RtaHZFZEoSBEMhui3MUp82U9DH7pHtZ55VEVlhQRERkAFJDoh/jEaXhnt06ERPoheXGiC9fGk3UHO/lYfyNrJ2toxrEFghDQ/+KUpQy3gBBoaG1t4a1EhoQCDyIiIs1LAYl+iE+clu1wYq2bMmDowrWxFbo4GbR8MT3DRunzbTBZd7CTj/W7W31xIKLtmQV89C/PcsfeS9hrViejn4eJszp58kPlLU5ZSpHJfFkP3Qdez/L2ZSosKSIiMgAoINEP8YlTV/uyWjel6RRzV013yRpPobHgg1csYag+34aT7w52/HjkzQfrd7fK4myI9TrXZsLLIQjxu633Zbv//I7Htt6Xl0sIQvR1Os2sIpPKehARERFQQELqVClFzKRxFBoLnnvRI/Ul66K0mIyH+D3L37MXI28+uPf/cR2e5PqyHsfb6U87mt2/7r+R9l/8nLYvfbk3G+L59dv4x/qb8NjW+/Kp/T/CZRtt3huAgDWDEJWYTjOmwIOIiIgkKSAhdamYC9f4okYXIY2j0F3R5OulpvjHyy9/z169Y88h/WK10LqLuRscP87aXta2C7Uhd9v5LtiL2XY52hy3KeuitJiLzPizTWZKAL11eJLrS3ucHM6TFbAcqBe7yRoRSy67iC2eXszTl13EvTscy3b/+R3377IPb0601MDDla0/YdngU2CBptMUERGR6lNAQmoq353OfCe2pQ7dqNT49XKst9zrAFIvaNPuLue7EK7EBW8p/Ut+xvFFUr7+xcsP+c+c3ur8QOo60tadbHMxRRrjx1nby9p23P7kxX1W2yD/BXsx2y5Hm+M2FbooLUZusDG3Dk/WupPDeQrVrKg0M2sBzgYOA9YCbgOOdveF1WpDMghx85/P5aO3PsHNn1jEih33oWP5Ddy/4z7su2/Ihjh1y0Vs+/zZRQUeQNNpioiISPUoINFE6uXiuJTt9PWOa/KEudSxzckL0JYXby06XTx+nHXhmu+iOd9FeqF1FBMUyHcBPfile1hn6Zupd5fzXQhX4oK3lAuX5GdcqH/DO7tXGxKQ3F9p60hbd7LNhS54k4+ztpe17bj9yYv7rLZB/gv2YrZdjjYn29Tf9PvcYGOyDk++OhS5f5tqfEH8dWAfYArwBjADuBr4VLUakAxC7HbLO9gLMPyWd3j36dvSOvR2ttx5WzYf/DRXtv6EwV56YUkFHERERKQaFJAok3LcXS40PrrQ46w7roXel+8Cu9jx3aX2tb93XJMnzLkp4IXu+CUvQAcvX1RUunjycdaFa6G76vnu3OdbR7F3s5M/k49bW1sY9Nai1LvL+S6EoXIXvMVIfsaF+rcsygCJl09W509bR9q6k20upkhj8nHa9rK2nWx//Duf1TYofMFeaNvlaHNum2qhHtqQcBRwhrs/B2BmpwHPmNkEd3+hUhu965ZfMuiXl9Nz4JF8/M+LmPAirPfnRYz9/P50X3cp4w/Yi22ev4RhKx5m+fOXwPMqLCkiIiL1TQGJMinH3eX4ArGY4mFpj7PuuBZ6X74L7GLHd/elr/HP/t5xLeZudtYF6Lov3lpUunjWxXaxF8257yvUtlKDAvkuLLoPvJ6lT92Xenc530VnpS54+6JQ/4qZ6SbrgjYrkFFOpdTOqBf12KZ6YGZtwCbAw/Fz7v6smS0BtgMqFpAYct3FbD3/bZ647mK2n9LGihXvsP0UGD3oVobtNp/lg24tGPwCBR5ERESkfgzq6emp6gZLHXtrZp8EzgU2A54FTnH32xKvXw5MBQyY6e5HFtOOzs7unvYyTNfZ1jac9vZlZcmQSGYNxOn1xbwv+bgvWRa5F9ixIQseZtSj57G4AhkS9VKIMv78mlmz91H9a2z11L+xY0c8DHwg3zJmNh54EdjM3Z9PPP8C8N/ufk3We1euXNnT3d33Y+7DZ+3JovtfY/TUDfjAtHMYfN8PWbnraQC9j3s23qnP668XLS2D6e5eWetmVIz619jUv8ZXL31sbW0peMwRGQhqEZD4b2Aa8ElWjb0d7u5rjL01s82Axwnpsb8BDgAuBSa5+/xomRMAB44G3qxVQKIcqlW/oRT1dLFQCc3eP2j+Pqp/ja2e+ldkQKINWATs4O7/TDy/GPiiu9+c9d7+HneSQeJ6OUZUQj19JypB/Wts6l/jq5c+FnPMERkIBtdgm0cB57j7c+6+GDgN+KSZTUhZdhrwsLtf4+4r3P1a4JHoeQDc/WfufiuwpBqNr6Q4RbqZTzRFRBqZu7cTMiTeHz8XBc9HAnMque2ucTuGIUo6RoiIiEiTqGpAImvsLSGYsF3KW7ZLLht5JGNZERGRargUON3MNjWzkcA5wK1x5p6IiIiIFKfaRS1HRD8X5zzfTri7lLZ82rKT+tuQlpZBtLUN7+9qaGkZXJb11Cv1r/E1ex/Vv8bWoP07GxgNzAaGAbcDh9S0RSIiIiINqNoBiY7o56ic59tIH3LRUcKyJenu7inL+LF6GYdWKepf42v2Pqp/ja2e+jd27IjCCwHu3g18NfonIiIiIn1U1SEbfRh7+1hy2cgO0fMiIiIiIiIi0qCqnSEBq8beziLMspFv7O1VwNfM7EDgeuBzwI7AofECZjaUEFhpAXrMbC1gpbuvqGgvRERERERERKTPajHLxtnAHwhjb18hBBIOATCzg81sabxgVPByP+BbhGEa3wL2zQle3Aa8Ha3jsOjxbZXuhIiIiIiIiIj03aCenp5at6FWXgdeqHUjRESkbkwAxlZw/TruiIhIrNLHHJGGMJADEiIiIiIiIiJSI7UYsiEiIiIiIiIiA5wCEiIiIiIiIiJSdQpIiIiIiIiIiEjVKSAhIiIiIiIiIlWngISIiIiIiIiIVJ0CEiIiIiIiIiJSdUNq3YBGZWYtwNnAYcBawG3A0e6+sJbt6gszOwfYGxgPLAX+BJzu7m8mljkU+A6wITAXONbdH65Bc/vFzAYDfwWmAuPd/eXo+Ybvn5l9FDgT2AZ4B/iNux8bvdbQ/TOzccB5wIcJf7ceBU5298ei1xuqf2b2/4CvANsBw919SM7rnwTOBTYDngVOcffbEq9vDlxM+B4vAn7q7udWqfkF5eufme0FfBXYFmgBHge+6e73JZap6/7Vgo459f07naWZjzmg4w4N0r9mP+aAjjsijUwZEn33dWAfYAqwcfTc1bVrTr90A4cA7yL8Id8YmBm/aGYfAn4OfBkYDdwA/NnMRla9pf13MrAs+UQz9M/M9gCuB35M+Bw3Bi6PXmv4/gEXAesBWwIbAP8A/mhmgxq0f4sIfTop9wUz2wz4HfADYFT080Yzmxi93gL8AXgSGAt8BjjdzL5QjYYXKbN/hM/ofGBzQvuvA24xs/HQMP2rBR1z6vt3OktTHnNAxx0aq3/NfswBHXdEGpYCEn13FHCOuz/n7ouB04BPmtmEGrerZO7+TXd/1N073f11wh2BPRKLTAd+5+63ufty4EfAcmDf6re278xsS+BYQpQ8qRn69wPgYne/3t2Xu/s77v5I9Foz9G9z4LfuvsjdVwC/IJz8vosG7J+73+ruvwSeS3l5GvCwu1/j7ivc/Vrgkeh5gN2ACcA33H1Z9DlfAhxTjbYXI1//3P1ad7/R3dvdvcvdf064Sz45WqTu+1cjOubU8e90miY/5oCOOw3Tv2Y/5oCOOyKNTAGJPjCzNmAToDc1z92fBZYQ7vY0uo8AjyX+vx2r97WHkLrYMH2N0mZnEE4M23Nebuj+mdk6wE7AEDN7xMwWmtndZvaBaJGG7l/kR8D+ZjbWzNYiXJz9NUpXb4b+Ja3Wn8gjrOrPdsC/3H1pxusNxczeB4whpDxDk/WvHHTMabzf6WY+5oCOOzRH/2ID6pgDOu6I1BsFJPpmRPRzcc7z7UC9pusVxcz2J0SET0w8PYLG7+uJwAJ3vzHltUbv32jC7/KBhPHlGxHGl/85upBp9P4B/I0w7vM/hLsa+xHuUEFz9C+pUH+apr9mtj4h1fnH7v509HTT9K+MdMxpvL428zEHdNxphv7FBswxB3TcEalHCkj0TUf0c1TO822EO1YNycwOAC4DPpNIu4TQ34bta1So6FTguIxFGrp/rPo+XuHuc6LU0h8ArcAuNHj/ojuNdwD/IvRjOHAWcJ+ZbUCD9y9Fof40RX/NbCNgFuEi5huJl5qif2WmY04D9XUAHHNAx52G7l+OAXHMAR13ROqVAhJ94O7twIvA++PnoqJAI4E5NWpWv5jZ4YTxcp9291k5Lz/G6n0dBGzP6im29exDhCJFj5vZQkIaHsAcMzuWBu9fNJ58PtCT81JP9K+h+0coKrYpcL67L4nGuF5O+Ps1lcbvX67V+hPZgVX9eQzYMkqZTnu97kXF0u4DbnH346J051jD96/cdMxpuN/ppj7mgI47NH7/kpr+mAM67ojUM0372XeXEirwzgLeAM4BbnX3+TVtVR+Y2QmEqas+4e6zUxa5DPiLmV1J+GN+AmHaubRU1Hr0G8KdjtjGwP3Ax4GnCCf0jdw/CJWlTzSzXxLu6JxCKLD1d0Lkv2H75+4LzexfwLFm9nVCvw4lpFjOARbSYP2LKnq3AkOj/68VvbQcuAr4mpkdSKhg/zlgR0KfAe4FXgD+N9ofBhzN6invNVWgf0b4fZzp7t9KeXvd969GdMyp49/pHAPhmAM67jRM/5r9mAM67og0MgUk+u5swhjK2cAw4HbCNGaN6DygC5hlZr1Puvu60c+/Rnd1LmPVfNt7uXtDpLK5+zIS066ZWfy9XxAVMGro/kV+TDhRuotwUvQo8KnoLlYz9O+zhAJjLxBOOJ4BDnD354DnGrB/XwSuSPz/7ejnpu7+rJntR5gTfgahYvi+8YWnu3eb2acJd5ffIIxz/ZG7/6pKbS9GZv+A04F3AyeZ2UmJZY6OKqE3Qv9qQcec+v6d7jVAjjmg404j9a/Zjzmg445IwxrU05ObbSciIiIiIiIiUlmqISEiIiIiIiIiVaeAhIiIiIiIiIhUnQISIiIiIiIiIlJ1CkiIiIiIiIiISNUpICEiIiIiIiIiVaeAhIiIiIiIiIhUnQISIiIiIiIiIlJ1CkiIiIiIiIiISNUpICEiImVnZjPN7MwKrXusmT1lZmtXYv15tjvfzD5axe09ZGaTqrU9ERERkWpTQEJERBrN14GZ7v527gtm9hsz+7iZDTOzBflWUu0AQx/8GDij1o0QERERqRQFJEREpGGY2TBgGnBNxiI7Av8AtgUer1a7KuRmYE8zG1frhoiIiIhUwpBaN0BERKrLzE4HTgBGAq8Cx7r7nWY2GDgNmA60AXcCx7j7m9H7vgicCawL/CRa7kh3v6OIbe4dvXci8ES03jnRa/OBC4BDgQnAX4Bp7v5OyqqmAO3u/nLKNkYDg9z9TTP7AvBInvZcDWwC/MHMuoEz3P2HZvYZ4AfAu4F/Al929ydT3r8V8Gfgm+7+y772z8zGADOBDwErgXnA7u6+Mnr9YeATwJVZfRERERFpVMqQEBEZQMzMgOOAye4+gnCxOz96+Xjgs8DuwEbAIuDC6H1bAz8Hvhi99i5g4yK3uQMwAzg6et8lwM1RtkPs88AngU0J2Q2HZazufYDnrP8jZtYOvARsHD0+D/iKmbWb2e65K3H3LwIvAp9293WjYMSWwC+Bk4CxhIDDH8xsaM723g/cChwfBSP6079TgZej7W0AfBPoSbzvSWC7jH0hIiIi0tCUISEiMrB0A8OArc3sdXefn3jtGOC4OPvAzL4LvBhlRnwO+KO73xu99j+EwEYxjgIucfcHo/9faWbfBHYG7ome+5m7vxqt+w/A9hnragM6kk+4+51Am5ldA/wWuAN4FNgurc5EHl8A/uTut0ft+DFwIrALcHe0zK7Al4BD3D1+rj/96wQ2BCa4+zPAfTlt6oheFxEREWk6CkiIiAwg7v6MmZ0EfBeYZGa3AqdEF8sTgBvNbGXiLd2EO/cbETIQ4vW8ZWZvFLnZCcA0Mzs+8dzQaJ2xZAHKZTmvJS0CRiSfMLOXCcNIRgB7A62E49u/zWyGu59SZDs3Al6I/+PuK83sJcLwjdgxwD2JYAT0r38/InwWt4XkFS5197MTy44A2otsv4iIiEhDUUBCRGSAcffrgOvMbCRheME5hKEYLwFHuPvfct9jZv8Gtkr8fzhheEIxXgLOcvez+tt2YA5wcvIJd9/YzHYGvufunzCznwLz3P3yAuvqyfn/q4QhIQCY2SBgPPBKYpljgNPN7KfuHrejz/1z9w7CsI1TzWwb4C4zmx1lfUDY51kFPEVEREQamgISIiIDSFRD4t3A34B3gLeBlujli4GzzGyau79gZmOBXdz9JuB64EEz+xDwEGE6ymLrEF1GyLy4I3rvcGAP4N7ogrwUDxGGZ7zb3ZOBgh1ZVcTy/RRXBPI1YLPE/38DfN3MPgLcSxiusRz4e2KZDkItiDvN7Gx3/3p/+hcVw3wKeBZYTMhIWRm9tlbUr2lF9EVERESk4aiopYjIwDIMOBtYSBhGsD7wjei18whTTd5mZh3AA4RZLXD3ecBXgOuAfxOGTqwx00Uad/8HYUaOC6L3PUN20cpC61pBmJXikJyXdgQeibIa3kuYraKQHwDfigpfftXdPVrv+YT982lC0csVOW1oBz4GfMrMvt/P/m1BqHmxFLgfuMjdZ0WvfRq4O649ISIiItJsBvX05GasioiIFBZNZ1nUtJ9l3u5YQvHHHUosWtlQzOxB4Evu/nit2yIiIiJSCRqyISIiDcXdXydkQTQ1d59S6zaIiIiIVJKGbIiIiIiIiIhI1WnIhoiIiIiIiIhUnTIkRERERERERKTqFJAQERERERERkapTQEJEREREREREqk4BCRERERERERGpOgUkRERERERERKTqFJAQERERERERkar7/5nDuPSXW8E6AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 864x576 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set_style(\"darkgrid\")  # darkgrid, whitegrid, dark, white and ticks\n",
    "plt.rc(\"axes\", titlesize=15)  # fontsize of the axes title\n",
    "plt.rc(\"axes\", labelsize=14)  # fontsize of the x and y labels\n",
    "plt.rc(\"xtick\", labelsize=13)  # fontsize of the tick labels\n",
    "plt.rc(\"ytick\", labelsize=13)  # fontsize of the tick labels\n",
    "plt.rc(\"legend\", fontsize=15)  # legend fontsize\n",
    "plt.rc(\"font\", size=13)  # controls default text sizes\n",
    "\n",
    "colors = sns.color_palette(\"deep\")\n",
    "fig = plt.figure(constrained_layout=True, figsize=(12, 8))\n",
    "subfigs = fig.subfigures(nrows=2, ncols=1)\n",
    "\n",
    "fig.supxlabel(\"seq len (# tokens)\")\n",
    "fig.supylabel(\"latency (s)\")\n",
    "fig.suptitle(f\"Small seq len and greedy search on {model_name} don't tell the whole (inference) story...\")\n",
    "for row, (plot_name, timings) in enumerate(all_timings.items()):\n",
    "    subfigs[row].suptitle(f\"setup #{1+row}: {plot_name} (seq len / beam search)\")\n",
    "    axs = subfigs[row].subplots(nrows=1, ncols=2)\n",
    "    for col, accumulated in enumerate([False, True]):\n",
    "        plot_axis = axs[col]\n",
    "        for index, (k, v) in enumerate(timings.items()):\n",
    "            axis = range(len(v))\n",
    "            color = colors[index]\n",
    "            v = np.array(v)\n",
    "            # remove extreme values\n",
    "            p99 = np.percentile(v, 99)\n",
    "            v[v > p99] = p99\n",
    "            v = np.cumsum(v) if accumulated else v\n",
    "            plot_axis.scatter(axis, v, label=k, s=2)\n",
    "\n",
    "        title = f\"latency for the full sequence\" if accumulated else f\"latency for each token\"\n",
    "        plot_axis.title.set_text(title)\n",
    "\n",
    "# legend deduplication\n",
    "handles, labels = plt.gca().get_legend_handles_labels()\n",
    "by_label = dict(zip(labels, handles))\n",
    "fig.legend(by_label.values(), by_label.keys(), bbox_to_anchor=(1, 1), loc=\"upper left\", markerscale=5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Profiling model at the kernel level\n",
    "\n",
    "Below we reload the decoder model with `ONNX Runtime` kernel profiling enabled.\n",
    "It will help us to understand on which part of the computation graph the GPU spends its time.  \n",
    "\n",
    "The number of events that `ONNX Runtime` can save is limited to [1 million](https://github.com/microsoft/onnxruntime/blob/a4b5fa334aa939fb159bdc571ed3d56ca8d31fc7/onnxruntime/core/common/profiler.cc#L10).\n",
    "It is not an issue as we have seen that timings per token are mostly stable, so having only n first token information don't change anything.\n",
    "\n",
    "The main information it gives us is that 30% of the time is spent on matrix multiplication when caching is used.  \n",
    "The rest of the time is spent on mostly memory bound operations:\n",
    "* element-wise operations which require little computation (`add`, `mul`, `div`, etc.)\n",
    "* copy pasting tensors `GPU` <-> `GPU` with little transformation in between (`transpose`, `concat`, `cast`, etc.)\n",
    "\n",
    "It matches the information provided by both `nvidia-smi` and `Nvidia Nsight` (the GPU profiler from Nvidia): the GPU is under utilized.  \n",
    "That's why we think that a tool like `TensorRT` which will perform aggressive kernel fusion, reducing time spent on memory bounded operations, should be a good fit for autoregressive models.\n",
    "\n",
    "> there is a nice opportunity to increase the speedup by reducing the number of casting operations. We keep this work for the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-06-25T23:32:08.273218Z",
     "iopub.status.busy": "2022-06-25T23:32:08.272993Z",
     "iopub.status.idle": "2022-06-25T23:32:24.271872Z",
     "shell.execute_reply": "2022-06-25T23:32:24.271352Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAikAAAExCAYAAAC5/jJ1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/R0lEQVR4nO3debyd09n/8U9OCCI4Womhhhi/fuhTfWhVi6K02qipFB1Qs1bVUEPVPFRCS7WouaFmfcxqLFFDFdFKTZcaTiiJBIkknIY45/fHWrtu2xn2mfaQfN+v13ntve9h3de+d9jXvta67zWovb0dMzMzs3rTVOsAzMzMzDriJMXMzMzqkpMUMzMzq0tOUszMzKwuOUkxMzOzuuQkxczMzOrSfLUOwMwMQFIl90PYBBgL/DEifjqwETUOSd8GhkbE2FrH0hFJI4GXgG9GxC01DqdLksYBb0TE9rWOxZykmFn9WL/wfCHgHuAk4NbC8qeBbYE3qxhXI/g2sAQpgTObazhJMbO6EBEPl55LGpafvlBcnv29elFZT0haKCJaq3i8wcDgiHivWse06nKSYmYNRVILhe4eSWOBtYBjgdOAkcC9wPeBTwAXAJ8HngF2j4gJhbaagMOAPYHlgInAyRFxSTcx7AEcAqwIvAM8BfwwIp4qdG18F/g6sA3QCpwdEceXtbMWMAbYKC+6HfhxREzO6zfO72UT4Ee5vSnALyPinML7/1Z+XuoyOz4ijusk9vYc+wr5HDUBfwAOKX7ZS1oeOBX4KrAgcD9wQEREXl96n98DvgZsBTwGbNbVuSu0vwlwE/DbiDgyL9sTOAhYBZicz9mphX3Gkj7rk4CTgdWATfPnsRbwM+BXwMqkZHafiHiqsH+vPm+rHQ+cNbO5wfLACcBRwN7AF4Hzgavy3/akH2VXSRpU2O+3eZ/zgVHA9cDFkrbs7ECSNgLOJX2xfx3YHXgIWKxs09OAd/OxLwCOlfSjQjurAA+SEoDvAbsBawI3l8VI3v8JUlfXOOBsSZ/P604kJTJ/J3WZrQ9c2Fn82SHAsqRE6iTSOTu5ENsngAcAAfuSupMWBu6WtFBZW78EZgI7AL/o5ril9r9G6sY7tZCgHAr8DrgB2DI/P1HS/mW7jyQlT6eQzv9LefnypHN+MrAzMAK4uq+ft9WWKylmNjf4BLB+RLwAIOl/gEOBXSPi0rxsEOmLcXXgmZwk7Af8oPBL+m5JS5OqMp0N8Pw8MCEiTiksu6mD7Z6KiH3y8zskjQCOlPS7iGjLx5gMfL1UwZA0AXgW+AYfHYtzZUSclLcZB3wT2A54JCJekPQW0NRB11hnZgI75Dhuk7QA8HNJp0TEW6RqxsLA2vk1kh4EWkhJ2dmFth6OiB9RIUlbAdcAR0XEL/OyRfP5OKlQbbpL0lDgqHzOPsjLPwlsFhH/KLQJ6d/AlyLiX3lZEykJEfBsHz5vqyFXUsxsbtBSSlCy5/PjPR0s+1R+/ArQBlwvab7SH/BnYO083qEj/wA+K+kMSRtJGtLJdteXvb4OWIZUwYDULXI90FY49kukRGDdsn3vLD2JiPeBfxXa6Y0bc4JSjG0hUpdJKba7gBmF2GYC4zuI7VYq9y3gWlLX0i8Ly9cnJUXXln0W9wBL8tH3+moxQSloKSUo2dP5sbRvbz9vqyEnKWY2N5he9vq9DpaXli2YH5cABgNvA+8X/saSqsxLd3SgiLgb+AFpHMk44A1JZ0tauGzTKZ28LrW7BHB42bHfB1YijZfo7v0tSO9VEtuOHcS2SQexvd6D424FvMXHE7gl8uNTZce7Ny8vHrOz400ve90vn7fVlrt7zGxe9RYwB/gS6Rd2ufIv8v/K3QWXSBpO6nY5g1RpOKKw2Yiy3UqvJxWOfz0djx95o7vg+6iS2G4ijXcpN7PsdSX3tyn5MXAwcKekL0dE6VLyt/LjlnSchEQvj1fU68/basdJipnNq+4h/bJeLCLu6k0DETEVOE/SdsAaZau3JQ3+LNmOlAT8O7/+M2mg7PiI6O0Xb0lPKytbS/pZoctnO9IVSE8WYvs2aVxNf15SPIN0JdB9pHE6m0bEDOCv+fjLRERPuo96os+ft1WfkxQzmydFREg6l3TFz6mky2cXJCUOq0XEnh3tJ+l40iDNcaSKx2eBL/PRKgrAmpLOA/6P1DW0B/CTQmJwHPAIcKuki3NbnwI2B8ZGxLgevJ1nSYnHNqQk6LWIeK2L7Rchjf+4IL/fo0mX+5YqGqeTrji6R9JvgVdJY0O+DDwQEVf2ILaPiIg3JW1OuqT5FklbRMR0SccBZ0paAfgLaTjCasAmEbFtb49XOG6vPm+rLY9JMbN52Y9IXRq7AH8ijU8YRfqS7MyjpKrJucAdpCtGjgPOLNvuMGBRUpKyTz7OWaWVEfEc8AXSZcrnA7cBxwOz+XCQb6XOIQ2uvTjHt3c32/+KVNW5EjgGuAg4shDbGzm2Z0ldWXeSLvtdDJhQ3lhPRcQk0kDWkcB1kobk+6HsTbqs+MYc23dJyUx/6c3nbTU0qL29r1VGMzMrqfd5avLN3H4cEWd1u7FZjbmSYmZmZnXJSYqZmZnVJXf3mJmZWV1yJcXMzMzqkpMUMzMzq0u+T4qZzTPyJIN/B84oTTIn6TDSRH3jqhjHENJsvV8gzYWzYESUz3xc2nZr0kzFqwIvAsdHxNXdtD8WWCsiyufZqSuSNibd+v7TEfFkN5sj6VOku8/+T0S8OMDhWR1wJcXM5iXfJt2I7YrCssOAjascx1BgT9I9Uh7qbCNJG5Dus3Iv6f4htwJXSvpqNYKsNxHxKnA16d4uNg9wkmJm85IDgD/kmYRrJiKmA5+IiK/x8cn2io4G/hIRB0TEvRFxKHA7NfqSlrRQLY5b5vfAzpI+WetAbOC5u8fM5gmSVgG+COxfWNYCfBI4VtKxefEmETFO0lBgNKn60gz8E/h5RNxZ2H8c6Xb2d5Lu2LokaY6YvfOv/k51N1+PpAVIsw4fULbqKuD3khaLiLe7aqPQ1pC837rAphHxvKTlSXeR/Srp9vD3AwdEROR9RpJuSvc90nw7WwGPSdozL9+RdNfYnUiTDl5E6opqKxx3LWAMaVoASAnWjyNichex7gEcAqwIvEOaGfmHEfFU3uRB0mSBOwFnV/L+rXG5kmJm84qvkL70nigs2xZ4m/QFu37+ezyvuwD4AWnsyLbAK6R5djYoa3d9Ppzddw/gf4Ab+iHelYH5SbemL3qGD+e16ZakBUnVms8AG+YE5RPAA4CAfUmJ2MLA3R1US35JSkJ2AH5RWH4qMAvYHriMVN3ZvnDcVUgJxYKkRGc30jw5N+exQR3FuhFpuoE/kLq3did1hy1W2iYndw8Dm1Xy/q2xuZJiZvOKdYBnir/0I+LvkuYA/46Ih0vLJf0/YGfgB4UBtneQ5q05mlRZKBkBrB8RL+ftJgIP5Inzbu9DvIvnx+lly6eVre9UrgbdBCwLbFSo7hxESkrWLk0qKOlBoIWUGBQrFA9HxI8KbY7MT/8SEYfk53dJ2oI0m/I1edmxwGTg6xHxXt53Ainp+gZpfE25zwMTIuKUwrKbOtjuCWCvLt+8zRVcSTGzecVSpK6ZSnwOGARcW1qQk5trgfJKyuOlBCVv9yAwhfSFW0sLk7pXRgBfLut+2gy4C5ghaT5J85GqJeNJXUJFHSUTkLq4ip4mJUPFY1wPtBWO8RIpEersqqN/AJ+VdIakjXI3VUfeAEZ0VpGxuYeTFDObVyxImmG4EksDsyLi3bLlrwND83iRkikd7D8lt9EXpYrJYmXLFy9b35llSGNwro+I18vWLUEaU/J+2d8mwHJl25bvWzK97PV7pHNcPMbhHRxjpQ6OAUBE3E3qYtsIGAe8IelsSQuXbTqb1BPg3oC5nD9gM5tXvEWqplRiEjBM0tCyRGVJ4N2IKCY7IzrYf0Ruoy9eIH2prw7cV1i+OtAGPNfN/v8CzgTGSpocEb8rrHuL1I1yYgf7zSx73du5U94iVVIu7GBdpxWt3L12iaThpO6jM3JMRxQ2ayYlkTW9SssGnpMUM5tXBGmQa7nyCgDAo6Qv5+2BS+G/N4LbnjTgtOh/JS1fGJPyJVKS8kifgo2YLele0oDV8wqrdgT+WsmVPRHxB0nDgLMkzYyIy/KqP5MGyz4VEa19ibMLfyYNlB3f3ZVMHYmIqcB5krYD1ihbPZLukzSbCzhJMbN5xYPAMZKG5y/AkmeBUZJuJ12tEhHxjKQrSV/ui5CqGnuRqhj7lbU7lXTVz7GkZGcMaZxKl4NmJX2dPHg1vy5dGfNoREzMz08Exkn6NemKoW/kvy0qfdMR8bucqPxe0qyIuAE4nXTFzT2Sfgu8SqoSfRl4ICKurLT9LhxHStRulXQxqXryKWBzYGxHd/iVdDzpZnvj8vafzTEdUbbpuqTP0+ZyHpNiZvOKcaQuiPIv+ENJlybfSqqgrJOX7wVcQrq09kZgBWDLiCivpDxEuhrm16RLmZ8Etqkgnt+RBuLukV9fm/82KW2Qj7U9aRDqHaR7lXyneK+WSkTEacApwFWSNo+IN0i35H+W1J1yJ+mS4sVIVzD1WUQ8l4/xLnA+cBtwPGk8yfOd7PYoqWpyLun97kdKds4sbZC7gdYh3YnX5nKD2tt7291oZtZYJJ0JrBIRo/qpvXHAGxGxfXfbWv+QtA/wU2C13nQjWWNxJcXM5iWnAZtIquhGaFZf8rignwAnO0GZNzhJMbN5RkT8m3Szsr5eHmy1sRRwOemOtDYPcHePmZmZ1SVXUszMzKwu+RJks15qb29vnzOnrfsN69DgwYP44IPGrKI69tpp5Pgde+0U459//sFvAMMr3ddJilkvtbfD9Onld01vDM3NQx17DTRy7NDY8Tv22inGP3z4IhO72fwj3N1jZmZmdclJipmZmdUlJylmZmZWl5ykmJmZWV1ykmJmZmZ1yUmKmZmZ1SUnKWZmZlaXnKSYmZlZXfLN3Mz6YPjwRfqtrdbZc5g1o7Xf2jMza3ROUsx6qalpECOPuLXf2msZPYpZ/daamVnjc3ePmZmZ1SUnKWZmZlaXnKTYXEHSWEkX1joOMzPrPx6TYlUnaR3gSGBDYCjwBjAeODsi7qlg/3HA3RFx0kDGaWZmteVKilWVpM2BB4EXgHWBRYBPA1cA29YwtI+QNH+tYzAzm9e5kmLV9jvgsog4rLBsJvB/+Q9JOwE/A1YE3gFuAg6OiHcknUWqwKwv6Qjg1YhQbmcBSRcAO+T9ToiI80oHkbQhcAqwBjANOAc4PSLaJW0M3A38ADgeGE5KoMzMrEacpFjVSFoNWBnYp5tN3wa+AzwDrERKUo4CfhYR+0tai467e7YHdsztbwNcLen2iJgoaQ3gT8D3gFuAVYHbgKnApXn/wcA3gM8C7/fhrfZac/PQqhxn8OCmqh2rvzn22mnk+B177fQlficpVk3D8+OrpQWStiIlCYOABSJiwYi4rbDP85LOAXapoP17IuKm/Pw6SdOBtYGJwA+BayPixrz+2VyV2YUPkxSAwyPi7Z69rf4zffq7VTlOc/PQqh2rvzn22mnk+B177RTj7+kNMJ2kWDW9kR+XBZ4FyElFs6QNgPvhv+NWjgFWBxYgVTimVND+pLLX7/Bhl82KwKaStiusbwJeKbxuK3ttZmY15CTFquk54EVgJ9L4j4+RNAS4ATgMuDgiWiXtD/y0sFlbL449Mbf3oy62aY+I9l60bWZmA8BJilVNHqD6I+BGSW8CZwH/BhYC1subDSFVT6blBGUNYP+ypiYDq/Tw8OcA90m6HbgdaAdWA4ZHxH29ekNmZjagfAmyVVVE3A5sQEoQHgdmAU8BXwI2jYhZwH7AqZJmAWeTLk8uOgNYV9J0SU9VeNwngS2BA0ndQlOAsXw4TsbMzOrMoPZ2V7fNeqm9vycYnDp1Zr+115VGHojn2GunkeN37LVTNnB2POkeWRVxJcXMzMzqksekmPVSW1s7LaNH9Vt7rbPn9FtbZmZzAycpZn1Qre4ZM7N5kbt7zMzMrC45STEzM7O65O4esz7o6S2eK9U6ew6zZrQOSNtmZo3CSYpZLzU1DaI/L0Euahk9ilkD0rKZWeNwd4+ZmZnVJScpZmZmVpfc3WN1Q9IRwEHAwsAmEfFojUMyM7MacpJi/UbSSsAYYENgGDANeAzYMSLe62bfZYFfAGtFxNN5WTuwYUQ8MKCBm5lZXXJ3j/WnP5Em7xOwCLA+cAcwqIJ9RwJtpQTFzMzMlRTrF5I+SUpOtouIt/PifwPnFrbZjzQL8VLAM8ChEXG/pB1JMxIPzjMfvw7/vbjlTkltwFURsaekFuBC4CvA54CXgO8CawInkmY1vhbYNyLm5OP+HtgMaAZeAU6KiCvyuj3yfmtHxBRJI4B/AEdHxEX9eIrMzKyHPAuy9RtJTwIzSInJY8AzEdGe1+0M/BYYBYwHdgV+A6wRERMlbQzcHRHzFdr7WHdPTlLeB7YCngd+T6rY/Jk0nuWT+dgHRcTleZ89gBuA6cAOwKWkpKTUrXQJsAzwDeA24NWI2LWCt9yvsyAXtYwexfvvfzAgbQMMHtzEBx+0DVj7A8mx104jx+/Ya6cY//zzD+7RLMiupFh/2hg4mFQtWQuYLum3wEnAD4DzIuJveduLJO0JfAc4pYfHOT8ingGQdAWpkvKFiHgHeEfSONJ/BJcDlFVErpL00xxrqWtpP+BR4BFgflICVHMDOTV7I0/97thrp5Hjd+y1U4y/pzfAdJJi/SYi3gCOBI6UNBT4NnAB8CqwHHBN2S4v5OU9Nanw/F3gg4iYWrZsEQBJTcBxwI6kbqZ20tVDwwtxvyvpQuB0YPeIaNz/G5iZzUU8cNYGRES8GxFjgQnA2qSxICPLNlspL+9Mf/RF7gzsCXwLWDwimoEnKAzmlbQ6KZE5BzhF0lL9cFwzM+sjV1KsX0haHDiM1MUSpARja1K3z2jgYeBMSTcBjwPfJyUvO3fR7GRgVaAvlyAvCswBpgJNknYDPgPckuMeShpo++uIOFbSAsCVkjaLiIEbFGJmZt1yJcX6y3vACOA64C1SUnAUcEBEXJuvpjkeuAx4kzQO5BsRMbGLNn8OnCBpmqTzehnXJcDfSINsXwXWAO4vrD8bmJJjA/gxafDtcb08npmZ9RNf3WPWewN6dc/UqTMHpG1o7IF4jr12Gjl+x147ZQNne3R1jyspZmZmVpecpJiZmVld8sBZs15qa2unZfSoAWm7dfacAWnXzKyROEkx64OBHDdiZjavc3ePmZmZ1SUnKWZmZlaX3N1j1gc9nYeir1pnz2HWjNaqHtPMrFacpJj1UlPTIAbqPimdaRk9illVPaKZWe24u8fMzMzqkpMUMzMzq0vu7rGGI2ld0rxAXwIWIE1E+CdgTERM6kO77cCGEdGXCQ3NzKyfuJJiDUXS5qRZkQNYOyIWBb5MmrTwy7WMzczM+pcrKdZozgGuiIjDSwty9eREAEk7AT8DVgTeAW4CDo6Id/L6A4CDgCWAGcAlEXGkpCdyc3dKagOuiog9q/SezMysA66kWMOQtBqwCnBFF5u9DXwHaAY2zH9HFfYfDWwZEYsAa5KSGCLiM3n/r0bEMCcoZma150qKNZLh+fHVzjaIiNsKL5+XdA6wS349BxgErClpYkRMBx4eiEAHUnPz0D63MXhwU7+0UwuOvXYaOX7HXjt9id9JijWSqfnxU8AzHW2Qx6wcA6xOGlQ7GJgCEBEvSvousB9woaQJwAkRcedAB96fpk9/t89tNDcP7Zd2asGx104jx+/Ya6cYf09vgOnuHmsYEfEc8Dywc0frJQ0BbgCuApbPg2oPJ1VPSm1cFxGbk8akXAPcKKmU4rcPXPRmZtZTrqRYo/khcLOk14GzIuI1SUsCuwOvkKon0yKiVdIawP6lHSWJNKD2L0ArafxKO9CWN5kMrEq6esjMzGrMlRRrKBFxF7ABsAbwT0kzSUnFCOBuUlfOqZJmAWfz0UG2Q0hdQZOA6cABwLci4j95/c+BEyRNk3ReFd6OmZl1wZUUazgR8RiwTSerL8h/RSfk/f4JfLGLdn8P/L4fQjQzs37gSoqZmZnVJScpZmZmVpfc3WPWS21t7bSMHlXVY7bOnlPV45mZ1ZKTFLM+mDp1Zq1DMDOba7m7x8zMzOqSkxQzMzOrS+7uMeuDnt7iub+1zp7DrBmtNY3BzGygOEkx66WmpkGMPOLWmsbQMnoUs2oagZnZwHF3j5mZmdWlPicpkr4r6Yn+CKaLY4yVdGHh9VOSdiy83kLS85JmSjpY0pGSbu7jMedI2rgvbVgiaZykowqvZ0laf4CONeD/Hs3MrDoq6u6RtBIwBtgQGAZMAx4DdoyIy4HLByzCDkTEmmWLfgOcHhHnDMTxJO0GHBURqwxE+z0laSTwEvAcsGZEzMnLNwDuj4hBXexecxExrD/akTQWmBMRexbarvq/RzMzGxiVVlL+RJqUTcAiwPrAHUC9fBmuBEyodRDVIGn+wstPAvsOUNtmZmY11W0lRdInScnJdhHxdl78b+DcvH43ClUGSeOAx4EVgc2AKcDepITm18DywJ+BXSJiZt6nHTgI2A1YmVSl2Ssinu8kphbgKOAeUjVhMHCnpDbgf4HvABtExGZ5+6GkSea+BSwGPALsX2pf0iLAWcA3gZmkmXIrImlZ4EJgHdIsuxOAAyNivKTFgdeAL0bE3wv7/AW4KyJOlDQfcFh+7yOAp4Cf5En0StWC+YH3ga2Aq0lVLfJ7OlbSpRExo4PY5gOOzG0vTvpcfhIRT3bWtqTXSRWzx4DdSYnsycD/kSbf+1w+59+LiGdyOzsBPyN95u8ANwEHR8Q7nZyzdmDDiHhA0njSv6+SBYBbI2IbSV8BfgGsBswh/bs5ICKmSDoM+G7h+JA+2+/z0X+PQ4FTgO2AhUgzJh8QES/n9eOA8cBI4Kukf68HR8SNHcVuZmbV020lJSLeJH1xXihpF0lrSOqugvJ9YDTQTPpS/QMpUdmI9GUg4ICyffYGtufDL+qbJA3uJrbXCl0HX42IYRHxXAebXgCsDnwBWAr4G3BLoXLwa2BVYA3gf4CtSYlPJZqAc4AVctuPA9dJmj8ipgHXAv/tjpC0GqkSdXFedHw+3hakysjFwO05wSnZAbgNGA4cUlh+HRCkRKQjhwK7AN/Isd0P3CVp0W7a3gj4V97ne8BpwEXAj4BPAM+QuthK3iYlhs2kBGdDUhLZrYhYJ39uw4D1gBnApXn1bGD/HNungWWAM/N+p5K6dS4p7R8RH3RwiDNIn/sXSJ/RG8DNZf+2dgV+RUpyzgIuycmNmZnVUKWXIG8MHAwcCKwFTJf0W+CkTra/JiL+BiDpMtKv7NMi4q287BZg3bJ9flWobBxGGveyHvBQpW+mI5KWIH2BrhARr+dlx+f3sp6kh0i/yEdFxOS8/nBg20raz7/IXy4c7yhSArYq8DRwPulL8ZCI+A+wB3B7RLyak70D8rFfzE1cJOlAYBRwWV72QERcnZ+/KxULD/wUuEdSR+NxfgCMiYhnc2wnkBKmUcCVXbT9XESUBirfJulN4I5C5eQKCuM+IuK2wjGfz7Hs0ulJ64CkZUjJ0gkRcV1u94HCJpMlncqHyV0lbTaREpBvRsSredmBwFvA54G/5k2vjoiH8vrzgdNJn19DDMBtbu55PjV4cFOv9qsHjr12Gjl+x147fYm/oiQlIt4g/Vo/Mv/C/DapOvEq0NbBLpMKz9/tZFn5XbBaCsd7V9JUYNlK4uvGivlxQtmX+/zAcqRf6QsUj08alFqRnASdTkrkmvnwfAyH9EUr6TVge0lXkb40987bLEEaiHxz7gIpxlZ878XYPiIiHs5XMp0MnFe2ernie4mIttxVtlw3bU8qe/0uXXx+kjYndZGtTjqXg0ndJhXJ3W1/Aq6NiDMLy9chdfd8BhhK6jLsyaDb0mdbPAezJE0hnYNSkjKpsP6d/O+ktndp64Hp09/tfqMyzc1De7VfPXDstdPI8Tv22inG39MbYPb4Zm4R8S4wVtKPgbVJ3Rv9YWTpSU6EhpPGvvTVxPy4akRMLV+Zy/7v5eO/UB5LBU4BlgbWi4hJ+Qt3Bh8dVHweqYIyC/gAKN0B7A3SGI7NIuLRLo7RUSJYdASpalPexit89Lw25dev9KDtLkkaAtxAGldzcUS0StqfVOGpZP/5SeNdnutgn6uAPwI7RMQMSVsCxUvLu4t9KqnLaCRQqtINI3UpvtL5bmZmVg8qGTi7OOkL6HLS+Id20hiKtUjjThbup1gOyoMYX83tvkgaO9IneZDlFcA5kg7M3SzNwCakwauz8vrjJT0JtObjlxskacGyZe8Di5IqC9PyF+CYj+2ZxuScAhwL/L40diIi2iWdCfxS0p4R8a/cxpeAf0bEaxW+x5dyF8vRZavGAoflgbotwOGkz7w/b5M6hFStmJYTlDVI40gqdQFpQOs3I6K9bN2ipPEuMyUtT0rGiiYDX5DUFBEfS1hy5ehS4ERJTwPTSWNPniUNnjYzszpWySXI75F+eV5H6sufShoUeUBEXNuPsVyYjzGVVN7fupOBkL2xFynBGidpJvBP0oDR0pfiT0hdAs/mdTeTKh5FK5ESmOLfoaRujhHAm6Qrex4q3zcPoP1jfl8XlbV7LHAjcKOkGaQBq/vS8xvtndTBPqeRxp7cCbwObEoaYPyxK4F6KyJmAfsBp0qaBZwNXNGDJnYljU96M9/kbZak0niZvUljaGaS/m2U/3u7kJQkvylpeicDrQ8iXan0KGns0NLAVv34b8vMzAbIoPb28h+v1Ve8JLXWsQwUSceRLkX+aq1jsX7TXg9z90ydOrPH+zVyH7djr51Gjt+x107ZmJTxfPzCmU55gsEqkLQkqZqzd3fbmpmZWeIJBgeYpNNJ42tujoja/uw2MzNrIHVRSan3uWb6IiIOJt1jxuYybW3ttIweVdMYWmfPqenxzcwGUl0kKWaNqjfjQczMrDLu7jEzM7O65CTFzMzM6pK7e8z6oKe3eK4nxdhbZ89h1ozWGkZjZvZxTlLMeqmpaRC1vk9Kf2kZPYpZtQ7CzKyMu3vMzMysLjlJMTMzs7rkJMWQ1CLpe5UuNzMzqwYnKWZmZlaXPHDWuiVpN9LM178BDiPNPHwN8MOI+EDSAsBvgW2ABUkzLh9ZmiVb0u7Az4HhpBmfBwFzImI3SSNJM1AvFxH/Lh4vIlbJr4cCJwDfAhYDHgH2j4jn8/r5cly7kWakfgr4SUQ8JmkEaVqCoqHAwRHx6+7aNjOz2nElxSq1ArAksDLwOWAHYKe8bte87P9FxKLApqREAUkbAmcD+wKfAO4CduzhsS8AVge+ACwF/A24RdL8ef3xwNbAFsAngYuB2yUtHhFTImJY6S/H8QZwS4Vtm5lZjbiSYpVqBY6JiA+A5yX9mTTd9uXAe8AwYA1Jf42IVwr77QL8MSLuyq8vlbRPpQeVtATwHWCFiHg9LzseOBBYT9KDwAHAqIgoVUwuknQgMAq4rNDWpsBZwBYR8Xx3bQMPVBrn3KC5eWitQ6jI4MFNDRNruUaOHRo7fsdeO32J30mKAbwPdFQ5mD+vmw+YkhOUkneA0t3ALiNVWc4AVs0JzGG5y2RZ4LGydl/qQWwr5scJkspjWw5YgpQg3SypvWz9sqUXkj4N/BHYLSIerrDtecr06e/WOoSKNDcPbZhYyzVy7NDY8Tv22inG39MbYDpJMYAWYJXiAknDSN0fLwJrdrVzRMwBxgBjJDWTqhUXAxsBrwIjy3YZCZTGfJRm6Fu4sH6ZwvOJ+XHViJhafmxJg0gJ02YR8WhH8UlaFvgTqRJ0Q6Vtm5lZbXlMigGMBfaWtKGkwZIWB84E/gn8vbudJW0qaZ08jqOVlDSUqi5/ALaX9BVJ8+VLmtcr7RsRb5KShd3zsT8N7FVYPwW4AjhH0qfy8ZolbStpWES051h/KWnVvH6YpK9JWkbSoqQE5cqIOKsYd3dt9+wUmplZf3OSYkTE5cCRpAGubwFPAgsB38xVku4sSUpGpgGTSINs985t3wf8GLgwt70FcHXZ/rsCWwJvA6cDF5Wt3wsIYJykmaTkaQeg1L1zLOmqoRslzQD+RRog2wT8L/Bp4IeSZhX+flhh22ZmViOD2tv9/2KrLkkXAvNFxG61jqWP2uemuXumTp3Z/YZ1oJH75xs5dmjs+B177ZSNSRlPuuiiIq6kmJmZWV1ykmJmZmZ1yVf3WNVFxJ61jqE/tLW10zJ6VK3D6BetsysZemRmVl1OUsz6oFHGcZRr9D5uM5s3uLvHzMzM6pKTFDMzM6tL7u4x64Oe3uK5nnQUe+vsOcya0VqDaMzMPs5JilkvNTUNYm65T0pJy+hRzKp1EGZmmbt7zMzMrC45SbG6JWlsvjutmZnNg9zdY30maRywPvA+aWLBl4CTI+LaWsZlZmaNzZUU6y8nRsQw4JOkWZWvkLRKbUMyM7NG5kqK9auImCPpAuAMYG3geUnbAEcDK5NmST4pz7yMpJHAecB6pJmHXwJ2jojITS6Q29sBeAc4ISLOy/suS5pdeR1gCDABODAixuf1xwEb5uW7AK3AWRExuhSvpLWAX5FmS24FLgeOiYj3+/nUmJlZD7mSYv1K0hBgv/zyOUmbAxcBBwKfAHYFzpK0Ud7mF8DLwJLAEsBuwLRCk9sDN+d9f5z3XSGvawLOAVYAlgIeB66TNH9h/42A14Glga2BgyV9J8c6ArgPuA74FKnLanPgZ308DWZm1g9cSbH+8nNJPwUWIY1N2TMiJki6BTgzIu7P2z0i6TJSZeMvwHukBGOliHiGVPUouicibsrPr5M0nVShmRgRL5MSHAAkHQUcAKwKPJ0XTwLGREQ7MF7S+aRE6IocwxOlygzwqqRTgDHACX09IY2quXlorUPo0uDBTXUfY2caOXZo7Pgde+30JX4nKdZfTo6IkyQtTqqcbJIfVwQ2kXRwYdvBQClpOZTUFXSzpIWBPwI/i4jS7TomlR3nHVIihKQlgNOBjYFmoC1vM7yw/cScoJS0ANvl5ysCX8qJT8mgHN88q97n9GnkeYcaOXZo7Pgde+0U4+/pDTCdpFi/iohpkvYEXpC0NTARGBsRp3Wy/VRS9eMASSsBNwKHAcdUcLhTSN0460XEJEmLADNIiUbJCpIGFRKVkcC/8/OJwN0RMXdMZWxmNpdxkmL9LiLeknQ6abzJQcBYSQ8DD5GqFJ8GBkXEY5J2BB4hVTjeJnX/fFDhoRYF3gWmSRpG6qYptzRwqKQzgLWAvYBSVedS4BBJu5O6f94jJTGrRcTtPXrTZmbW7zxw1gbKmaQEYRlSYnAa8Aap++YMYFje7rOkwauzgKdIg187rLp04BhgBPAmaSzLQ3w8wbk/xzEZuCXHdQVAREwmdUttQ0qSpgHXAytV/jbNzGygDGpvb+9+K7MGlC9B3iAiNhugQ7TPjXP3TJ06s9ZhdKmR++cbOXZo7Pgde+2UjUkZD6xb6b6upJiZmVldcpJiZmZmdckDZ22uFRHHDWT7bW3ttIyeuy4Map09p9YhmJn9l5MUsz6o9/EbnWn0Pm4zmze4u8fMzMzqkpMUMzMzq0vu7jHrg57e4rmeVDP21tlzmDWjtWrHM7O5g5MUs15qahrE3HaflIHSMnoUs7rfzMzsI9zdY2ZmZnXJSYqZmZnVJScpNk+RdJSkcV2s30CS54owM6sDTlJsriDp55LaJe1a61jMzKx/OEmxhiepiTTT8lvA3jUOx8zM+omv7rG5wdeATwHbALdIWisingSQNAo4DVgeGAc8X9xR0qrABcA6wIvA76sWtZmZdclJis0N9gZui4hbJU0A9gF+LGll4DpgD+AqYFPgeuBRAEnzAbcAdwNfB5YFbq5++POG5uah/dLO4MFN/dZWtTVy7NDY8Tv22ulL/E5SrKFJWgbYEtghL7oIOF7SYcBOwCMRcVled6ekG0hVF4D1gJHAoRHRCvxL0q+A86sU/jylv+YKauR5hxo5dmjs+B177RTj7+lNJD0mxRrdHqSxKLfk15cBCwE7kiojLWXbv1R4viwwJSLe7WS9mZnVkJMUa1h5wOweQDPwb0mTgaeBwaQun1dJlZKi4utXgRGShnay3szMasjdPdbItgCWAz5PSjhKPgPcThqrcoyknYFrgY1Jg2sfy9s9DEwExuTuoWWAg6sRuJmZdc9JijWyfYAbImJ82fLJkv6a128PjCFdwXMfcCGwNkBEzJG0FXAeMIV0dc/5wBlVid7MzLrkJMUaVkRs3cW6LxZe3tTFds8CXy5b/Ou+RWZmZv3BY1LMzMysLrmSYtZLbW3ttIweVeswGkLr7Dm1DsHMGpCTFLM+mDp1Zq1D6JVGv++Cmc0b3N1jZmZmdclJipmZmdUld/eY9UFPb/FcTxx7Gisza0Zrv7RlZv3PSYpZLzU1DWLkEbfWOgzrg5bRo5hV6yDMrFPu7jEzM7O65CTFzMzM6pKTFDMzM6tLHpNiVSdpXeAo4EvAAsBk4E/AmIiYVMO4dgOOiohVahWDmZl9yJUUqypJmwMPAAGsHRGLkubOeZOPz6FjZmbzMFdSrNrOAa6IiMNLC3L15EQASUOBU4DtgIVICc0BEfFyXj8OGA+MBL5Kmr344Ii4sdSepO2AI4FVgP8AF0XEzyUtS5oFeR1gCDABODAixktaHzgXGCKpdMHHlhExbgDOgZmZVcCVFKsaSauREocrutjsDOAL+W8F4A3gZkmDC9vsCvwKWAw4C7gkJzdI+jpwCXAcsASwGnBb3q+JlCStACwFPA5cJ2n+iPgrsC/wYkQMy3/j+vqezcys91xJsWoanh9f7WilpCZSAvLNiHg1LzsQeAv4PPDXvOnVEfFQXn8+cDqwKvAE8GPg3Ii4JW87g1SNIVdjXi4c7yjggLzv0/3yDq3hNDcPrdqxBg9uqurx+lsjx+/Ya6cv8TtJsWqamh8/BTzTwfrhpIG0L5UWRMQsSVOA5fgwSZlUWP+OJIDSLUhHAtd3dHBJS5ASmo2BZqCtcFybR1VzosVGn9ixkeN37LVTjL+nd4t2d49VTUQ8BzwP7NzJJlOB2aREAwBJw4ARwCsVHqaFVBnpyCnA0sB6ecDucnn5oPzY1uFeZmZWE66kWLX9kDTG5HXgrIh4TdKSwO6kCsqlwImSngamk8aePAs8UmH7ZwNXSboXuAsYCvxPRDwALAq8C0zLyc+Ysn0nAyMkLRoRM/ryJs3MrO9cSbGqioi7gA2ANYB/SppJGjMyAhgHHAQ8BjxKGj+yNLBVRHxQYfu3AnsAvyCNZQnga3n1Mfk4b5Ku7HkIKLZbSmxekjRdki+JNjOroUHt7e21jsGsUbV7gsHG1jJ6FFOnzqza8eamsQWNxrHXTtmYlPHAupXu60qKmZmZ1SUnKWZmZlaXPHDWrJfa2tppGT2q1mFYH7TOnlPrEMysC05SzPqgmuMZ+lMj93E3cuxm1jPu7jEzM7O65CTFzMzM6pK7e8z6oKe3eK4njr1zrbPnMGtG64Aew8y65yTFrJeamgbh+6TMnVpGj2JWrYMwM3f3mJmZWX1ykmJmZmZ1yd091iFJxWr3AvlxdmlBRAyrbkRmZjavcZJiHSomIZIuBOaLiN062lbS/BHxfrViMzOzeYOTFOsxSeOAfwAjgU2BX0i6DLgQWAcYQppl+MCIGJ/3OQ7YEPgbsGdu6ncRcWxevzhwfm5vPuDfwL4RcX9h3wnALkArcFZEjC7E9GXgVGB1YBJwRkSc113bef02wNHAynnfkyLi8v45W2Zm1lsek2K9tTvwG2Cx/NgEnAOsACwFPA5cJ2n+wj4bAS8DywBbAUdK+lJedygwNO/fDGxLSiaK+74OLA1sDRws6TsAklYEbgd+B3wS2A04RdIO3bUtaXPgIuBA4BPArsBZkjbq9ZkxM7N+4UqK9dYfI+Ke/PxdUvLxcmmlpKOAA4BVgafz4uci4tz8/GFJ/yBN2f0g8B4pwRDw94h4rux4k4AxEdEOjJd0PikZuQLYGXg8IsYW2j6PVLG5tpu2fwKcWaqqAI/kqtAuwF96elJs7tHcPLTf2xw8uGlA2q2WRo7fsddOX+J3kmK91VJ8IWkJ4HRgY1K1oi2vGl7YbFJZG+8ApbtynQbMD1wCLC3pFuCwiHg9r5+YE5Ti8bfLz5cDXipr+wVSxaW7tlcENpF0cGHfwcD92DxtIOYHavR5hxo5fsdeO8X4e3ojRicp1lttZa9PIXXFrBcRkyQtAswABlXSWES8A/wc+LmkpYDLSMnFLnmTFSQNKiQqI/mwO+gV4BtlTa6Ul3fX9kRgbEScVkmcZmZWPU5SrL8sSur2mSZpGDCmJztL+ibwPPAcMAv4D/BBYZOlgUMlnQGsBewFlKofVwJHS9qF1P3zv8A+wH4VtP1rYKykh4GHSFWUTwODIuKxnrwHMzPrXx44a/3lGGAE8CbpKpyH+GiS0Z2VgZtJ1ZcW0hU8hxfW309KVCYDtwBnkhISIuIlUiVl/3z8PwBHR8Q13bUdEXeSEp7TgDfIVwYBvg+MmVmNDWpvb+9+K7MaypcgbxARm9U6ljLtnrtn7tQyehRTp87s93bnprEFjcax107ZmJTxpAsmKuJKipmZmdUlJylmZmZWlzxw1upeRBxX6xg60tbWTsvoUbUOwwZA6+w5tQ7BzHCSYtYnAzFuoRoauY+7kWM3s55xd4+ZmZnVJScpZmZmVpd8CbJZL7W1tbc3NVV0Q10zs7lC6+w5zJrR2qN9+nIJssekmPVSU9MgfJ8UM5uXtIwexawqHs/dPWZmZlaXXEkxyySNJM2mvFxE/Lubzc3MbIC5kmIDTtI4Se2Svl22fL28vKUH7RxVtuy43MY5ZcsXlPRWXjeyr+/BzMyqz0mKVcszpIn8ivbKy/vqOWAnSUMLy7YnTUZoZmYNykmKVct1wGclrQQgaRHgW8DvSxtI2knSE5JmSJok6TxJC+d1ZwEbAkdLmiUpCm2/AjwMFCs1ewEXFAPIVZe7y5Z9rDpjZmb1wUmKVct/gMuBPfLrnYH7gEmFbd4GvgM0kxKSDYGjACJif+B+4MSIGBYRKmv/AnKlRpKA1YEbB+KNmJlZdXjgrFXTBcCdko4F9gaOBRYvrYyI2wrbPp/HmexSYds3A+dIWhP4AXAp8F6/RG1mZv/V3Dy0+40KBg9u6vE+JU5SrGoi4klJE4GjgRHA7aSKCgCSNgeOIVVBFgAGA1MqbHuOpLHAj0jjUTbo1+DNzAygx3Nnld3MrUf7urvHqu18UpJycUR8UFooaQhwA3AVsHxELAocDhRv6drWTdsXAPsAT0fEcx2snwksXLZsmR5Fb2ZmVeNKilXblaSBruPLlg8hVU+mRUSrpDWA/cu2mQys0lnDEfGipI346DiXovHALyStAzwB7Aus2PO3YGZm1eBKilVVRPwnIu6OiGlly2cB+wGnSpoFnA1cUbb7GcC6kqZLeqqT9h+MiBc7WTcOOJ3UzTQJWBJ4sC/vx8zMBo4nGDTrvXbP3WNm85KW0aOYOnVmj/bpywSDrqSYmZlZXXKSYmZmZnXJA2fNeqmtrZ2W0aNqHYaZWdW0zp5T1eM5STHrg572zdaLYh9xo3HstdPI8Tv2xuTuHjMzM6tLTlLMzMysLjlJMTMzs7rkJMXMzMzqkpMUMzMzq0tOUszMzKwuOUkxMzOzuuQkxczMzOqSJxg0672pwMRaB2Fm1kBWAIZXurGTFDMzM6tL7u4xMzOzuuQkxczMzOqSkxQzMzOrS05SzMzMrC45STEzM7O65CTFzMzM6tJ8tQ7ArJFIGgyMBnYDFgTuBPaJiDdqGVclJI0FvgvMLiw+LCLOqU1EnZO0E/Aj4DPA0IiYr2z9FsCvgJWAF4CDI+LOqgfaia7il7QxcC/wTmGXCRHxxaoG2QlJY4AtgeWAWcCtwOER8VZhm12AY4GlgX8CP4yI8TUI9yO6i13SbsDFwLuF3W6OiJ2rHGqHJJ0MfAf4JPAf4C+kf9sv5/V1ed6h69j7ct6dpJj1zBHA1sB6wJuk//D+AHy9lkH1wCURsWetg6jANOAcYCHg/OIKSSsB1wF7A9cAOwDXS1ozIlqqHGdnOo0/+yAihlU3pIp9AHwPeBJoBi4FxgJbAUjaAPgdsC1wH/AT4E+SVo2IGTWIt6jL2LMXI2KVqkdWmT8Ap0bE25KGAicBVwFfrPPzDl3Entf36rw7STHrmb2BEyLiRQBJhwHPS1ohInz32X4SEXfAf6sO5XYFxkfEZfn15ZL2zcuPr06EXesm/roWEUcWXk6VdCYpGSzZC7iuVLmSdBqwP+nL85KqBdqBCmKvaxHxbOHlIKANUH5dt+cduo2915ykmFVIUjOwPPDf8mpEvCBpBqms3whJyrckbQe8AdwIHB8Rs2ocU099hsJnkD2elzeKwZJeAeYnvZcjI+KJGsfUma8Axdg+Q6pOABAR7ZL+Tn2e//LYAZaTNBl4H3gQ+FlEvFT1yDoh6TukismiwBzg4Lyq7s97F7FDL8+7B86aVW6R/Ph22fLppP8o691vgdWBJUi/vr4MXFDTiHpnERr3MwB4FlgbWJH0eUwA7pG0TC2D6oikbwH7kroWShri/HcS+1+ATwPLAJ8jjZ24S9LC1Y+wYxFxRUQsRhp3chxp7Ak0wHnvIvZen3dXUswqNzM/Lla2vBmohz7hLpUNsHtK0kHAOEm7RcTszvarQzNp0M8AICImA5Pzy+nAzyRtTxrXdFGt4ionaQfgPGCriHi8sKqz8/9ClULrVmexl7pps8mS9iJ98X8B+HN1o+xaREyWdAHwoqTlaYDzXlIee1/OuyspZhWKiOnAy8D/lpblQZyLkn4NN5q2/DioplH03BMUPoPss3y8rN9I2qijz0HSD0hf8t+MiHvLVn/k/EsaRKoM1cX57yb2cu35r27OfZn5gIVJFYi6Pu8dKMZeruLz7kqKWc+cDxwu6V7S1T1jgDvq6KqSTuXLYm+PiOmSViVdwntTRPynxqF9TL7Ue35gSH69YF41m3TFxqGSdgb+CGwPrAPsUoNQO9RN/JuQkt0XgaHAT4ElgTuqH+nHSTqAdJnr1yLi0Q42uQC4XdIlwP3AAaTL8a+vXpQd6y52SaNIX+qvAosDp5DGZz1czTg7IqkJ+CFwTURMkbQsqYu2hdRFWM/nvcvY+3LeXUkx65nRwM3Ao6T/4AaTLnlsBPuSyq/vkO7v8jDwg9qG1KnvA62kL+7B+XkrsEJEvABsBxxF6uI5Cti2zhLFTuMnDXT8M6l8/yKp5L15RLxSm1A/5kxSdfBeSbNKf6WVEfEA6QvpAlLJ/tvAN+rkMtguYwc2Bh4h3UPlKdI9PTavo8Hj3wCezP+N/o10X5HNImJOnZ936CJ2+nDeB7W3tw9YxGZmZma95UqKmZmZ1SUnKWZmZlaXnKSYmZlZXXKSYmZmZnXJSYqZmZnVJScpZmZmVpecpJiZDQBJ4ySd1cX6jSW1S1qih+3uLellSW2SjutzoA1C0nGSnqx1HFZdTlLMrK5J+q6kVyRNk3R62bpPSWqRtGQF7bTnOXJ6evxe7VeBh0gTsb3Zg1gWB84GTgM+BfxyAOKqKUkj8zlft2zVL0mTYto8xLfFN7O6lasMFwK7ke7OequkeyLilrzJ2cCJEfF6jULstYh4jw8nGqzUCqT/b98SEZN6e2xJQ/LxG0a+O2m93BnWqsRJipnVs5WAtyPiaoA8Z9L/A26R9C3SrLAXd9eIpJb89FpJABMjYmRetw9wKLA8aU6dMRFxQVf7SVoZOB1YD1gECOCYQvLULUkbA/cCwyPiDUm7AWcBW5Nu774i6Vbiu0fES3n97/PuL+Z4VoyIFknfBI4D1gQmAVcAx5cSkfw+xub3uB1wF7CDpC+S5lH5HDANuAk4vHSrdUnjgKdJszXvTZoI8VLgsIhoy9sMycf+LrAUabqIX0fEb/L6NUiVn41IUwP8GTgozwbdkZfy46P5Pd4XERvnrq3tI2Kt3O5YYAnSPDYHAQsBvwOOBI4h3UK+DTgjIsYUzvtiOZ5t8j6PA4dExGOdxGM15O4eM6tn/wKGSvqspE+QvkwnFL5o9o6ISub2+Fx+3IvUxfI5AEnbkhKDXwNrkZKDc/KXfqf7AcOA24DNSXPx/B9wnaTVe/k+SxYAfgbsDqwPNAPn5nVXA1vk55/P8bwi6WvA5fl9rJn33R74RVnbB5MmqlsXOFLSp0lzON2U38N2pFl1y5O+7wJzgC8C+wMHAjsW1l9CmtzxYFICuQcpqUHS0sBfgCdzzJuRzt2NeVK6jnw+P26R3+N2nWwHKfFZkTQ3zL7AYcCfSOdxA1LyNFrSOjmeQcCtpK6yLUmzZ/8FuCfHanXGlRQzq1sRMU3SrqRf7wsBl0bEHZLOAy4Chku6kjQl/JkRcW4n7UzNv8qnl/2C/ynwh4goDXB9Ln+hHQ7c3Nl+EfEEaVbXkpNzYrM9cFIf3vJ8wI8iIgAk/RK4WNKgiGiVVBq/MrUUj6SfA6dFRKnK8oKkw4HLJB1aSOLui4hTSweSdClwdUT8qrBsP+DvkkZExJS8+OmIOKZwfvYCvgJcmWfT3gn4ekTcnrd5sfB+9gOeiIjDC8fYBXiLlCw90sE5mJof3+yi2lLydj5fH5Bm2z0EWDoiSsncc5KOIM08PT4/rk2qXrXmbY7On933gVOxuuIkxczqWkRcT2E6ekkbkGYOPoTUzbILqUtigqQHI+KfPWj+//HxysEDwFZd7SRpYeBY0q/xpYH5gQWBCT04dkdmlxKU7DVgCGl6+7c62Wcd4PM5MSlpIiV1S5G6fwDKuzPWAVaRVKyKDMqPKwOlJKX8Pb0GjMjPP0vqUrm3i9g2KpuJuGRlOk5SeuLpnKCUvE6u4pQtK8W7DjAUKCWfJQvmeKzOOEkxs4YhaQFS98eepPEqQyLiz3ndOFLZvydJSme660L6Jak74qekLql3SdWeIX087pxO4uiqa74JOB64toN1UwvP3+lgvwuBMzrY79XC8/c7iKnSoQJNpO6Vn3awrj8GO3cUW1fxNuXjbthBWzP6IR7rZ05SzKyRHAncExEPS1qbj/4/bAgwuIt93+9g/TPAl0hdRyUbkCozXe23Aanr6f8AJJV+iT9X2dvoV48Dq0fE873Yb81e7Ff0D9IX/ybA7R2sfxz4NmnAcXny0JnSVUddfZa99TiwJNAWES92t7HVnpMUM2sI+SqR75K6GCB19cyRtC/wFGmcxIldNNECfEXSfaRulWmkwbfXShpPGkS6RT7Gdt3s9xywraQbSUnMsaQug1o4gXS100TgGlI1Zi3g8xFxWBf7jQEelnQucB4wE1gd+GZE7FPJgSPiOUnXABdK+gkpCVgWGBkRfyBdIr4XcLWkMaTKzkqkxOWQiJjZQbNTSFcBfS1flfSfiHi7kngqcDfwIGng7mGkgcRLkT73uyPi/n46jvUTX91jZnUvX5VxPunS1ZkAeeDj90mXD18PnNzNZaSHkH7xvwL8PbdxA/Bj0iWsTwM/AX4YETd3tR/pSpYppMtfbwMezs+rLiLuAEblGB/Jf0eQLqfuar8JpKtjRgL3kQYCn0LPu2F2IV3y/BvSl/5Y0qXhRMRrpEpVG6nS8hQpcZmd/zqKaw5wAKlL7zXgxh7G06k8iPgbwD3ABaRE9xpA+VhWZwa1t1dy9Z6ZmZlZdbmSYmZmZnXJSYqZmZnVJScpZmZmVpecpJiZmVldcpJiZmZmdclJipmZmdUlJylmZmZWl5ykmJmZWV1ykmJmZmZ16f8DrYdoCAkL/IEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "enc_fp16_onnx = create_model_for_provider(encoder_fp16_model_path, \"CUDAExecutionProvider\", log_severity=3)\n",
    "enc_fp16_onnx_binding: IOBinding = enc_fp16_onnx.io_binding()\n",
    "dec_onnx = create_model_for_provider(\n",
    "    dec_if_fp16_model_path, \"CUDAExecutionProvider\", enable_profiling=True, log_severity=3\n",
    ")\n",
    "dec_onnx_binding: IOBinding = dec_onnx.io_binding()\n",
    "_ = model_gen.generate(inputs=input_ids, max_length=10, min_length=10)\n",
    "profile_name = dec_onnx.end_profiling()\n",
    "\n",
    "with open(profile_name) as f:\n",
    "    content = json.load(f)\n",
    "\n",
    "op_timings = defaultdict(lambda: 0)\n",
    "for c in content:\n",
    "    if \"op_name\" not in c[\"args\"]:\n",
    "        continue\n",
    "    op_name = c[\"args\"][\"op_name\"]\n",
    "    if op_name == \"If\":\n",
    "        continue  # subgraph\n",
    "    time_taken = c[\"dur\"]\n",
    "    op_timings[op_name] += time_taken\n",
    "\n",
    "op_timings_filter = dict(sorted(op_timings.items(), key=operator.itemgetter(1), reverse=True)[:10])\n",
    "total_kernel_timing = sum(op_timings.values())\n",
    "op_timings_percent = {k: 100 * v / total_kernel_timing for k, v in op_timings_filter.items()}\n",
    "\n",
    "plt.barh(list(op_timings_percent.keys()), list(op_timings_percent.values()))\n",
    "plt.title(\"Time spent per kernel\\n(top 10 kernels)\")\n",
    "plt.xlabel(\"% total inference time\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "366de2154be95d5d8dfb409ee394df3294ef97f2f15d871b0df4dd91a89cdb4b"
  },
  "kernelspec": {
   "display_name": "Python 3.9.12 ('fast_transformer')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}