11_RoBERTa_Fine_Tuning_Emotion_classification.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "title: 12 Emotion Classification with Fine-tuned BERT\n",
    "description: Emotion classification using fine-tuned BERT model\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/drive/1nwCE6b9PXIKhv2hvbqf1oZKIGkXMTi1X?usp=sharing\" target=\"_blank\"><img align=\"left\" alt=\"Colab\" title=\"Open in Colab\" src=\"https://colab.research.google.com/assets/colab-badge.svg\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wj6eoKzotv5I"
   },
   "source": [
    "## Emotion Classification using Fine-tuned BERT model\n",
    "\n",
    "In this tutorial, I will show to fine-tune a language model (LM) for emotion classification with code adapted from this [tutorial](https://zablo.net/blog/post/custom-classifier-on-bert-model-guide-polemo2-sentiment-analysis/) by MARCIN ZABŁOCKI. I adapted his tutorial and modified the code to suit the emotion classification task using a different BERT model. Please refer to his tutorial for more detailed explanations for each code block. I really liked his tutorial because of the attention to detail and the use of high-level libraries to take care of certain parts of the model such as training and finding a good learning rate. \n",
    "\n",
    "Before you get started, make sure to enable `GPU` in the runtime and be sure to \n",
    "restart the runtime in this environment after installing the `pytorch-lr-finder` library.\n",
    "\n",
    "This tutorial is in a rough draft so if you find any issues with this tutorial or have any further questions reach out to me via [Twitter](https://twitter.com/omarsar0). \n",
    "\n",
    "Note that the notebook was created a little while back so if something break it's because the code is not compatible with the library changes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "G2tokZqttmTA"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install transformers tokenizers pytorch-lightning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I0jZnNegGhZj"
   },
   "source": [
    "Note: you need to Restart runtime after running this code segment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "k9ZKIIGvuW5m"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!git clone https://github.com/davidtvs/pytorch-lr-finder.git && cd pytorch-lr-finder && python setup.py install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "qqRRWe4UuuIh",
    "outputId": "479b5e60-10b5-4d84-c8fc-291ec32feceb"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'1.10.0+cu111'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from typing import List\n",
    "import torch.nn.functional as F\n",
    "from transformers import DistilBertTokenizer, AutoTokenizer, AutoModelWithLMHead, DistilBertForSequenceClassification, AdamW, get_linear_schedule_with_warmup\n",
    "import logging\n",
    "import os\n",
    "from functools import lru_cache\n",
    "from tokenizers import ByteLevelBPETokenizer\n",
    "from tokenizers.processors import BertProcessing\n",
    "import pytorch_lightning as pl\n",
    "from torch.utils.data import DataLoader, Dataset\n",
    "import pandas as pd\n",
    "from argparse import Namespace\n",
    "from sklearn.metrics import classification_report\n",
    "torch.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_whSBDujRiga"
   },
   "source": [
    "## Load the Pretrained Language Model\n",
    "We are first going to look at pretrained language model provided by HuggingFace models. We will use a variant of BERT, called DistilRoBERTa base. The `base` model has less parameters than the `larger` model. \n",
    "\n",
    "[RoBERTa](https://arxiv.org/abs/1907.11692) is a variant of of BERT which \"*modifies key hyperparameters, removing the next-sentence pretraining objective and training with much larger mini-batches and learning rates*\".\n",
    "\n",
    "Knowledge distillation help to train smaller LMs with similar performance and potential."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BvHNcMckSR4M"
   },
   "source": [
    "First, let's load the tokenizer for this model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 145,
     "referenced_widgets": [
      "bfa062ba95014af8993654adebeed30b",
      "494133cceef14cae8fa7efa5d18bfb45",
      "522e5a6c954c4297b21f8e0cea44a171",
      "269ff42456ee47bba72aca0923542c29",
      "511c63b5846447f1a98f99efa7b414ef",
      "9f2624b1ec3a4de3aeac12b403131745",
      "55662051a2f845c2a9de8fc27eea8485",
      "b93fa5f05abe46f8a15cd4b93f743243",
      "6b01657a3a0f47569757d62cff6970bd",
      "8fcfdc4bfbf74c73a38aa13175269565",
      "352725603b554637bceb08047c134523",
      "7b89a5097be146cb94844ff9af75bb77",
      "dd64efe54437454c9d919530cceadff7",
      "ccdae4645e744fd1a083d857d0e56a0a",
      "9404f7978dba4adfb4b757b8446bfae5",
      "43fdffaac68940beb54c21e0f5d21135",
      "0e3236fcfc07425ab83f1b158e3c0f00",
      "cb4074b47b464adb863bf9d8048ff10a",
      "bc402b2b5ac64749abee5ef047a78022",
      "b5cc3132707b4ee5885bc3bab6b384f4",
      "4f896b64ee4c4102aa5826f3927ed297",
      "eb8e48d71030482286bba58939a3ba25",
      "2ab5305315964703bf834e87f1881aaf",
      "aa7dcbcebf7e450da1d472da998b93ab",
      "728bb27db36d49319bcd129245ce2ade",
      "f8c4c65ab8eb4602b9cc67a18f17a80f",
      "52f517a67ec549d8bb3d952bc5c8471e",
      "a065514ff5ab45418085d2b5450c81ea",
      "09efbdba1ef1407aab615418a44a6571",
      "ae1118ec19ec4fb095c1e5616c366ee2",
      "a67576e0883e45de9f7c6e25d32601e4",
      "f80c7217e70b408684909176198e8720",
      "e894699ebf924a978689b1754e28449e",
      "93f62d912f43409ab8c9bd245b137bde",
      "0df8766d48974f799cf4a253ebfd8206",
      "a6f26948e4fe449b99e58a584be39492",
      "d1519847727e42a6b6a46fd1b425068c",
      "843b224c4f454a2695bbc99869987d80",
      "3f5081a276b04d38aca64760de0a078e",
      "4f51720e43824ae59e343090b4718f15",
      "697378036251425d965f8e52b36cfa60",
      "e940a875278a4ce297c7fe35be8014f8",
      "7306f377658349a5a36123251c68be41",
      "0a8b8f25fda04d358c8eb4575b5998fd"
     ]
    },
    "id": "BPbTd5lmuzQn",
    "outputId": "68c9cb51-f420-45a7-e500-235fd96c2038"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bfa062ba95014af8993654adebeed30b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/480 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7b89a5097be146cb94844ff9af75bb77",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/878k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ab5305315964703bf834e87f1881aaf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/446k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "93f62d912f43409ab8c9bd245b137bde",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.29M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained('distilroberta-base')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7KAbKMqJSWRo"
   },
   "source": [
    "Now let's load the actual model with the LM head that takes care of the prediciton for the LM. When fine-tuning we don't use the head and instead use the base model. The code below shows how to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 104,
     "referenced_widgets": [
      "950b4d43541b44cc959d62b45c6d14e3",
      "6a85a6adc199436f89b9038868ae0f48",
      "2fd1ea2ed94447f1b6cc11e407aca6f2",
      "2cd0df845e9f414e8c3a9c5d7dd9eac1",
      "8d421622ad4146cfb83237de735d3e34",
      "d1aea3f85fbb40b1ae318bcc5044b2f1",
      "0f3f0061d0e54bbbb274c5302463e67f",
      "c01bbd73be554b199bc3cfac4fd25952",
      "391e7c73480b41209fd65fbacebe9e76",
      "4d130af1b2a742869aabb3bdf7402331",
      "59d8e41de3c8447ab01d1b525709f6ae"
     ]
    },
    "id": "PCXYlMydzQlP",
    "outputId": "872c8d52-a9d3-4848-b369-6faedd5165c0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/modeling_auto.py:882: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "950b4d43541b44cc959d62b45c6d14e3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/316M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = AutoModelWithLMHead.from_pretrained(\"distilroberta-base\")\n",
    "base_model = model.base_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K2_8S8BXSpNa"
   },
   "source": [
    "Let's now try out the tokenizer first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5fidSmH-zrY_",
    "outputId": "b90e076e-5b23-44f2-ec35-aa3a9849fac4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['input_ids', 'attention_mask'])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"Elvis is the king of rock!\"\n",
    "enc = tokenizer.encode_plus(text)\n",
    "enc.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "m8F8yQCDTDQi",
    "outputId": "30c4c9d9-d559-44a8-af7e-3482324a39b4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [0, 9682, 9578, 16, 5, 8453, 9, 3152, 328, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}\n"
     ]
    }
   ],
   "source": [
    "print(enc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P3wSCLKW0ndh"
   },
   "source": [
    "`input_ids` are the numerical encoding of the tokens in the vocabulary. `attention_mask` is an addition option used when batching sequences together and you want to tell the model which tokens should be attented to ([read more](https://huggingface.co/transformers/glossary.html#attention-mask)). The attention mask information helps when dealing with variance in the size of sequences and we need a way to tell the model that we don't want to attend to the padded indices of the sequence.\n",
    "\n",
    "We are only using `input_ids` and `attention_mask`\n",
    "\n",
    "We need to also unsqueeze to simulate batch processing\n",
    "\n",
    "Using DistilBertForSequenceClassification: https://huggingface.co/transformers/model_doc/distilbert.html#distilbertforsequenceclassification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Mxsts4uT0PgA",
    "outputId": "b4f42ac5-7577-464d-a346-90eec73b8b28"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 10, 768])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = base_model(torch.tensor(enc[\"input_ids\"]).unsqueeze(0), torch.tensor(enc[\"attention_mask\"]).unsqueeze(0))\n",
    "out[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZiCO-n_1AHIf",
    "outputId": "4ea312ee-7ba1-458d-dbc4-31e9d05596bd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 768])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## size of representation of one of the tokens \n",
    "out[0][:,0,:].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "srwIb9nr4g4t"
   },
   "source": [
    "`torch.Size([1, 768])` represents batch_size, number of tokens in input text (lenght of tokenized text), model's output hidden size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iAsg0H6g53Bf",
    "outputId": "9cac2261-c90e-4bbc-8db2-2bbaa458035f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 9682, 9578, 16, 5, 8453, 9, 3152, 2]\n",
      "<s>Elvis is the king of rock</s>\n",
      "Length: 9\n",
      "torch.Size([9, 768])\n"
     ]
    }
   ],
   "source": [
    "t = \"Elvis is the king of rock\"\n",
    "enc = tokenizer.encode_plus(t)\n",
    "token_representations = base_model(torch.tensor(enc[\"input_ids\"]).unsqueeze(0))[0][0]\n",
    "print(enc[\"input_ids\"])\n",
    "print(tokenizer.decode(enc[\"input_ids\"]))\n",
    "print(f\"Length: {len(enc['input_ids'])}\")\n",
    "print(token_representations.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9RFifOoY7Hsc"
   },
   "source": [
    "## Building Custom Classification head on top of LM base model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vSUMm4Oq7nvR"
   },
   "source": [
    "Use Mish activiation function as in the one proposed in the original tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tCEDXLxq628O"
   },
   "outputs": [],
   "source": [
    "# from https://github.com/digantamisra98/Mish/blob/b5f006660ac0b4c46e2c6958ad0301d7f9c59651/Mish/Torch/mish.py\n",
    "@torch.jit.script\n",
    "def mish(input):\n",
    "    return input * torch.tanh(F.softplus(input))\n",
    "  \n",
    "class Mish(nn.Module):\n",
    "    def forward(self, input):\n",
    "        return mish(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C6Ln6KWm74ku"
   },
   "source": [
    "The model we will use to do the fine-tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9VDRSRsc71H2"
   },
   "outputs": [],
   "source": [
    "class EmoModel(nn.Module):\n",
    "    def __init__(self, base_model, n_classes, base_model_output_size=768, dropout=0.05):\n",
    "        super().__init__()\n",
    "        self.base_model = base_model\n",
    "        \n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Dropout(dropout),\n",
    "            nn.Linear(base_model_output_size, base_model_output_size),\n",
    "            Mish(),\n",
    "            nn.Dropout(dropout),\n",
    "            nn.Linear(base_model_output_size, n_classes)\n",
    "        )\n",
    "        \n",
    "        for layer in self.classifier:\n",
    "            if isinstance(layer, nn.Linear):\n",
    "                layer.weight.data.normal_(mean=0.0, std=0.02)\n",
    "                if layer.bias is not None:\n",
    "                    layer.bias.data.zero_()\n",
    "\n",
    "    def forward(self, input_, *args):\n",
    "        X, attention_mask = input_\n",
    "        hidden_states = self.base_model(X, attention_mask=attention_mask)\n",
    "        \n",
    "        # maybe do some pooling / RNNs... go crazy here!\n",
    "        \n",
    "        # use the <s> representation\n",
    "        return self.classifier(hidden_states[0][:, 0, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wjgME-3O8Yfo"
   },
   "source": [
    "### Pretest the model with dummy text\n",
    "We want to ensure that the model is returing the right information back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Y6H9eF8A8XeV",
    "outputId": "4cc0eccb-40bb-420d-f679-7fa3b548b9a6"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/modeling_auto.py:882: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
      "  FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "classifier = EmoModel(AutoModelWithLMHead.from_pretrained(\"distilroberta-base\").base_model, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-sjfHJ_L9iNH"
   },
   "outputs": [],
   "source": [
    "X = torch.tensor(enc[\"input_ids\"]).unsqueeze(0).to('cpu')\n",
    "attn = torch.tensor(enc[\"attention_mask\"]).unsqueeze(0).to('cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "o6QhCuEC-y2z",
    "outputId": "2dd22943-cb2c-4235-faae-a9024cbf69ec"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0115, -0.1552,  0.0227]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier((X, attn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I-N7WSY7Cb7v"
   },
   "source": [
    "## Prepare your dataset for fine-tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jDWkjaLV-5tj"
   },
   "outputs": [],
   "source": [
    "!mkdir -p tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wMMm5Ye1Db-m",
    "outputId": "0babb6a0-f763-4684-82f9-55819619a807"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('tokenizer/tokenizer_config.json',\n",
       " 'tokenizer/special_tokens_map.json',\n",
       " 'tokenizer/vocab.json',\n",
       " 'tokenizer/merges.txt',\n",
       " 'tokenizer/added_tokens.json',\n",
       " 'tokenizer/tokenizer.json')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## load pretrained tokenizer information\n",
    "tokenizer.save_pretrained(\"tokenizer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3FVtbmrzDkF8",
    "outputId": "bcbae7d2-f35a-4118-81f4-10f7638f7e50"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "merges.txt\t\t tokenizer_config.json\tvocab.json\n",
      "special_tokens_map.json  tokenizer.json\n"
     ]
    }
   ],
   "source": [
    "!ls tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BhTEgIaLEDRo"
   },
   "source": [
    "Implement CollateFN using fast tokenizers.\n",
    "This function basically takes care of proper tokenization and batches of sequences. This way you don't need to create your batches manually. Find out more about Tokenizers [here](https://github.com/huggingface/tokenizers/tree/master/bindings/python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3SCLBZsMDn4s"
   },
   "outputs": [],
   "source": [
    "class TokenizersCollateFn:\n",
    "    def __init__(self, max_tokens=512):\n",
    "\n",
    "        ## RoBERTa uses BPE tokenizer similar to GPT\n",
    "        t = ByteLevelBPETokenizer(\n",
    "            \"tokenizer/vocab.json\",\n",
    "            \"tokenizer/merges.txt\"\n",
    "        )\n",
    "        t._tokenizer.post_processor = BertProcessing(\n",
    "            (\"</s>\", t.token_to_id(\"</s>\")),\n",
    "            (\"<s>\", t.token_to_id(\"<s>\")),\n",
    "        )\n",
    "        t.enable_truncation(max_tokens)\n",
    "        t.enable_padding(length=max_tokens, pad_id=t.token_to_id(\"<pad>\"))\n",
    "        self.tokenizer = t\n",
    "\n",
    "    def __call__(self, batch):\n",
    "        encoded = self.tokenizer.encode_batch([x[0] for x in batch])\n",
    "        sequences_padded = torch.tensor([enc.ids for enc in encoded])\n",
    "        attention_masks_padded = torch.tensor([enc.attention_mask for enc in encoded])\n",
    "        labels = torch.tensor([x[1] for x in batch])\n",
    "        \n",
    "        return (sequences_padded, attention_masks_padded), labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4hu70Ng0Eqls"
   },
   "source": [
    "## Getting the Data and Preview it\n",
    "Below we are going to load the data and show you how to create the splits. However, we don't need to split the data manually becuase I have already created the splits and stored those files seperately which you can quickly download below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JZ3SoJH3fUsq",
    "outputId": "4b4adfed-68b5-4673-f010-96ac2e578804"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-04-02 02:29:43--  https://www.dropbox.com/s/ikkqxfdbdec3fuj/test.txt\n",
      "Resolving www.dropbox.com (www.dropbox.com)... 162.125.80.18, 2620:100:601b:18::a27d:812\n",
      "Connecting to www.dropbox.com (www.dropbox.com)|162.125.80.18|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: /s/raw/ikkqxfdbdec3fuj/test.txt [following]\n",
      "--2022-04-02 02:29:44--  https://www.dropbox.com/s/raw/ikkqxfdbdec3fuj/test.txt\n",
      "Reusing existing connection to www.dropbox.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com/cd/0/inline/BindkPg73vrwbieJ5UPi7q0aVj4zSdWDIAJEXTLje7fzw332Q4is5tqTFH6-p-vAaxn9i19935V16q5W7VLMnWygO4NIy8JPhtF_og-e53ggh1bjKDWOubFHkfDLdqFeUpBy_deZU3Mq24B26W7AuDV2n-mg0iFL1CUofID0gxW3Kw/file# [following]\n",
      "--2022-04-02 02:29:44--  https://uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com/cd/0/inline/BindkPg73vrwbieJ5UPi7q0aVj4zSdWDIAJEXTLje7fzw332Q4is5tqTFH6-p-vAaxn9i19935V16q5W7VLMnWygO4NIy8JPhtF_og-e53ggh1bjKDWOubFHkfDLdqFeUpBy_deZU3Mq24B26W7AuDV2n-mg0iFL1CUofID0gxW3Kw/file\n",
      "Resolving uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com (uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com)... 162.125.8.15, 2620:100:601c:15::a27d:60f\n",
      "Connecting to uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com (uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com)|162.125.8.15|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 206760 (202K) [text/plain]\n",
      "Saving to: ‘test.txt’\n",
      "\n",
      "test.txt            100%[===================>] 201.91K  --.-KB/s    in 0.05s   \n",
      "\n",
      "2022-04-02 02:29:45 (3.82 MB/s) - ‘test.txt’ saved [206760/206760]\n",
      "\n",
      "--2022-04-02 02:29:45--  https://www.dropbox.com/s/1pzkadrvffbqw6o/train.txt\n",
      "Resolving www.dropbox.com (www.dropbox.com)... 162.125.80.18, 2620:100:601b:18::a27d:812\n",
      "Connecting to www.dropbox.com (www.dropbox.com)|162.125.80.18|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: /s/raw/1pzkadrvffbqw6o/train.txt [following]\n",
      "--2022-04-02 02:29:46--  https://www.dropbox.com/s/raw/1pzkadrvffbqw6o/train.txt\n",
      "Reusing existing connection to www.dropbox.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com/cd/0/inline/BilN3fBwXofFrI1cRcd9FtYqHNPbVbTn1_yppqDJ7X-LrJJhV_knWzdwIaw03J-6tBnJCcH5rd6QvOFO2_WQ0FpfkezzCc0a1OfVMsJ2J0VvVEYH-893SFHLPPHpK-vTGrLX_Pq136GsnSKOvUD4_j6IcG29LQZIegH-zv_h6dgUbw/file# [following]\n",
      "--2022-04-02 02:29:46--  https://uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com/cd/0/inline/BilN3fBwXofFrI1cRcd9FtYqHNPbVbTn1_yppqDJ7X-LrJJhV_knWzdwIaw03J-6tBnJCcH5rd6QvOFO2_WQ0FpfkezzCc0a1OfVMsJ2J0VvVEYH-893SFHLPPHpK-vTGrLX_Pq136GsnSKOvUD4_j6IcG29LQZIegH-zv_h6dgUbw/file\n",
      "Resolving uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com (uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com)... 162.125.3.15, 2620:100:601c:15::a27d:60f\n",
      "Connecting to uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com (uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com)|162.125.3.15|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1658616 (1.6M) [text/plain]\n",
      "Saving to: ‘train.txt’\n",
      "\n",
      "train.txt           100%[===================>]   1.58M  --.-KB/s    in 0.07s   \n",
      "\n",
      "2022-04-02 02:29:46 (24.0 MB/s) - ‘train.txt’ saved [1658616/1658616]\n",
      "\n",
      "--2022-04-02 02:29:46--  https://www.dropbox.com/s/2mzialpsgf9k5l3/val.txt\n",
      "Resolving www.dropbox.com (www.dropbox.com)... 162.125.80.18, 2620:100:601b:18::a27d:812\n",
      "Connecting to www.dropbox.com (www.dropbox.com)|162.125.80.18|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: /s/raw/2mzialpsgf9k5l3/val.txt [following]\n",
      "--2022-04-02 02:29:47--  https://www.dropbox.com/s/raw/2mzialpsgf9k5l3/val.txt\n",
      "Reusing existing connection to www.dropbox.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com/cd/0/inline/BinDvaaKEwfVAZXuoMsG0dUJTaMOqqlvpcEUBGJ6T-1huQAZIZlOy5DP5WiDm60aNF3zascBKEBnyXQsBIdcOfUvxkpMqw_NNwWfEUYlRRc4D9qEYnm4DYLIfhW2TJtRKNBzB1bPC2KNpOCZTFbkMkeQHGaVEZ4Tk214eeFK-seb1g/file# [following]\n",
      "--2022-04-02 02:29:48--  https://uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com/cd/0/inline/BinDvaaKEwfVAZXuoMsG0dUJTaMOqqlvpcEUBGJ6T-1huQAZIZlOy5DP5WiDm60aNF3zascBKEBnyXQsBIdcOfUvxkpMqw_NNwWfEUYlRRc4D9qEYnm4DYLIfhW2TJtRKNBzB1bPC2KNpOCZTFbkMkeQHGaVEZ4Tk214eeFK-seb1g/file\n",
      "Resolving uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com (uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com)... 162.125.3.15, 2620:100:6022:15::a27d:420f\n",
      "Connecting to uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com (uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com)|162.125.3.15|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 204240 (199K) [text/plain]\n",
      "Saving to: ‘val.txt’\n",
      "\n",
      "val.txt             100%[===================>] 199.45K  --.-KB/s    in 0.03s   \n",
      "\n",
      "2022-04-02 02:29:48 (5.57 MB/s) - ‘val.txt’ saved [204240/204240]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://www.dropbox.com/s/ikkqxfdbdec3fuj/test.txt\n",
    "!wget https://www.dropbox.com/s/1pzkadrvffbqw6o/train.txt\n",
    "!wget https://www.dropbox.com/s/2mzialpsgf9k5l3/val.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "r_03fxufWX_G"
   },
   "outputs": [],
   "source": [
    "## export the datasets as txt files\n",
    "## EXERCISE: Change this to an address\n",
    "\n",
    "train_path = \"train.txt\"\n",
    "test_path = \"test.txt\"\n",
    "val_path = \"val.txt\"\n",
    "\n",
    "## emotion labels\n",
    "label2int = {\n",
    "  \"sadness\": 0,\n",
    "  \"joy\": 1,\n",
    "  \"love\": 2,\n",
    "  \"anger\": 3,\n",
    "  \"fear\": 4,\n",
    "  \"surprise\": 5\n",
    "}\n",
    "\n",
    "emotions = [ \"sadness\", \"joy\", \"love\", \"anger\", \"fear\", \"surprise\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-FJ-wN1_zmkV"
   },
   "source": [
    "### A Quick Look at the dataset\n",
    "Below is a few code sniphets to get a good idea of the dataset we are using here. You can skip this whole subsection if you like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "t23zHggkEpc-",
    "outputId": "60c12470-7ff0-43fb-a867-321fd0266862"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-04-02 01:07:36--  https://www.dropbox.com/s/607ptdakxuh5i4s/merged_training.pkl\n",
      "Resolving www.dropbox.com (www.dropbox.com)... 162.125.3.18, 2620:100:6018:18::a27d:312\n",
      "Connecting to www.dropbox.com (www.dropbox.com)|162.125.3.18|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: /s/raw/607ptdakxuh5i4s/merged_training.pkl [following]\n",
      "--2022-04-02 01:07:36--  https://www.dropbox.com/s/raw/607ptdakxuh5i4s/merged_training.pkl\n",
      "Reusing existing connection to www.dropbox.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com/cd/0/inline/Biks1uZUpPg-4hPIU8S6gBvKR2bKR8WtT5cwtOf4Kc8EbggGQBsIjoyL2n3m3mrxFeoFYX6uWurmaLJRYsVqqWRzGyzLF_JBk6frRedoLHUAC4BoZMNUV624AW9XwRGyXvyYa0W4_P6I0lHGmx9xgcGpqkS0C4_J99RwktDpqp8BuQ/file# [following]\n",
      "--2022-04-02 01:07:37--  https://uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com/cd/0/inline/Biks1uZUpPg-4hPIU8S6gBvKR2bKR8WtT5cwtOf4Kc8EbggGQBsIjoyL2n3m3mrxFeoFYX6uWurmaLJRYsVqqWRzGyzLF_JBk6frRedoLHUAC4BoZMNUV624AW9XwRGyXvyYa0W4_P6I0lHGmx9xgcGpqkS0C4_J99RwktDpqp8BuQ/file\n",
      "Resolving uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com (uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com)... 162.125.4.15, 2620:100:6018:15::a27d:30f\n",
      "Connecting to uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com (uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com)|162.125.4.15|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: /cd/0/inline2/Bikixw43BxlYqPt-13I6GiwXazCK74atIS0dtaeeTf_dj-Wkjl_9eczKwufjPW3iO25EISJ5q5RXb8IME7cJfu4G9vGjXM9klJeLpUoZXzNMHsECzUtKaoyOoCnmaUvUrP_r4-YJXoNwkYnJxXUeOXH-aBXaLnKBsc3cOdc_2sTUxrCVd244Mu7EIaTG3mAdy76eCK3SgNiTTyExcShQVZz7-xmtJ_qKccsnJC6sdAOuATCkS42esAwgk88MiWyOsUi5N0DvfRnOb6kioBX1cObwiZ-bwb53p-fP_Os0WeidckaaHkkh24Wij4rtPJaP68L8A2B1_wTkTvNZt4B1YOGG_-I08i-6KlrRSbz-1EnbIjXfS-_589CBiZDbAjJ-tbH61_RKak64LiE9BPo7FHJtlO3vmIsFMyjW1Pg_N7EU1A/file [following]\n",
      "--2022-04-02 01:07:37--  https://uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com/cd/0/inline2/Bikixw43BxlYqPt-13I6GiwXazCK74atIS0dtaeeTf_dj-Wkjl_9eczKwufjPW3iO25EISJ5q5RXb8IME7cJfu4G9vGjXM9klJeLpUoZXzNMHsECzUtKaoyOoCnmaUvUrP_r4-YJXoNwkYnJxXUeOXH-aBXaLnKBsc3cOdc_2sTUxrCVd244Mu7EIaTG3mAdy76eCK3SgNiTTyExcShQVZz7-xmtJ_qKccsnJC6sdAOuATCkS42esAwgk88MiWyOsUi5N0DvfRnOb6kioBX1cObwiZ-bwb53p-fP_Os0WeidckaaHkkh24Wij4rtPJaP68L8A2B1_wTkTvNZt4B1YOGG_-I08i-6KlrRSbz-1EnbIjXfS-_589CBiZDbAjJ-tbH61_RKak64LiE9BPo7FHJtlO3vmIsFMyjW1Pg_N7EU1A/file\n",
      "Reusing existing connection to uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com:443.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 49991846 (48M) [application/octet-stream]\n",
      "Saving to: ‘merged_training.pkl’\n",
      "\n",
      "merged_training.pkl 100%[===================>]  47.68M  51.9MB/s    in 0.9s    \n",
      "\n",
      "2022-04-02 01:07:38 (51.9 MB/s) - ‘merged_training.pkl’ saved [49991846/49991846]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://www.dropbox.com/s/607ptdakxuh5i4s/merged_training.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PQrMSUTRF06B"
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "## helper function\n",
    "def load_from_pickle(directory):\n",
    "    return pickle.load(open(directory,\"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 315
    },
    "id": "XGz89mNSHaYM",
    "outputId": "1340de4b-868b-4f86-882c-11bd810d4cf6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fd9245da5d0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEZCAYAAAB7HPUdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAZbUlEQVR4nO3df7RdZX3n8ffHRH5olQS5zcIkGNQMDo4K9BZodc0olBBACVagOCpZNDazHPzVdqZEp5YpaBfWjo50KmOUaHApEGktKaCYRmkrlB8JMlBAmgvCIimQaGKwUn76mT/2c+UQ7s09F849OznP57XWXWfvZz/nnO+Gm8/Z99nP3ke2iYiIOryg7QIiIqJ/EvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERWZMPQlHSTplo6fhyV9WNK+ktZI2lAeZ5b+knS+pBFJt0o6rOO1Fpf+GyQtnsodi4iIZ9Nk5ulLmgZsAo4AzgS22j5P0jJgpu2zJB0PfAA4vvT7rO0jJO0LrAOGAQPrgV+xva2nexQREeOaPsn+RwN3275P0iLgzaV9JXANcBawCLjIzafJ9ZJmSNq/9F1jeyuApDXAQuDi8d5sv/3287x58yZZYkRE3davX/8j20NjbZts6J/G0yE9y/YDZflBYFZZng3c3/GcjaVtvPZnkLQUWApwwAEHsG7dukmWGBFRN0n3jbet6xO5kvYATgS+vuO2clTfk/s52F5ue9j28NDQmB9UERHxHE1m9s5xwM22HyrrD5VhG8rj5tK+CZjb8bw5pW289oiI6JPJhP47eeb4+2pgdAbOYuDyjvbTyyyeI4HtZRjoamCBpJllps+C0hYREX3S1Zi+pBcDxwD/paP5PGCVpCXAfcCppf0qmpk7I8AjwBkAtrdKOhe4qfQ7Z/SkbkRE9Mekpmz22/DwsHMiNyJiciSttz081rZckRsRUZGEfkRERRL6EREVmezFWbuFecuu7Ov73XveCX19v4iI5ypH+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUpKvQlzRD0mWSfiDpTkm/JmlfSWskbSiPM0tfSTpf0oikWyUd1vE6i0v/DZIWT9VORUTE2Lo90v8s8C3brwHeANwJLAPW2p4PrC3rAMcB88vPUuACAEn7AmcDRwCHA2ePflBERER/TBj6kvYB/iNwIYDtx23/BFgErCzdVgInleVFwEVuXA/MkLQ/cCywxvZW29uANcDCnu5NRETsVDdH+gcCW4AvSfq+pC9KejEwy/YDpc+DwKyyPBu4v+P5G0vbeO3PIGmppHWS1m3ZsmVyexMRETvVTehPBw4DLrB9KPAznh7KAcC2AfeiINvLbQ/bHh4aGurFS0ZERNFN6G8ENtq+oaxfRvMh8FAZtqE8bi7bNwFzO54/p7SN1x4REX0yYejbfhC4X9JBpelo4A5gNTA6A2cxcHlZXg2cXmbxHAlsL8NAVwMLJM0sJ3AXlLaIiOiT6V32+wDwVUl7APcAZ9B8YKyStAS4Dzi19L0KOB4YAR4pfbG9VdK5wE2l3zm2t/ZkLyIioitdhb7tW4DhMTYdPUZfA2eO8zorgBWTKTAiInqn2yP92IXMW3ZlX9/v3vNO6Ov7RcTUyW0YIiIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIq0lXoS7pX0m2SbpG0rrTtK2mNpA3lcWZpl6TzJY1IulXSYR2vs7j03yBp8dTsUkREjGcyR/pvsX2I7eGyvgxYa3s+sLasAxwHzC8/S4ELoPmQAM4GjgAOB84e/aCIiIj+eD7DO4uAlWV5JXBSR/tFblwPzJC0P3AssMb2VtvbgDXAwufx/hERMUndhr6Bb0taL2lpaZtl+4Gy/CAwqyzPBu7veO7G0jZe+zNIWippnaR1W7Zs6bK8iIjoxvQu+73J9iZJvwyskfSDzo22Lcm9KMj2cmA5wPDwcE9eMyIiGl0d6dveVB43A9+gGZN/qAzbUB43l+6bgLkdT59T2sZrj4iIPpkw9CW9WNJLRpeBBcA/AauB0Rk4i4HLy/Jq4PQyi+dIYHsZBroaWCBpZjmBu6C0RUREn3QzvDML+Iak0f5fs/0tSTcBqyQtAe4DTi39rwKOB0aAR4AzAGxvlXQucFPpd47trT3bk4iImNCEoW/7HuANY7T/GDh6jHYDZ47zWiuAFZMvMyIieiFX5EZEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRka5DX9I0Sd+XdEVZP1DSDZJGJF0qaY/SvmdZHynb53W8xkdK+12Sju31zkRExM5N5kj/Q8CdHeufBD5j+9XANmBJaV8CbCvtnyn9kHQwcBrwWmAh8DlJ055f+RERMRldhb6kOcAJwBfLuoCjgMtKl5XASWV5UVmnbD+69F8EXGL7Mds/BEaAw3uxExER0Z1uj/T/N/AHwM/L+suAn9h+sqxvBGaX5dnA/QBl+/bS/xftYzznFyQtlbRO0rotW7ZMYlciImIiE4a+pLcCm22v70M92F5ue9j28NDQUD/eMiKiGtO76PNG4ERJxwN7AS8FPgvMkDS9HM3PATaV/puAucBGSdOBfYAfd7SP6nxORET0wYRH+rY/YnuO7Xk0J2K/Y/tdwHeBk0u3xcDlZXl1Wads/45tl/bTyuyeA4H5wI0925OIiJhQN0f64zkLuETSx4HvAxeW9guBr0gaAbbSfFBg+3ZJq4A7gCeBM20/9TzePyIiJmlSoW/7GuCasnwPY8y+sf0ocMo4z/8E8InJFhkREb2RK3IjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqMiEoS9pL0k3Svp/km6X9Mel/UBJN0gakXSppD1K+55lfaRsn9fxWh8p7XdJOnaqdioiIsbWzZH+Y8BRtt8AHAIslHQk8EngM7ZfDWwDlpT+S4Btpf0zpR+SDgZOA14LLAQ+J2laL3cmIiJ2bsLQd+Nfy+oLy4+Bo4DLSvtK4KSyvKisU7YfLUml/RLbj9n+ITACHN6TvYiIiK50NaYvaZqkW4DNwBrgbuAntp8sXTYCs8vybOB+gLJ9O/CyzvYxntP5XkslrZO0bsuWLZPfo4iIGFdXoW/7KduHAHNojs5fM1UF2V5ue9j28NDQ0FS9TURElSY1e8f2T4DvAr8GzJA0vWyaA2wqy5uAuQBl+z7Ajzvbx3hORET0QTezd4YkzSjLewPHAHfShP/Jpdti4PKyvLqsU7Z/x7ZL+2llds+BwHzgxl7tSERETGz6xF3YH1hZZtq8AFhl+wpJdwCXSPo48H3gwtL/QuArkkaArTQzdrB9u6RVwB3Ak8CZtp/q7e5ERMTOTBj6tm8FDh2j/R7GmH1j+1HglHFe6xPAJyZfZkRE9EKuyI2IqEhCPyKiIt2M6Uf01bxlV/b1/e4974S+vl9Em3KkHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFchuGiD7LbSaiTTnSj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiCf2IiIpMGPqS5kr6rqQ7JN0u6UOlfV9JayRtKI8zS7sknS9pRNKtkg7reK3Fpf8GSYunbrciImIs3RzpPwn8vu2DgSOBMyUdDCwD1tqeD6wt6wDHAfPLz1LgAmg+JICzgSOAw4GzRz8oIiKiPyYMfdsP2L65LP8UuBOYDSwCVpZuK4GTyvIi4CI3rgdmSNofOBZYY3ur7W3AGmBhT/cmIiJ2alJj+pLmAYcCNwCzbD9QNj0IzCrLs4H7O562sbSN177jeyyVtE7Sui1btkymvIiImEDXoS/pl4C/BD5s++HObbYNuBcF2V5ue9j28NDQUC9eMiIiiq5CX9ILaQL/q7b/qjQ/VIZtKI+bS/smYG7H0+eUtvHaIyKiT7qZvSPgQuBO25/u2LQaGJ2Bsxi4vKP99DKL50hgexkGuhpYIGlmOYG7oLRFRESfdHNr5TcC7wFuk3RLafsocB6wStIS4D7g1LLtKuB4YAR4BDgDwPZWSecCN5V+59je2pO9iIiIrkwY+ra/B2iczUeP0d/AmeO81gpgxWQKjIiI3skVuRERFUnoR0RUJKEfEVGRhH5EREUS+hERFelmymZERFfmLbuyr+9373kn9PX9BkGO9CMiKpLQj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiCf2IiIpMGPqSVkjaLOmfOtr2lbRG0obyOLO0S9L5kkYk3SrpsI7nLC79N0haPDW7ExERO9PNkf6XgYU7tC0D1tqeD6wt6wDHAfPLz1LgAmg+JICzgSOAw4GzRz8oIiKifyYMfdt/D2zdoXkRsLIsrwRO6mi/yI3rgRmS9geOBdbY3mp7G7CGZ3+QRETEFHuuY/qzbD9Qlh8EZpXl2cD9Hf02lrbx2p9F0lJJ6ySt27Jly3MsLyIixvK8T+TaNuAe1DL6esttD9seHhoa6tXLRkQEzz30HyrDNpTHzaV9EzC3o9+c0jZee0RE9NFzDf3VwOgMnMXA5R3tp5dZPEcC28sw0NXAAkkzywncBaUtIiL6aPpEHSRdDLwZ2E/SRppZOOcBqyQtAe4DTi3drwKOB0aAR4AzAGxvlXQucFPpd47tHU8OR0TEFJsw9G2/c5xNR4/R18CZ47zOCmDFpKqLiIieyhW5EREVSehHRFQkoR8RUZEJx/QjIqIxb9mVfX2/e887oeevmSP9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIirS99CXtFDSXZJGJC3r9/tHRNSsr6EvaRrwF8BxwMHAOyUd3M8aIiJq1u8j/cOBEdv32H4cuARY1OcaIiKqJdv9ezPpZGCh7feW9fcAR9h+f0efpcDSsnoQcFffCoT9gB/18f36Lfu3exvk/RvkfYP+798rbA+NtWF6H4voiu3lwPI23lvSOtvDbbx3P2T/dm+DvH+DvG+wa+1fv4d3NgFzO9bnlLaIiOiDfof+TcB8SQdK2gM4DVjd5xoiIqrV1+Ed209Kej9wNTANWGH79n7WMIFWhpX6KPu3exvk/RvkfYNdaP/6eiI3IiLalStyIyIqktCPiKhIQj8ioiJVh76kt0mq+r/B7kqNuRP3jIhOtQfebwEbJP2ppNe0XcxUkzRT0uvbrqMX3MxAuKrtOqaKpGmSftB2HVNN0isk/UZZ3lvSS9quqVckzZJ0oaRvlvWDJS1pu66qQ9/2u4FDgbuBL0v6R0lLB+wX7xpJL5W0L3Az8AVJn267rh65WdKvtl3EVLD9FHCXpAParmWqSPod4DLg86VpDvDX7VXUc1+mmZ7+8rL+z8CHW6umqDr0AWw/TPOLdwmwP/B2mjD5QKuF9c4+ZR9/E7jI9hHAb7RcU68cAfyjpLsl3SrpNkm3tl1UD80Ebpe0VtLq0Z+2i+qhM4E3Ag8D2N4A/HKrFfXWfrZXAT+H5jol4Kl2S9oF773TT5JOBM4AXg1cBBxue7OkFwF3AH/eZn09Ml3S/sCpwP9ou5geO7btAqbYx9ouYIo9ZvtxSQBImg4M0oVDP5P0Mso+SToS2N5uSZWHPvAO4DO2/76z0fYju8LYW4+cQ/Mn5vds3yTplcCGlmvqCdv3SXoTMN/2lyQNAb/Udl29Yvvv2q5hiv2dpI8Ce0s6BvivwN+0XFMv/R7NbWZeJelaYAg4ud2SckUukmYBo+PCN9re3GY90T1JZwPDwEG2/52klwNft/3GlkvriXJk+OfAvwf2oLl1yc9sv7TVwnqkzJxbAiwARHNw8kUPUCiVv14Ootm/u2w/0XJJdY/pSzoFuBE4hWb444Zyz/+BUWYmvVTSC8vY8BZJ7267rh55O3Ai8DMA2/8CDMxJeOD/AO+k+ctsb+C9NN88NyhOojnPdIrtk21/YcAC/xRg73J/sZOASyUd1nJZdYc+8IfAr9pebPt0mm/2GrRx1AXlRO5bgXtpzl/891Yr6p3HS0iMjpm+uOV6es72CDDN9lO2vwQsbLumHnob8M+SviLpreWoeJB8zPZPyxDk0cCFwAUt11R96L9gh+GcHzN4/01G/yGdQDP00fqJpB5aJenzwIwy/e9vgS+0XFMvPVJuQX5L+Yvtdxmg30/bo5Movk7zF83dkr7YblU9NTpT5wTgC7avpBmma9WgfbJO1rckXQ1cXNZPA77ZYj1T4Ypykc+/Ae8rJzsfbbmmnrD9Z+UE4MM046Z/ZHtNy2X10ntoQv79wO/SfAHRO1qtqMdsP1EuXjLNENZJNMNYg2BTOSg5BvikpD3ZBT60cyJX+k2aucIA/2B7kC4OAaBcmLXd9lNlCOQlth9su66YmKS9gQNs9/O7ovtC0nE0V8W/GbgGWAV8u8xn3+2Vqd8LgdtsbyhTp19n+9ut1lVj6Ev6nu03SfopzRGGOjb/HNgKfMr251opsIfKL97v0QTHUknzaWa7XNFyac9bx/+/TtuBdcDv276n/1X1jqS3AX8G7GH7QEmHAOfYPrHl0npC0sXApcA3bT/Wdj29Iumlth8uB1vPYntrv2vqVGXoT6RcUHGd7YParuX5knQpsB443fZ/KB8C19k+pOXSnjdJ5wIbga/RfHCfBryK5nYT77P95vaqe/4krQeOAq6xfWhpu83269qtrHcGccq0pCtsv1XSD3n2QaVtv7Kl0oBdYHxpV2T7xzR/cg6CV9n+U+AJaC4845m/hLuzE21/3vZPbT9sezlwrO1LaW5hsLt7YowT7wNzlDaoU6ZL4Av4T7ZfafvAjp9WAx9yIndcth9ou4YeebyMC49Oa3wVMCh/Sj8i6VSaeydBc7Xj6EnqQQjH2yX9Z2BaGZb7IHBdyzX10uiU6c0AZZLB3/L0/8/dlm1LuhLY5f4qy5H+4Dsb+BYwV9JXgbXAH7RbUs+8i2aGy2bgobL87vIh9/42C3s+JH2lLN4NvJbmQ/pimllKrd+lsYcGfcr0LnkX2IzpV6CcoziSZljnets/armk2AlJd9DcCfWbwFt23N72icBekfQp4PU8PWX6t4BbbZ/VXlW9U6ZKvxq4j+aqcdH8EdDqd1ok9CsgaTbwCjqG83a8ydzuqAwH/A4wj2fu22+3VVMvSPog8D7glcCmzk3sAicCe0nSO3jmlOlvtFlPL0l6xVjttu/rdy2dEvoDTtInaY6gbqfc15smOHb7aX+SrgP+gWZ20i/uU277L1srqockXWD7fW3XEc9dudfOm2jOMV1r++aWS0roDzpJdwGvH6R50KMk3TIIU09rM871FfD0XzKDchfRP6KZmfRXpekkmluhfLy9qhL6A69c4n6K7X9tu5Zek/RxmmsOBva7cmP3VQ643mD70bK+N3BL29f/ZMrm4HuE5oZda+mYqmn7g+2V1DMfAj4q6TGa6xAG6kgxdnv/AuzF09OI9+SZ52hakdAffKvLz8Cx/ZJyqft8mn9cEbuS7TTXWqyhGc46BrhR0vnQ3oFXhndityXpvTRH+3OAW2impV5n++hWC4sAJC3e2XbbK/tVS6cc6Q8oSbexk6tS254r3CMforlvy/W23yLpNcCftFxTBJKm0XyB0bvarmVHCf3B9dbyeGZ5HL3K890Mxi0KAB61/agkJO1p+weSdvub5MXur9zG/BWS9rD9eNv1dEroD6jRC0AkHTN6h8biLEk3A8vaqaynNkqaAfw1sEbSNpqrHyN2BfcA10paTfkeZwDbn26vpIR+DSTpjbavLSu/zoDc38T228vi/5T0XWAfmvsMRewK7i4/LwBe0nItv5ATuQNO0q8AK2gCUcA24Ld3hSsDI6L/EvqVkLQPwIB9MXrELqv89fmsgLV9VAvl/EKGdyog6QSaW/Tu1Xy3A9g+p9WiIgbff+tY3ovmS+1b//7fhP6Ak/R/gRfR3KL3izRfNHJjq0VFVMD2+h2arpXU+r+9hP7g+3Xbr5d0q+0/lvS/aO7THhFTaIcvRn8BMExzbq1VCf3BN3rfj0ckvRzYCuzfYj0RtVjP01+M/gRwL7CkzYJgQKbuxU79TZnL/ingZuCHwNfaLSmiCmcBh9g+kObiyJ/R3ACxVQn9wfcD4KnyxSJ/AVxPczFTREytP7T9sKQ3AUfRnFO7oOWaEvoV+Jjtn+5qv3gRFRj9NrcTgC/YvhLYo8V6gIR+DXbJX7yICmyS9Hmaryu9StKe7AKZm4uzBpykK2i+uOEY4DDg34Abbb+h1cIiBpykFwELgdtsb5C0P/A6299uta6E/mDbVX/xIqIdCf2IiIq0Pr4UERH9k9CPiKhIQj8ioiIJ/YiIivx/xQTNrHTszP8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = load_from_pickle(directory=\"merged_training.pkl\")\n",
    "\n",
    "## using a sample\n",
    "data= data[data[\"emotions\"].isin(emotions)]\n",
    "\n",
    "\n",
    "data = data.sample(n=20000);\n",
    "\n",
    "data.emotions.value_counts().plot.bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Comaf36-Hb6X",
    "outputId": "bd2ef6e3-5a10-4443-d5ea-2add82ba033f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "text        20000\n",
       "emotions    20000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jYxc8fx_H3ad"
   },
   "source": [
    "Data has been preprocessed already, using technique from this paper: https://www.aclweb.org/anthology/D18-1404/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "gYKK7ujRHfRt",
    "outputId": "04993b5a-7487-4175-8638-8860020e194d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-768c72c3-7f1d-4346-a1d6-34016f0a7435\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>emotions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>35968</th>\n",
       "      <td>i feel fantastic and i m still alive pagetitle...</td>\n",
       "      <td>joy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30417</th>\n",
       "      <td>i were asked recently about making a lightweig...</td>\n",
       "      <td>love</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49194</th>\n",
       "      <td>i was small i always feel jealous of my brothe...</td>\n",
       "      <td>anger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5172</th>\n",
       "      <td>i am feeling hopeless and this is my therapy</td>\n",
       "      <td>sadness</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77433</th>\n",
       "      <td>i know how you feel i was physically abused as...</td>\n",
       "      <td>sadness</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-768c72c3-7f1d-4346-a1d6-34016f0a7435')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-768c72c3-7f1d-4346-a1d6-34016f0a7435 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-768c72c3-7f1d-4346-a1d6-34016f0a7435');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                                                    text emotions\n",
       "35968  i feel fantastic and i m still alive pagetitle...      joy\n",
       "30417  i were asked recently about making a lightweig...     love\n",
       "49194  i was small i always feel jealous of my brothe...    anger\n",
       "5172        i am feeling hopeless and this is my therapy  sadness\n",
       "77433  i know how you feel i was physically abused as...  sadness"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JXovcl56NFPp"
   },
   "outputs": [],
   "source": [
    "## reset index\n",
    "data.reset_index(drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pSzoz9InH0Ta",
    "outputId": "1dec6c28-d407-408d-dc82-aa9c4370e05c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['joy', 'love', 'anger', 'sadness', 'fear', 'surprise'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## check unique emotions in the dataset\n",
    "data.emotions.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rJm31gKShQus"
   },
   "source": [
    "## Split the data and store into individual text files\n",
    "\n",
    "If you are using your own dataset and want to split it for training, you can uncomment the code below. Otherwise, just skip it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6ooNxSnPiztL"
   },
   "outputs": [],
   "source": [
    "## uncomment the code below to generate the text files for your train, val, and test datasets.\n",
    "\n",
    "'''\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "# Creating training and validation sets using an 80-20 split\n",
    "input_train, input_val, target_train, target_val = train_test_split(data.text.to_numpy(), \n",
    "                                                                    data.emotions.to_numpy(), \n",
    "                                                                    test_size=0.2)\n",
    "\n",
    "# Split the validataion further to obtain a holdout dataset (for testing) -- split 50:50\n",
    "input_val, input_test, target_val, target_test = train_test_split(input_val, target_val, test_size=0.5)\n",
    "\n",
    "\n",
    "## create a dataframe for each dataset\n",
    "train_dataset = pd.DataFrame(data={\"text\": input_train, \"class\": target_train})\n",
    "val_dataset = pd.DataFrame(data={\"text\": input_val, \"class\": target_val})\n",
    "test_dataset = pd.DataFrame(data={\"text\": input_test, \"class\": target_test})\n",
    "final_dataset = {\"train\": train_dataset, \"val\": val_dataset , \"test\": test_dataset }\n",
    "\n",
    "train_dataset.to_csv(train_path, sep=\";\",header=False, index=False)\n",
    "val_dataset.to_csv(test_path, sep=\";\",header=False, index=False)\n",
    "test_dataset.to_csv(val_path, sep=\";\",header=False, index=False)\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rAD1J6c0dLp8"
   },
   "source": [
    "## Create the Dataset object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aOOI69vwIYcN"
   },
   "source": [
    "Create the Dataset object that will be used to load the different datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ktr6xeMuISin"
   },
   "outputs": [],
   "source": [
    "class EmoDataset(Dataset):\n",
    "    def __init__(self, path):\n",
    "        super().__init__()\n",
    "        self.data_column = \"text\"\n",
    "        self.class_column = \"class\"\n",
    "        self.data = pd.read_csv(path, sep=\";\", header=None, names=[self.data_column, self.class_column],\n",
    "                               engine=\"python\")\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.data.loc[idx, self.data_column], label2int[self.data.loc[idx, self.class_column]]\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.data.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9EYQRq3qJH7n"
   },
   "source": [
    "Sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uGWw4wGEJGhJ",
    "outputId": "e0983e49-770d-495f-a4d4-50d499ff4011"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('i now feel compromised and skeptical of the value of every unit of work i put in',\n",
       " 4)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = EmoDataset(train_path)\n",
    "ds[19]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0h6tTn9hd6v8"
   },
   "source": [
    "## Training with PyTorchLightning\n",
    "\n",
    "[PyTorchLightning](https://www.pytorchlightning.ai/) is a library that abstracts the complexity of training neural networks with PyTorch. It is built on top of PyTorch and simplifies training.\n",
    "\n",
    "![](https://pytorch-lightning.readthedocs.io/en/latest/_images/pt_to_pl.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RJHhNRcZK7sV"
   },
   "outputs": [],
   "source": [
    "## Methods required by PyTorchLightning\n",
    "\n",
    "class TrainingModule(pl.LightningModule):\n",
    "    def __init__(self, hparams):\n",
    "        super().__init__()\n",
    "        self.model = EmoModel(AutoModelWithLMHead.from_pretrained(\"distilroberta-base\").base_model, len(emotions))\n",
    "        self.loss = nn.CrossEntropyLoss() ## combines LogSoftmax() and NLLLoss()\n",
    "        #self.hparams = hparams\n",
    "        self.hparams.update(vars(hparams))\n",
    "\n",
    "    def step(self, batch, step_name=\"train\"):\n",
    "        X, y = batch\n",
    "        loss = self.loss(self.forward(X), y)\n",
    "        loss_key = f\"{step_name}_loss\"\n",
    "        tensorboard_logs = {loss_key: loss}\n",
    "\n",
    "        return { (\"loss\" if step_name == \"train\" else loss_key): loss, 'log': tensorboard_logs,\n",
    "               \"progress_bar\": {loss_key: loss}}\n",
    "\n",
    "    def forward(self, X, *args):\n",
    "        return self.model(X, *args)\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        return self.step(batch, \"train\")\n",
    "    \n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        return self.step(batch, \"val\")\n",
    "\n",
    "    def validation_end(self, outputs: List[dict]):\n",
    "        loss = torch.stack([x[\"val_loss\"] for x in outputs]).mean()\n",
    "        return {\"val_loss\": loss}\n",
    "        \n",
    "    def test_step(self, batch, batch_idx):\n",
    "        return self.step(batch, \"test\")\n",
    "    \n",
    "    def train_dataloader(self):\n",
    "        return self.create_data_loader(self.hparams.train_path, shuffle=True)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return self.create_data_loader(self.hparams.val_path)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return self.create_data_loader(self.hparams.test_path)\n",
    "                \n",
    "    def create_data_loader(self, ds_path: str, shuffle=False):\n",
    "        return DataLoader(\n",
    "                    EmoDataset(ds_path),\n",
    "                    batch_size=self.hparams.batch_size,\n",
    "                    shuffle=shuffle,\n",
    "                    collate_fn=TokenizersCollateFn()\n",
    "        )\n",
    "        \n",
    "    @lru_cache()\n",
    "    def total_steps(self):\n",
    "        return len(self.train_dataloader()) // self.hparams.accumulate_grad_batches * self.hparams.epochs\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        ## use AdamW optimizer -- faster approach to training NNs\n",
    "        ## read: https://www.fast.ai/2018/07/02/adam-weight-decay/\n",
    "        optimizer = AdamW(self.model.parameters(), lr=self.hparams.lr)\n",
    "        lr_scheduler = get_linear_schedule_with_warmup(\n",
    "                    optimizer,\n",
    "                    num_warmup_steps=self.hparams.warmup_steps,\n",
    "                    num_training_steps=self.total_steps(),\n",
    "        )\n",
    "        return [optimizer], [{\"scheduler\": lr_scheduler, \"interval\": \"step\"}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OGc7Vw1moHxr"
   },
   "source": [
    "## Finding Learning rate for the model\n",
    "\n",
    "The code below aims to obtain valuable information about the optimal learning rate during a pretraining run. Determine boundary and increase the leanring rate linearly or exponentially.\n",
    "\n",
    "More: https://github.com/davidtvs/pytorch-lr-finder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 474,
     "referenced_widgets": [
      "69c22be4a46740149cf5ea6823a5eed9",
      "7c615a5ac40b43ad80f49e5b520c7cd5",
      "b4038fd3724d429fb0026753c504c40d",
      "c8b4362c1387438d8bff0a98838c62f6",
      "3c8b0bd72bb34489ba4f7336b04127be",
      "4599afc67ec241fb879ac419d92cb01d",
      "abb8e33775974bdb8718d09b1ba6875b",
      "66f1cd46a86e44558cf2657469a63c62",
      "e0d5d58dd5584898a7fe092a9fb371fb",
      "fd994e14a67a4adf80a480b6ee20afa8",
      "2e849223bca94e42a426180c54ecd8fb"
     ]
    },
    "id": "xL4lNPDFoFyU",
    "outputId": "aa34657a-62da-4718-8495-768b30c9849e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/modeling_auto.py:882: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
      "  FutureWarning,\n",
      "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:309: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69c22be4a46740149cf5ea6823a5eed9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stopping early, the loss has diverged\n",
      "Learning rate search finished. See the graph with {finder_name}.plot()\n",
      "LR suggestion: steepest gradient\n",
      "Suggested LR: 3.65E-02\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU5dn4/8+VfSUEEtawy75DZBEXcAEEl1arltrWpYi2LqVafi61j1rbp34fu1i7qNhSXFHrUlGwolXEBYSwExDBgBACSViyZ5LMzPX7YyYxwCQkIZMzSa736zWvyZxzn3OuOWKu3Mu5b1FVjDHGmBOFOR2AMcaY0GQJwhhjTECWIIwxxgRkCcIYY0xAliCMMcYEZAnCGGNMQBFOB9CcUlJStG/fvk6HYYwxrcb69esPq2pqoH1tKkH07duXjIwMp8MwxphWQ0S+rmufNTEZY4wJyBKEMcaYgIKWIESkl4h8KCLbRSRTRH4aoIyIyOMisltEtojIuFr7PCKyyf9aGqw4jTHGBBbMPgg3cJeqbhCRRGC9iLynqttrlbkYGOh/TQSe8L8DlKvqmNMNoqqqiuzsbFwu1+meyoS4mJgY0tLSiIyMdDoUY9qEoCUIVT0IHPT/XCwiO4CeQO0EcTnwrPpmDFwjIh1FpLv/2GaRnZ1NYmIiffv2RUSa67QmxKgqR44cITs7m379+jkdjjFtQov0QYhIX2As8PkJu3oC+2t9zvZvA4gRkQwRWSMi36rn3PP85TLy8/NP2u9yuejcubMlhzZOROjcubPVFI1pRkFPECKSALwGzFfVokYc2kdV04HvAY+JyIBAhVR1oaqmq2p6amrAobyWHNoJ++9s2oKdh4pxVXmcDgMIcoIQkUh8yeEFVX09QJEDQK9an9P821DV6vcsYCW+GkjwqcKaNfDGG773IK2X8dhjj1FWVhaUczdUQUEBf/vb31rsen379uXw4cMAnHXWWU0+z+LFi8nJyWmusIwJGa4qD5f+5RPufGWT06EAwR3FJMA/gB2q+oc6ii0FfugfzTQJKFTVgyKSLCLR/vOkAFM4vu8iOJYvh9694aKL4Prrfe+9e/u2N7O2kiDcbneTjvvss8+afE1LEKatyi+uoNLtZfnWQ7yztdm6YpssmDWIKcAPgPNrDVedJSK3iMgt/jLLgSxgN/A08BP/9qFAhohsBj4EHjlh9FPzW74cvvMdyM6GkhIoKvK9Z2f7tjcxSZSWljJ79mxGjx7NiBEjePnll3n88cfJyclh2rRpTJs2DYAVK1YwefJkxo0bx1VXXUVJSQkA69ev57zzzmP8+PHMmDGDgwd9/2imTp3KT3/6U8aMGcOIESNYu3ZtzfVuvPFGJkyYwNixY3nzzTcByMzMZMKECYwZM4ZRo0axa9cu7rnnHr766ivGjBnDggULTor94YcfZvDgwZx99tnMmTOH3/3udzXXnj9/Punp6fzpT3/irbfeYuLEiYwdO5YLL7yQ3NxcAI4cOcL06dMZPnw4c+fOpfbqhQkJCTU/P/roo5x55pmMGjWKBx54AIC9e/cydOhQbrrpJoYPH8706dMpLy/n1VdfJSMjg2uvvZYxY8ZQXl7epP8uxoSi3CJfH1p8VDi/fDOTY6WVzgakqm3mNX78eD3R9u3bT9p2Eq9XtWdPVV+DUuBXWpqvXCO9+uqrOnfu3JrPBQUFqqrap08fzc/PV1XV/Px8Peecc7SkpERVVR955BF96KGHtLKyUidPnqx5eXmqqvrSSy/pDTfcoKqq5513Xs15P/roIx0+fLiqqt5777363HPPqarqsWPHdODAgVpSUqK33XabPv/886qqWlFRoWVlZbpnz56a4060du1aHT16tJaXl2tRUZGeccYZ+uijj9Zc+8c//nFN2aNHj6rXf2+efvppvfPOO1VV9fbbb9eHHnpIVVXffvttBWq+c3x8vKqqvvvuu3rTTTep1+tVj8ejs2fP1o8++kj37Nmj4eHhunHjRlVVveqqq2q+13nnnafr1q0LGHeD/nsbE6KWbcnRPne/ra+t368D7l2mP3tpY9CvCWRoHb9T29RcTE32+edQWFh/mYICWLsWJk6sv9wJRo4cyV133cXdd9/NJZdcwjnnnHNSmTVr1rB9+3amTJkCQGVlJZMnT2bnzp1s27aNiy66CACPx0P37t1rjpszZw4A5557LkVFRRQUFLBixQqWLl1a89e+y+Vi3759TJ48md/85jdkZ2dzxRVXMHDgwHrj/vTTT7n88suJiYkhJiaGSy+99Lj911xzTc3P2dnZXHPNNRw8eJDKysqaYaarVq3i9dd9XU+zZ88mOTn5pOusWLGCFStWMHasr4uppKSEXbt20bt3b/r168eYMb5HYcaPH8/evXvrjdmY1q66BjF1cBd+Mu0MHv/vLi4d3YNpQ7o4Eo8lCICDByHsFK1tYWHQhHbvQYMGsWHDBpYvX87999/PBRdcwP/8z/8cV0ZVueiii1iyZMlx27du3crw4cNZvXp1wHOfOGpHRFBVXnvtNQYPHnzcvqFDhzJx4kSWLVvGrFmzeOqpp+jfv3+jv0+1+Pj4mp9vv/127rzzTi677DJWrlzJgw8+2ODzqCr33nsvN99883Hb9+7dS3R0dM3n8PBwa04ybV5uUQWR4UJyXCS3TTuD/2w7yH1vbOXd+efQYfMG3++q7t19f6i2wKg9m4sJfDfc662/jNcLPXo0+tQ5OTnExcXx/e9/nwULFrBhwwYAEhMTKS4uBmDSpEl8+umn7N69G/D1I3z55ZcMHjyY/Pz8mgRRVVVFZmZmzblffvllAD755BOSkpJISkpixowZ/PnPf65p79+4cSMAWVlZ9O/fnzvuuIPLL7+cLVu2HBfDiaZMmcJbb72Fy+WipKSEt99+u87vWFhYSM+evsdXnnnmmZrt5557Li+++CIA77zzDseOHTvp2BkzZrBo0aKaPpcDBw6Ql5dX7z2tL25jWrO8IhddEmMQEaIiwvi/74xm6MaP8fZqmcEzJ7IaBPiycVKSr1O6Lh07woQJjT711q1bWbBgAWFhYURGRvLEE08AMG/ePGbOnEmPHj348MMPWbx4MXPmzKGiogKAX//61wwaNIhXX32VO+64g8LCQtxuN/Pnz2f48OGAb2qJsWPHUlVVxaJFiwD45S9/yfz58xk1ahRer5d+/frx9ttv88orr/Dcc88RGRlJt27duO++++jUqRNTpkxhxIgRXHzxxTz66KM1cZ955plcdtlljBo1iq5duzJy5EiSkpICfscHH3yQq666iuTkZM4//3z27NkDwAMPPMCcOXMYPnw4Z511Fr179z7p2OnTp7Njxw4mT54M+Dqvn3/+ecLDw+u8p9dffz233HILsbGxrF69mtjY2Mb+ZzEmJOUVV9Clwzc15zFbP+OpN/8fkZUnPABaUuIbPPPqqzBrVvACqqtzojW+mtxJraq6bJlqbGzgDurYWN/+EFJfR21zKS4uVlXV0tJSHT9+vK5fvz6o12sO1kltWrMLf79Sb342w/chiINnaqOeTmprYqo2a5YvG6elQUICdOjge09LC36WDlHz5s1jzJgxjBs3jiuvvJJx48ad+iBjTJPlFrm+qUE0ZvBMkFgTU22zZsG+fb4bnpPj63OYMKFFOoMaa+XKlUG/RnX/gTEm+FxVHopcbrp2iPFtCOLgmYayBHEikUYPZTXGmNOVV+Trf+yS6K9BBHHwTEO1iyYm1eDMp2RCi/13Nq1ZbrGvI7qmBlE9eKY+TRw801BtPkHExMRw5MgR++XRxqn61oOIiYlxOhRjmqT6IbmaPggRWLgQ6hilVxEVjfeJJ4PaBN7mm5jS0tLIzs4m0FoRpm2pXlHOmNYo19/E1DWx1h851YNnbr7Z1yEdFgZeLyWxCdx23s1MSBhcM4FdMLT5BBEZGWkrjBljQl5esYuo8DA6xp2wZG6AwTPxZ55Jwkub+N27OxnVsyNnD0wJSkxtPkEYY0xrkFfke0gu4MJXJwyeEeD/XTmKnYeKueOljbx9+9n06Nj8D4y2+T4IY4xpDXKLXN+MYGqA+OgInvzBeCrdXn78wgYq3M2/Cp0lCGOMCQG5Ra5vRjA10IDUBH531WjG9uqI0Pyd1dbEZIwxISCvuIJzBqY2+riZI7oxc0S3IERkNQhjjHFcWaWbYpf7uIn6QkEw16TuJSIfish2EckUkZ8GKCMi8riI7BaRLSIyrta+60Rkl/91XbDiNMYYp33zFHVoPccTzCYmN3CXqm4QkURgvYi8p8evLX0xMND/mgg8AUwUkU7AA0A6oP5jl6rqyQsKGGNMK5dX7H8Gor3UIFT1oKpu8P9cDOwAep5Q7HLgWf+ss2uAjiLSHZgBvKeqR/1J4T1gZrBiNcYYJ1U/Rd3YTupga5E+CBHpC4wFPj9hV09gf63P2f5tdW03xpg2p2aajUYMc20JQU8QIpIAvAbMV9WiIJx/nohkiEiGTadhjGmN8ooriIoIIyk28tSFW1BQE4SIROJLDi+o6usBihwAetX6nObfVtf2k6jqQlVNV9X01NTGDxEzxhin5RW56FrXU9QOCuYoJgH+AexQ1T/UUWwp8EP/aKZJQKGqHgTeBaaLSLKIJAPT/duMMabNyS2qOH6SvhARzFFMU4AfAFtFZJN/231AbwBVfRJYDswCdgNlwA3+fUdF5GFgnf+4X6nq0SDGaowxjsktdjGkW6LTYZwkaAlCVT+B+p/99i+YfWsd+xYBi4IQmjHGhJS8ogrObcJT1MFmT1IbY4yDSivclFS4Q26IK1iCMMYYR4XqQ3JgCcIYYxz1zTMQVoMwxhhTyzdPUVsNwhhjTC35/iamLtYHYYwxprbcIhcxkWF0iAm95XksQRhjjINyiyrokhgTck9RgyUIY4xxVK5/mo1QZAnCGGMclF9cEZL9D2AJwhhjHJVb5ArJeZjAEoQxxjimpMJNaaUn5NairmYJwhhjHBLKz0CAJQhjjHFMXpF/mg1rYjLGGFNbXrF/mg3rpDbGGFNbzTxM1sRkjDGmttyiCmIjw0mMDr2nqMEShDHGOCavuCIk16KuZgnCGGMcklvkCtn+BwhighCRRSKSJyLb6tifLCJviMgWEVkrIiNq7dsrIltFZJOIZAQrRmOMcVJekYsuiaHZ/wDBrUEsBmbWs/8+YJOqjgJ+CPzphP3TVHWMqqYHKT5jjHGMqpJbVBGSS41WC1qCUNVVwNF6igwDPvCX/QLoKyJdgxWPMcaEkpIKN+VVnpB9SA6c7YPYDFwBICITgD5Amn+fAitEZL2IzKvvJCIyT0QyRCQjPz8/qAEbY0xzya1+SK491iAa4BGgo4hsAm4HNgIe/76zVXUccDFwq4icW9dJVHWhqqaranpqamrQgzbGmOaQ538GIjWE+yAcG3yrqkXADQDiG+O1B8jy7zvgf88TkTeACcAqh0I1xphml1tcPQ+T1SBOIiIdRSTK/3EusEpVi0QkXkQS/WXigelAwJFQxhjTWuW1giamoNUgRGQJMBVIEZFs4AEgEkBVnwSGAs+IiAKZwI/8h3YF3vA/OBIBvKiq/wlWnMYY44Tcogrio8JJCNGnqCGICUJV55xi/2pgUIDtWcDoYMVljDGhILc4tB+SA3uS2hhjHBHqD8mBJQhjjDlt/7t8B9/+26dUebwNPsY3D5PVIIwxps2qdHt5ae0+Nu4rYNEnexp0jO8paqtBGGNMm/bJ7nyKXG56dozlsfd3kX2s7JTHFLncuKq8VoMwxpi27O0tB0mMieCFuRMBeHBpJqpa7zGhvlBQNUsQxhjTRBVuD+9l5jJjeDf6psTzs4sG8v6OPFZsz63zGFeVh1/+exvhYcLwHh1aMNrGswRhjDFN9PGXhymucHPJqO4A3DClH0O6JfLg0kxKK9wnla90e7nl+fWs3XuUP1w9mjO6JLZ0yI1iCcIYY5po2daDJMVGMuWMFAAiw8P4zbdHcrDQxR/f+/K4sm6Pl/kvb2Tlznz+99sjuXxMTydCbhRLEMYY0wSuKg/vbc9l5vBuRIZ/86t0fJ9k5kzozT8/20tmTiEAXq9yz+tbWb71EPfPHsqcCb2dCrtRLEEYY0wTfPRlPiUVbmb7m5dqu3vmYDrGRvKLN7bh8Sq/ens7r67PZv6FA5l7Tn8Hom0aSxDGGNMEy7YcJDkukskDOp+0r2NcFL+YPZRN+wu45qnVLP5sL3PP7sdPLxjoQKRNZwnCGGMayVXl4f0ducwccXzzUm3fHtuTyf07k/H1MeZM6MUvZg/FPwlpqxG60wgaY0yIWrkzj7JKD5eM6lFnGRHhT98dw4c78/jO+F6tLjmAJQhjjGm0t7ccpHN8FBP7daq3XJcOMVxzZuvokA7EmpiMMaYRyird/HdHHjNHdCOijualtqJtfztjjGlmH36RT3mVJ+DopbbGEoQxxjTCsq05pCREMbHfyaOX2pqgJQgRWSQieSIScD1pEUkWkTdEZIuIrBWREbX2zRSRnSKyW0TuCVaMxhjTGKUVbj74Io+LR3QnPKz1dTo3VjBrEIuBmfXsvw/YpKqjgB8CfwIQkXDgr8DFwDBgjogMC2KcxhjTIB98kYerylsz91JbF7QEoaqrgKP1FBkGfOAv+wXQV0S6AhOA3aqapaqVwEvA5cGK0xhjGmrZloN0SYwmvW/9o5faCif7IDYDVwCIyASgD5AG9AT21yqX7d9mjDGOWrPnCNMGd2kXzUvgbIJ4BOgoIpuA24GNgKexJxGReSKSISIZ+fn5zR2jMcYAcLS0koKyKgZ2TXA6lBbj2INyqloE3AAgvkcM9wBZQCzQq1bRNOBAPedZCCwESE9Pr38ZJ2OMaaKs/BIA+qfGOxxJy3GsBiEiHUUkyv9xLrDKnzTWAQNFpJ9//3eBpU7FaYwxAFmHSwHol2I1iNMmIkuAqUCKiGQDDwCRAKr6JDAUeEZEFMgEfuTf5xaR24B3gXBgkapmBitOY4xpiKz8UiLDhV7JsU6H0mKCliBUdc4p9q8GBtWxbzmwPBhxGWNMU2Tll9C7U1ybn16jtvbzTY0x5jRkHS6lf2r7aV4CSxDGGHNKbo+Xr4+UtqsOarAEYYwxp5R9rJwqjzKgHXVQgyUIY4w5pazD7W+IK1iCMMaYU8rK9w1xtT4IY4wxx/kqv5SOcZF0io86deE2xBKEMcacwp7DJfRPaV/NS2AJwhhjTikrv/0NcYUGJggRiReRMP/Pg0TkMhGJDG5oxhjjvGJXFXnFFe2ugxoaXoNYBcSISE9gBfADfAsCGWNMm7bHPwdT/3Y2xBUaniBEVcvwrd/wN1W9ChgevLCMMSY0fDOCyWoQdRERmQxcCyzzbwsPTkjGGBM6svJLCBPo0znO6VBaXEMTxHzgXuANVc0Ukf7Ah8ELyxhjQsNXh0tJS44jOqL9/U3coNlcVfUj4CMAf2f1YVW9I5iBGWNMKPCNYGp/zUvQ8FFML4pIBxGJB7YB20VkQXBDM8YYZ3m96n8Gov11UEPDm5iG+Vd7+xbwDtAP30gmY4xpsw4VuXBVea0GcQqR/ucevgUsVdUqwNZ/Nsa0ae15BBM0PEE8BewF4oFVItIHKApWUMYYEwqqZ3Ed0A6fooYGJghVfVxVe6rqLPX5GphW3zEiskhE8kRkWx37k0TkLRHZLCKZInJDrX0eEdnkfy1t1DcyxphmkpVfSnxUOF0So50OxREN7aROEpE/iEiG//V7fLWJ+iwGZtaz/1Zgu6qOBqYCvxeR6qkSy1V1jP91WUNiNMaY5vZVfgn9UxMQEadDcURDm5gWAcXA1f5XEfDP+g5Q1VXA0fqKAIniu/MJ/rLuBsZjjDFB156HuEIDn4MABqjqlbU+PyQim07z2n8BlgI5QCJwjap6/ftiRCQDX8J4RFX/XddJRGQeMA+gd+/epxmSMcb4uKo85BSW0z+ll9OhOKahNYhyETm7+oOITAHKT/PaM4BNQA9gDPAXEeng39dHVdOB7wGPiciAuk6iqgtVNV1V01NTU08zJGOM8dlzuBTV9juCCRpeg7gFeFZEkvyfjwHXnea1b8BXO1Bgt4jsAYYAa1X1AICqZonISmAs8NVpXs8YYxqsehbXfu1woaBqDR3FtNnfmTwKGKWqY4HzT/Pa+4ALAESkKzAYyBKRZBGJ9m9PAaYA20/zWsYY0yhZ+b4hrlaDaCD/09TV7gQeq6usiCzBNzopRUSygQeASP95ngQeBhaLyFZAgLtV9bCInAU8JSJefAnsEVW1BGGMaVFZ+aV0T4ohLqpRvybblNP55vWO+1LVOafYnwNMD7D9M2DkacRljDGn7avD7XsEE5zemtQ21YYxpk1SVbLy2+8kfdXqrUGISDGBE4EAsUGJyBhjHHa4pJJil7vd1yDqTRCqmthSgRhjTKj4poO6fdcgTqeJyRhj2qQs/xDX/u14iCtYgjDGmJNk5ZcQHRFGz47tuyXdEoQxxpxgz+FS+qXEExbWPifpq2YJwhhjTtDeJ+mrZgnCGGNqqfJ42Xe0rN0PcQVLEMYYc5yv8ktwe5UzuliCsARhjDG1bM/xzSg0vEeHU5Rs+yxBGGNMLZk5RcREhrX7ZyDAEoQxxhxne04RQ7p1ILydj2ACSxDGGFNDVcnMKWSYNS8BliCMMaZG9rFyilxu63/wswRhjDF+2w/6OqiHdbcEAZYgjDGmRmZOEWECQ7pZggBLEMYYU2N7TiH9UxOIjQp3OpSQENQEISKLRCRPRLbVsT9JRN4Skc0ikikiN9Tad52I7PK/rgtmnMYYA74RTNb/8I1g1yAWAzPr2X8rsF1VR+Nbv/r3IhIlIp3wrWE9EZgAPCAiyUGO1RjTjh0rrSSn0GUJopagJghVXQUcra8IkCgiAiT4y7qBGcB7qnpUVY8B71F/ojHGmNOSmVPdQZ3kcCShw+k+iL8AQ4EcYCvwU1X1Aj2B/bXKZfu3nURE5olIhohk5OfnBzteY0wbtf1gIYA9A1GL0wliBrAJ6AGMAf4iIo36r6OqC1U1XVXTU1NTgxGjMaYdyMwpontSDJ3io5wOJWQ4nSBuAF5Xn93AHmAIcADoVatcmn+bMcYEhXVQn8zpBLEPuABARLoCg4Es4F1guogk+zunp/u3GWNMsyuv9PBVfgnDelj/Q20RwTy5iCzBNzopRUSy8Y1MigRQ1SeBh4HFIrIVEOBuVT3sP/ZhYJ3/VL9S1fo6u40xpsm+OFSEV+0J6hMFNUGo6pxT7M/BVzsItG8RsCgYcRljTG3VU2xYE9PxnG5iMsYYx2XmFNEhJoK05FinQwkpliCMMe1eZk4Rw3p0wPdIlqlmCcIY0665PV6+OFhkD8gFYAnCGNOu7TlcSoXba/0PAViCMMa0azUd1D0tQZwoqKOY2iKPVyksr+JYWSWF5VUUllVRUF7pf6+ivNJDSkI03ZJi6J4UQ7ekGLp2iCEy3HKxMaEoM6eIqIgwBqQmOB1KyLEEASzdnENphdv/8lBa6aakwk1ZhZsil5ujpZUUlFVyrKyKIlcVqnWfKyo8jEqP97htItA5PprIcMGriip4FUDxKkSGC7GR4cREhhMbFU5cVDixkeGICJVuLxVuD5VuL5UeL5VuL26PEh4mhIcJEeFCeFgYEf7PMZHhxESEEes/R4z/lRwXSUpCNCmJ0aQkRJGaEE2n+CgiLHGZdi4zp5DBXRPtj7gALEEA97y2hbJKT83nmMgwEqIjiIuKoENsBMlxUfTqFEdyXCQd46JIjoskOS6KpLhIOsb6tiXFRtIhJoLwMKHI5eZQoYuDheX+dxd5xS48XkUQwsJARBB8ycPtUcoqPZRXeXBVeSir9HCstAqvKtERYURFhBEXFUHHiDCiwsOI8Ccat0fxeBW31/de5fFSWF5FXpXvXOW1zlnlOTmriUD/lHjOGpDCWQM6M6l/Z5JtHhrTjqgq23OKmD6sm9OhhCRLEMCyO84hNjKcuOhw4qN8v+RPR1JsJEmxkQzulthMEZ4eVaWkws3hkkoOl1RwuLiCwyUV5BdXsOVAIa9tyOa5NV8jAkO7deCsAZ0ZmZZEh5hIEmIiSIyJICE6gsSYSBKiT//+GBMqDha6OFZWZf0PdbAEAfRLiXc6hKASERJjIkmMiQz4Xas8XrZkF7D6qyN89tURnl3zNZVub4AzQYeYCBbMHMK1E3oTZonCtHLVa0DYCKbALEEYIsPDGN+nE+P7dOK28wfiqvKQfaycYlcVJRVuil1uil1VFLvcfPBFHr/89zbe3HiA314xkoFdQ6OWZExTbM8pQgSGdLMEEYglCHOSmMhwzugSeETHj87ux6vrs/nN8h3MevxjfjL1DH4ybQDREbbIuwldqhrwKenMnEL6dY4nPtp+FQZid8U0iohwVXovpg3pwq/e2s6f/ruLZVsP8vDlI0hLjqXAP+y3oKyKgrJKilxuxvbqyOQBnW0aA9PijpRUcPdrW/l4Vz7nDEzlklHduWBoFxJjIgFfE9OY3h0djjJ0WYIwTZKSEM3jc8by7XE9uf+Nbcx5ek295fulxPO9Cb25cnyardhlWsTHu/K585XNFJZXccnI7qzOOsL7O3KJigjjvEGpXDCkCwcKyrl2Um+nQw1ZovUN6m9l0tPTNSMjw+kw2p3SCjdvbc4hPEzoGBdFx7hIkuMiSYqNIiYyjPd35PLCmn1kfH2MqPAwZo3sxvcm9uHMvslWqzDNrtLt5XcrdrJwVRYDuyTw+JyxDO3eAa9X2bj/GG9vOcg7Ww9xqMgFwOIbzmTq4C4OR+0cEVmvqukB91mCMC1l56FiXvz8a17feIBil5uRPZO4e+YQzh6Y4nRopo3Iyi/hjpc2su1AEd+f1JtfzBpGbNTJ/WNer7Jh3zEyc4q4dmLvdv3AqCUIE1LKKt0s3ZTDnz/YzYGCcs4+I4W7Zw5hZJrNpmma7q3NOdz92haiIsL4vytHMX24PfzWEI4kCBFZBFwC5KnqiAD7FwDX+j9GAEOBVFU9KiJ7gWLAA7jrCv5EliBalwq3h+fX7OMvH+ziWFkVl4zqzs+nD6ZvG38uxTS/ZVsOcvuSDaT36cTjc8bSLSnG6ZBaDacSxLlACfBsoARxQtlLgZ+p6vn+z3uB9MYwmgMAABMmSURBVOr1qRvKEkTrVOSq4ulVWfz94z1Uebz8cHJf7r54sA2dNQ3ywRe5zHt2PeN6J/PMjRMCNimZutWXIILW8Kaqq4CjDSw+B1gSrFhMaOsQE8ld0wfz0YKpXJXei0Wf7uHapz/ncEmF06GZEPfZ7sPc8vwGhvXowD+uT7fk0Mwc75kRkThgJvBarc0KrBCR9SIyz5nITEvr0iGG314xkj/PGcu2nEIu+/MnZOYUOh2WCVHrvz7G3Gcz6Nc5nmdumFDzbINpPo4nCOBS4FNVrV3bOFtVxwEXA7f6m6sCEpF5IpIhIhn5+fnBjtW0gEtH9+BfN5+FAlc+8RnLthx0OiQTYrYdKOT6f66lS2I0z82dYLMQB0koJIjvckLzkqoe8L/nAW8AE+o6WFUXqmq6qqanpqYGNVDTckamJfHmbVMY1r0Dt764gT+s2InX23ZG3Jmm251XzA8XrSUxOoLn506kS6J1SAeLowlCRJKA84A3a22LF5HE6p+B6cA2ZyI0TuqSGMOSeZO4anwaj3+wm1ueX0+F23PqA02b5fUqP3omgzARXrhpEmnJcU6H1KYFLUGIyBJgNTBYRLJF5EcicouI3FKr2LeBFapaWmtbV+ATEdkMrAWWqep/ghWnCW3REeH833dGcf/soazYnsv9b2yjLT27YxpnU3YBXx8p475ZQ9r8NP2hIGhzManqnAaUWQwsPmFbFjA6OFGZ1khEmHtOf4pcbh7/7y4Gd0tk7jn9nQ7LOODdbYeIDBcuGNrV6VDahVDogzCmQeZfMJCLR3Tjf5fv4MMv8pwOx7QwVeWdbYc4a0AKSbE2YqklWIIwrUZYmPD7q0czpFsH7liykV25xU6HZFrQjoPF7DtaxswRNoVGS7EEYVqVuKgInr4unejIcOY+m8Gx0kqnQzIt5D+ZhwgTuGiYNS+1FEsQptXp2TGWp34wnoMFLn78wnqqPIHXzzZty7vbDnFm306kJEQ7HUq7YQsGmVZpfJ9kHrlyJHe+spn/eTOT70/qTfaxcvYfLSP7WDnZx8o4UODiB5P68L2JtiBMa5eVX8LO3GIeuHSY06G0K5YgTKt1xbg0vswt4cmPvmLJ2n012+OjwunVKY5Kt5cHl2YytndHhna3Relbs/9kHgJghk3h3aIsQZhWbcGMwYzs6VtHolenWHolx9ExLhIR4UhJBTMe+5ifvbyJf986hZhIm8ittXp32yFGpyXRo2Os06G0K9YHYVq18DBh9qjuzB7VnVFpHUmOj6pZxrRzQjSPfmcUXxwq5nfv7nQ4UtNUBwrK2ZxdyMwR3Z0Opd2xBGHatGlDuvCDSX34+yd7+HR3o5YXMSFiRU3zko1eammWIEybd9+sofRPjeeuVzZTWFbldDimkd7ZdojBXRPpn5rgdCjtjiUI0+bFRoXz2DVjOFxSwf1v2ryPrUl+cQXr9h61h+McYgnCtAuj0joy/8KBvLU5hzc3HXA6HNNA7+/IRRVLEA6xBGHajR9PPYP0Psnc/+9tHCgodzoc0wD/2XaIPp3jGNIt0elQ2iUb5mrajfAw4Y/XjGHmY6s49/8+JDYynJjIMKIjwomufo8IIyYyzL+v+hVGfFQE/VLjGdQ1kUFdEkmKs8nigq2wvIrPvjrMjVP61YxMMy3LEoRpV3p1iuPZH03k/R25VFR5cbk9VFR5qXB7cNW8ezhcUomryoPLv73YVYWr6pspPbp2iGZQ10SG+Kce79rBVjVrbh98kUuVR615yUGWIEy7M75PMuP7JDfqGK9XOVBQzq68Yr7MLeHL3GJ25ZbwzGdf8/Guw/zrlskkxrSfWsWGfcdIS44N6nKf/9l2iG4dYhid1jFo1zD1swRhTAOEhQm9OsXRq1Mc5w/5Zjz+x7vyuf6f67hjyUae/mE6EeFtv1svp6Cc7zzxGZ0Toll03ZmMTEtq9msUuar46Mt8rknvRViYNS85pe3/azYmiM4ZmMpDlw3nw535/HrZDqfDaRGvZOzHqxARJlz91Gre257b7Nd44M1MqjzKVem9mv3cpuGCuSb1IhHJE5GAA89FZIGIbPK/tomIR0Q6+ffNFJGdIrJbRO4JVozGNIfvT+rDjVP6sfizvTy3eq/T4QSVx6u8vG4/5wxMYeltZzOoawLznstg0Sd7mu0ab246wBsbD3DH+QMZ0bP5ayem4YJZg1gMzKxrp6o+qqpjVHUMcC/wkaoeFZFw4K/AxcAwYI6I2By/JqT9YvZQLhjShQff2s5HX+Y7HU7QfPRlHgcLXXxvQm9SE6N5ad5kpg/ryq/e3s4Db27DfZprc+w/Wsb9b2xjfJ9kbp02oJmiNk0VtAShqquAow0sPgdY4v95ArBbVbNUtRJ4Cbg8CCEa02zCw4Q/zRnLoK6J3PbCBr5so8uhvvj5flISornQv6pbbFQ4T1w7nnnn9ueZ1V8z77n1HC2tpMLtwevVRp3b41XufGUTCjx2zZh20Z8T6hzvpBaROHw1jdv8m3oC+2sVyQYm1nP8PGAeQO/etjCMcU5CdAT/uC6db/31U25cvI5/3zqlTa1+dqjQxQdf5HLzeQOIrPXLOyxMuG/WUHp3iuOBpZmMe/i9mn3hYUJEmBAZHubbf+kwJvbvHPD8T6zczbq9x/jD1aPp1Sku6N/HnJrjCQK4FPhUVRta2ziOqi4EFgKkp6c37k8WY5pZj46x/P26dK5+ajXzns3gxZsmtZl1KKo7p797ZuCO4+9P6sPQ7h1Y//VRqjxKlceL2/9e6fHy3vZcrlm4hjkTenHPxUNJiv1mWPCm/QU89v4uLh3dg2+P7dlSX8mcQigkiO/yTfMSwAGg9r/ANP82Y1qFUWkd+ePVY/jxCxu457Ut/PGaMa3+SeDqzumzz0ihT+f4OsvV94zJghmD+eN7X/KPT/bw/o48HrpsOBeP6EZZpYf5L22ka4cYfv2tEa3+XrUljjbyiUgScB7wZq3N64CBItJPRKLwJZClTsRnTFNdPLI7P58+iH9vyuFvK79yOhwACsuqKHI1bbrzVbvyOVBQzpwJTW/GjYuK4Bezh/HmrWfTJTGan7ywgZueXc89r2/l66Nl/P7q0cfVKozzglaDEJElwFQgRUSygQeASABVfdJf7NvAClUtrT5OVd0ichvwLhAOLFLVzGDFaUyw3DrtDHbnlfDouzvpnxLPxSNbZkW0/OIK1n99lKzDpezJL2XPYd/rSGklURFhXDkujZvO6deo9RWWfL6PzvFRXDTs9BftGZmWxJu3TuEfn+zhj+9/iavKy0+mDmBSHX0Txjmi2naa7dPT0zUjI8PpMIyp4ary8L2n17D9YBH/uvmsoDx1XNvuvBKuevIzjvkXRuqSGE2/lHj6p8bTLyWevUfKeHV9NlUeLxcN7crN5/VnfJ9O9Z4zr8jF5Ec+YO45/bj34qHNGu++I2V8uDOPORN6ExVho5acICLrVTU94D5LEMYEV35xBd/666e4vV6W3nZ20Cb2O+CfAqPKo/zt2nEM69GBhOiTGwkOl1Tw7Gd7eXbN1xSUVTG+TzK3nDeAC4d2Cdj+/9cPd/Pouzv58OdT6ZdSd/+DaZ3qSxCWso0JstTEaP5+XTolLjdzn8mgvNLT7Nc4UlLBD/7xOSUVbp69cQIT+nUKmBwAUhKiuXP6YD6753wevHQYuUUubno2g6ueXM2m/QXHlfV6lSVr93HWgM6WHNohSxDGtICh3Tvwp++OZVtOIbc8v54jJRXNdu5iVxXX/3MdOQXlLLr+TIb16NCg4+KiIrh+Sj9W/nwqv71iJHuPlPKtv37K/Jc21iyo9Mnuw2QfO73OadN6WROTMS3oxc/38cDSbXSIieRXl49g9qjT67h2VXm44Z/rWLf3KE//MJ1pQ7o0+VwlFW6eWLmbpz/egwBzz+nHjoPFbNpfwOp7zyc6om08z2GOZ01MxoSI703szdu3n0PP5FhufXEDP3lhPYebWJtwe7zcvmQjq7OO8PurR59WcgDfk+ALZgzhg7vOY+aIbvz1w6/44Is8vjM+zZJDO2U1CGMc4PZ4WfhxFo+9t4uEmAgeumw4l4zq3uCHxArKKnlwaSb/3pTDQ5cN57qz+jZ7jJv2F/Dyuv387MKBdLEV89osG8VkTIjalVvMz1/dwub9BVwwpAvXnNmLcwel1jk9x4GCcv7+cRYvr9tPWaWHuy4axO0XDGzhqE1bUl+CCIWpNoxptwZ2TeS1Wybz90/28MTKr/jvF3nER4UzbUgXZo3sztTBqcRFRbDjYBELV2WxdHMOAlw2pgc3nzuAwd0Snf4Kpg2zGoQxIaLK42VN1hGWbz3EisxDHCmtJCYyjEFdE9mSXUhcVDhzJvTmR2f3o0fHWKfDNW2ENTEZ08p4vMraPUd5Z9tBNu4rYMbwrvxgUl+S4myuItO8rInJmFYmPEyYPKAzkwfY/ETGOTbM1RhjTECWIIwxxgRkCcIYY0xAliCMMcYEZAnCGGNMQJYgjDHGBGQJwhhjTECWIIwxxgTUpp6kFpF84OsGFE0CCk+zXF37Am0/cVt9n1OAww2IrbEa+p2bcozdp/Z3n+qKrTmOsfvUsvdpoKoGXixdVdvdC1h4uuXq2hdo+4nb6vsMZDj5ne0+2X1y+l7ZfXL2PtV+tdcmpreaoVxd+wJtP3HbqT4HQ1OuYfepeY9pS/epqddpyDF2n5y9TzXaVBNTWyAiGVrHxFnmG3afGsbuU8PYfQqsvdYgQtlCpwNoJew+NYzdp4ax+xSA1SCMMcYEZDUIY4wxAVmCMMYYE5AlCGOMMQFZgmhFRGSqiHwsIk+KyFSn4wllIhIvIhkiconTsYQqERnq/7f0qoj82Ol4QpWIfEtEnhaRl0VkutPxtCRLEC1ERBaJSJ6IbDth+0wR2Skiu0XknlOcRoESIAbIDlasTmqm+wRwN/BKcKJ0XnPcJ1Xdoaq3AFcDU4IZr1Oa6T79W1VvAm4BrglmvKHGRjG1EBE5F98v92dVdYR/WzjwJXARvl/464A5QDjw2xNOcSNwWFW9ItIV+IOqXttS8beUZrpPo4HO+BLpYVV9u2WibznNcZ9UNU9ELgN+DDynqi+2VPwtpbnuk/+43wMvqOqGFgrfcRFOB9BeqOoqEel7wuYJwG5VzQIQkZeAy1X1t0B9TSPHgOhgxOm05rhP/ua3eGAYUC4iy1XVG8y4W1pz/XtS1aXAUhFZBrS5BNFM/54EeAR4pz0lB7AE4bSewP5an7OBiXUVFpErgBlAR+AvwQ0tpDTqPqnqLwBE5Hr8ta6gRhc6GvvvaSpwBb4/NpYHNbLQ0qj7BNwOXAgkicgZqvpkMIMLJZYgWhFVfR143ek4WgtVXex0DKFMVVcCKx0OI+Sp6uPA407H4QTrpHbWAaBXrc9p/m3meHafGsbuU8PYfWogSxDOWgcMFJF+IhIFfBdY6nBMocjuU8PYfWoYu08NZAmihYjIEmA1MFhEskXkR6rqBm4D3gV2AK+oaqaTcTrN7lPD2H1qGLtPp8eGuRpjjAnIahDGGGMCsgRhjDEmIEsQxhhjArIEYYwxJiBLEMYYYwKyBGGMMSYgSxCmzRORkha+3mctfL2OIvKTlrymaR8sQRjTSCJS7xxmqnpWC1+zI2AJwjQ7SxCmXRKRASLyHxFZ71+lb4h/+6Ui8rmIbBSR9/1rbyAiD4rIcyLyKfCc//MiEVkpIlkicketc5f436f6978qIl+IyAv+qaMRkVn+betF5HEROWnNChG5XkSWisgHwH9FJEFE/isiG0Rkq4hc7i/6CDBARDaJyKP+YxeIyDoR2SIiDwXzXpq2y2ZzNe3VQuAWVd0lIhOBvwHnA58Ak1RVRWQu8P8Bd/mPGQacrarlIvIgMASYBiQCO0XkCVWtOuE6Y4HhQA7wKTBFRDKAp4BzVXWPfzqIuowDRqnqUX8t4tuqWiQiKcAaEVkK3AOMUNUxAOJbFnMgvnUPBN96D+eq6qom3y3TLlmCMO2OiCQAZwH/8v9BD98swJQGvCwi3YEoYE+tQ5eqanmtz8tUtQKoEJE8oCsnLwW7VlWz/dfdBPTFt8JZlqpWn3sJMK+OcN9T1aPVoQP/618lzYtvXYOuAY6Z7n9t9H9OwJcwLEGYRrEEYdqjMKCg+i/uE/wZ33KuS/0L6jxYa1/pCWUrav3sIfD/Tw0pU5/a17wWSAXGq2qViOzFt6zqiQT4rao+1chrGXMc64Mw7Y6qFgF7ROQq8C0pKSKj/buT+GZtgOuCFMJOoH+tpTCvaeBxSUCePzlMA/r4txfja+aq9i5wo7+mhIj0FJEupx21aXesBmHagzgRqd308wd8f40/ISL3A5HAS8BmfDWGf4nIMeADoF9zB+Pvw/gJ8B8RKcW3PkFDvAC8JSJbgQzgC//5jojIpyKyDd+6yQtEZCiw2t+EVgJ8H8hr7u9i2jab7tsYB4hIgqqW+Ec1/RXYpap/dDouY2qzJiZjnHGTv9M6E1/TkfUXmJBjNQhjjDEBWQ3CGGNMQJYgjDHGBGQJwhhjTECWIIwxxgRkCcIYY0xAliCMMcYE9P8D31/gAF4NtVIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr=0.1 ## uper bound LR\n",
    "from torch_lr_finder import LRFinder\n",
    "hparams_tmp = Namespace(\n",
    "    train_path=train_path,\n",
    "    val_path=val_path,\n",
    "    test_path=test_path,\n",
    "    batch_size=16,\n",
    "    warmup_steps=100,\n",
    "    epochs=1,\n",
    "    lr=lr,\n",
    "    accumulate_grad_batches=1,\n",
    ")\n",
    "module = TrainingModule(hparams_tmp)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = AdamW(module.parameters(), lr=5e-7) ## lower bound LR\n",
    "lr_finder = LRFinder(module, optimizer, criterion, device=\"cuda\")\n",
    "lr_finder.range_test(module.train_dataloader(), end_lr=100, num_iter=100, accumulation_steps=hparams_tmp.accumulate_grad_batches)\n",
    "lr_finder.plot()\n",
    "lr_finder.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YdqP56M1oXav",
    "outputId": "f6c0b138-6248-4543-b987-6d7430c015f1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0001"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 1e-4 \n",
    "lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 352
    },
    "id": "vMab6vu0Bow0",
    "outputId": "d3da4b83-d2ea-4f28-988a-61e796ae210b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR suggestion: steepest gradient\n",
      "Suggested LR: 3.65E-02\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU5dn48e892TdCIGENS0D2LUhkERBwAdSqVetCrXUp4q5Uy6u29lW7vPVXW7cuLm0prqh1RUFFq4gKCmERCItgQAyBJGzZJ9vcvz9mEgNMQhIyOZPk/lzXXJM55znn3HPE3HnOs4mqYowxxhzN5XQAxhhjgpMlCGOMMX5ZgjDGGOOXJQhjjDF+WYIwxhjjlyUIY4wxfoU6HUBzSkxM1L59+zodhgm0bdu874MGORuHMW3AmjVr9qtqkr99bSpB9O3bl/T0dKfDMIE2dar3fdkyJ6Mwpk0QkW/r2mePmIwxxvhlCcIYY4xfAUsQItJLRD4Wkc0ikiEit/spIyLyuIjsEJENInJyrX1VIrLe91oUqDiNMcb4F8g2iErgTlVdKyJxwBoR+UBVN9cqczYwwPcaBzzhewcoVdXUEw2ioqKCrKws3G73iZ7KBIv77vO+b9lyxObIyEiSk5MJCwtzIChj2p6AJQhV3Qvs9f1cKCJbgJ5A7QRxAfCsemcM/EJEOopId9+xzSIrK4u4uDj69u2LiDTXaY2TXL6Kb61eTKrKgQMHyMrKIiUlxaHAjGlbWqQNQkT6AqOBL4/a1RP4rtbnLN82gEgRSReRL0Tkh/Wce46vXHpeXt4x+91uN507d7bk0MaJCJ07d7aaojHNKOAJQkRigdeAuapa0IhD+6hqGvBj4FER6e+vkKo+rappqpqWlOS3K68lh3bC/jubtmDbvkLcFVVOhwEEOEGISBje5PCCqr7up8geoFetz8m+bahq9XsmsAxvDSTwVOGLL+CNN7zvAVov49FHH6WkpCQg526ow4cP8/e//73Frte3b1/2798PwKmnntrk8yx4/XWyc3KaKyxjgoa7oorz/voZd7yy3ulQgMD2YhLgX8AWVX24jmKLgJ/6ejONB/JVda+IJIhIhO88icBEjmy7CIwlS6B3bzjrLLj6au97797e7c2srSSIysrKJh23YsWKJl9zwRtvkJ2b2+TjjQlWeYVllFd6WLJxH+9ubLam2CYLZA1iInAlcHqt7qrniMgNInKDr8wSIBPYAfwDuMm3fQiQLiJfAR8DDx7V+6n5LVkCP/oRZGVBUREUFHjfs7K825uYJIqLizn33HMZNWoUw4cP5+WXX+bxxx8nOzubadOmMW3aNACWLl3KhAkTOPnkk7nkkksoKioCYM2aNUyZMoUxY8YwY8YM9u71/qOZOnUqt99+O6mpqQwfPpxVq1bVXO/aa69l7NixjB49mrfeeguAjIwMxo4dS2pqKiNHjmT79u3cfffdfPPNN6SmpjJv3rxjYv/tb3/LoEGDmDRpErNmzeJPf/pTzbXnzp1LWloajz32GG+//Tbjxo1j9OjRnHnmmeT4/ro/cOAA06dPZ9iwYcyePZvaqxfGxsbW/PzQQw9xyimnMHLkSO7z9VDatWsXQ4YM4brrrmPYsGFMnz6d0tJSXn31VdIzMrhi3jxSU1MpLS1t0n8XY4JRToG3DS0mPIRfv5XBoeJyZwNS1TbzGjNmjB5t8+bNx2w7hsej2rOnqveBkv9XcrK3XCO9+uqrOnv27JrPhw8fVlXVPn36aF5enqqq5uXl6eTJk7WoqEhVVR988EF94IEHtLy8XCdMmKC5ubmqqvrSSy/pNddco6qqU6ZMqTnvJ598osOGDVNV1XvuuUefe+45VVU9dOiQDhgwQIuKivSWW27R559/XlVVy8rKtKSkRHfu3Flz3NFWrVqlo0aN0tLSUi0oKNCTTjpJH3rooZpr33jjjTVlDx48qB7fvfnHP/6hd9xxh6qq3nrrrfrAAw+oquo777yjQM13jomJUVXV999/X6+77jr1eDxaVVWl5557rn7yySe6c+dODQkJ0XXr1qmq6iWXXFLzvaaccoqu/s9//MbdoP/exgSpxRuytc9d7+hra77T/vcs1p+/tC7g1wTStY7fqW1qLqYm+/JLyM+vv8zhw7BqFYwbV3+5o4wYMYI777yTu+66ix/84AdMnjz5mDJffPEFmzdvZuLEiQCUl5czYcIEtm3bxqZNmzjrrLMAqKqqonv37jXHzZo1C4DTTjuNgoICDh8+zNKlS1m0aFHNX/tut5vdu3czYcIEfv/735OVlcVFF13EgAED6o37888/54ILLiAyMpLIyEjOO++8I/ZfdtllNT9nZWVx2WWXsXfvXsrLy2u6mS5fvpzXX/c2PZ177rkkJCQcc52lS5eydOlSRo/2NjEVFRWxfft2evfuTUpKCqmp3qEwY8aMYdeuXfXGbExrV12DmDqoCzdNO4nH/7ud80b1YNrgLo7EYwkCYO/e7/vW18XlguzsRp964MCBrF27liVLlnDvvfdyxhln8L//+79HlFFVzjrrLBYuXHjE9o0bNzJs2DBWrlzp99xH99oREVSV1157jUFHzXQ6ZMgQxo0bx+LFiznnnHN46qmn6NevX6O/T7WYmJian2+99VbuuOMOzj//fJYtW8b999/f4POoKvfccw/XX3/9Edt37dpFREREzeeQkBB7nGTavJyCMsJChIToMG6ZdhLvbdrLL9/YyPtzJ9Phq7Xe31Xdu3v/UG2BXns2FxN4b7jHU38Zjwd69Gj0qbOzs4mOjuYnP/kJ8+bNY+3atQDExcVRWFgIwPjx4/n888/ZsWMH4G1H+Prrrxk0aBB5eXk1CaKiooKMjIyac7/88ssAfPbZZ8THxxMfH8+MGTP4y1/+UvO8f926dQBkZmbSr18/brvtNi644AI2bNhwRAxHmzhxIm+//TZut5uioiLeeeedOr9jfn4+PXt6h68888wzNdtPO+00XnzxRQDeffddDh06dMyxM2bMYP78+TVtLnv27CH3OA3QcTExFBYX11vGmNYot8BNl7hIRITwUBd//NEohqz7FE+vluk8czSrQYA3G8fHexul69KxI4wd2+hTb9y4kXnz5uFyuQgLC+OJJ54AYM6cOcycOZMePXrw8ccfs2DBAmbNmkVZWRkAv/vd7xg4cCCvvvoqt912G/n5+VRWVjJ37lyGDRsGeKeWGD16NBUVFcyfPx+AX//618ydO5eRI0fi8XhISUnhnXfe4ZVXXuG5554jLCyMbt268ctf/pJOnToxceJEhg8fztlnn81DDz1UE/cpp5zC+eefz8iRI+natSsjRowgPj7e73e8//77ueSSS0hISOD0009n586dANx3333MmjWLYcOGceqpp9K7d+9jjp0+fTpbtmxhwoQJgLfx+vnnnyckJKTOe3r1hRdyw/33E/XII6xcuZKoqKjG/mcxJijlFpbRpcP3NefUjSt46q3/R1j5UQNAi4q8nWdefRXOOSdwAdXVONEaX01upFZVXbxYNSrKfwN1VJR3fxCZMmWKrl69OqDXKCwsVFXV4uJiHTNmjK5Zsyag12uwrVu9Lz+skdq0Zmf+eZle/2y690MAO8/URj2N1PaIqdo553izcXIyxMZChw7e9+TkwGfpIDVnzhxSU1M5+eSTufjiizn55JOPf5AxpslyCtzf1yAa03kmQOwRU23nnAO7d3tveHa2t81h7NgWaQxqrGUtsJpadfuBMSbw3BVVFLgr6doh0rshgJ1nGsoSxNFEGt2V1RhjTlRugbf9sUucrwYRwM4zDdUuHjGpBmY+JRNc7L+zac1yCr0N0TU1iOrOM/VpYueZhmrzCSIyMpIDBw7YL482TtW7HkRkZKTToRjTJNWD5GraIETg6aehjl56ZeEReJ54MqCPwNv8I6bk5GSysrLwt1aEaaX27fO+H1X9rl5RzpjWKMf3iKlrXK0/cqo7z1x/vbdB2uUCj4eiqFhumXI9Y2MH1UxgFwhtPkGEhYXZCmNtzY03et9boKHemJaSW+gmPMRFx+ijlsz103km5pRTiH1pPX96fxsje3Zk0oDEgMTU5hOEMca0BrkF3kFyfhe+OqrzjAD/7+KRbNtXyG0vreOdWyfRo2PzDxht820QxhjTGuQUuL/vwdQAMRGhPHnlGMorPdz4wlrKKpt/FTpLEMYYEwRyCtzf92BqoP5JsfzpklGM7tURofkbq+0RkzHGBIHcwjImD0hq9HEzh3dj5vBuAYjIahDGGOO4kvJKCt2VR0zUFwwCuSZ1LxH5WEQ2i0iGiNzup4yIyOMiskNENojIybX2XSUi232vqwIVpzHGOO37UdTBNY4nkI+YKoE7VXWtiMQBa0TkAz1ybemzgQG+1zjgCWCciHQC7gPSAPUdu0hVj11QwBhjWrncQt8YiPZSg1DVvaq61vdzIbAF6HlUsQuAZ32zzn4BdBSR7sAM4ANVPehLCh8AMwMVqzHGOKl6FHVjG6kDrUXaIESkLzAa+PKoXT2B72p9zvJtq2u7Mca0OTXTbDSim2tLCHiCEJFY4DVgrqoWBOD8c0QkXUTSbToNY0xrlFtYRnioi/iosOMXbkEBTRAiEoY3Obygqq/7KbIH6FXrc7JvW13bj6GqT6tqmqqmJSU1vouYMcY4LbfATde6RlE7KJC9mAT4F7BFVR+uo9gi4Ke+3kzjgXxV3Qu8D0wXkQQRSQCm+7YZY0ybk1NQduQkfUEikL2YJgJXAhtFZL1v2y+B3gCq+iSwBDgH2AGUANf49h0Ukd8Cq33H/UZVDwYwVmOMcUxOoZvB3eKcDuMYAUsQqvoZ1D/227dg9s117JsPzA9AaMYYE1RyC8o4rQmjqAPNRlIbY4yDissqKSqrDLourmAJwhhjHBWsg+TAEoQxxjjq+zEQVoMwxhhTy/ejqK0GYYwxppY83yOmLtYGYYwxpracAjeRYS46RAbf8jyWIIwxxkE5BWV0iYsMulHUYAnCGGMcleObZiMYWYIwxhgH5RWWBWX7A1iCMMYYR+UUuINyHiawBGGMMY4pKqukuLwq6NairmYJwhhjHBLMYyDAEoQxxjgmt8A3zYY9YjLGGFNbbqFvmg1rpDbGGFNbzTxM9ojJGGNMbTkFZUSFhRAXEXyjqMEShDHGOCa3sCwo16KuZgnCGGMcklPgDtr2BwhgghCR+SKSKyKb6tifICJviMgGEVklIsNr7dslIhtFZL2IpAcqRmOMcVJugZsuccHZ/gCBrUEsAGbWs/+XwHpVHQn8FHjsqP3TVDVVVdMCFJ8xxjhGVckpKAvKpUarBSxBqOpy4GA9RYYCH/nKbgX6ikjXQMVjjDHBpKisktKKqqAdJAfOtkF8BVwEICJjgT5Asm+fAktFZI2IzKnvJCIyR0TSRSQ9Ly8voAEbY0xzyakeJNceaxAN8CDQUUTWA7cC64Aq375JqnoycDZws4icVtdJVPVpVU1T1bSkpKSAB22MMc0h1zcGIimI2yAc63yrqgXANQDi7eO1E8j07dvje88VkTeAscByh0I1xphml1NYPQ+T1SCOISIdRSTc93E2sFxVC0QkRkTifGVigOmA355QxhjTWuW2gkdMAatBiMhCYCqQKCJZwH1AGICqPgkMAZ4REQUygJ/5Du0KvOEbOBIKvKiq7wUqTmOMcUJOQRkx4SHEBukoaghgglDVWcfZvxIY6Gd7JjAqUHEZY0wwyCkM7kFyYCOpjTHGEcE+SA4sQRhjzAn7vyVbuPDvn1NR5WnwMd55mKwGYYwxbVZ5pYeXVu1m3e7DzP9sZ4OO8Y6ithqEMca0aZ/tyKPAXUnPjlE8+uF2sg6VHPeYAncl7gqP1SCMMaYte2fDXuIiQ3lh9jgA7l+UgarWe0ywLxRUzRKEMcY0UVllFR9k5DBjWDf6Jsbw87MG8OGWXJZuzqnzGHdFFb9+cxMhLmFYjw4tGG3jWYIwxpgm+vTr/RSWVfKDkd0BuGZiCoO7xXH/ogyKyyqPKV9e6eGG59ewatdBHr50FCd1iWvpkBvFEoQxxjTR4o17iY8KY+JJiQCEhbj4/YUj2Jvv5pEPvj6ibGWVh7kvr2PZtjz+78IRXJDa04mQG8UShDHGNIG7oooPNucwc1g3wkK+/1U6pk8Cs8b25t8rdpGRnQ+Ax6Pc/fpGlmzcx73nDmHW2N5Ohd0oliCMMaYJPvk6j6KySs71PV6q7a6Zg+gYFcav3thElUf5zTubeXVNFnPPHMDsyf0ciLZpLEEYY0wTLN6wl4ToMCb073zMvo7R4fzq3CGs/+4wlz21kgUrdjF7Ugq3nzHAgUibzhKEMcY0kruiig+35DBz+JGPl2q7cHRPJvTrTPq3h5g1the/OncIvklIW43gnUbQGGOC1LJtuZSUV/GDkT3qLCMiPHZ5Kh9vy+VHY3q1uuQAliCMMabR3tmwl84x4YxL6VRvuS4dIrnslNbRIO2PPWIyxphGKCmv5L9bcpk5vBuhdTxeaiva9rczxphm9vHWPEorqvz2XmprLEEYY0wjLN6YTWJsOONSju291NYELEGIyHwRyRURv+tJi0iCiLwhIhtEZJWIDK+1b6aIbBORHSJyd6BiNMaYxiguq+SjrbmcPbw7Ia7W1+jcWIGsQSwAZtaz/5fAelUdCfwUeAxAREKAvwFnA0OBWSIyNIBxGmNMg3y0NRd3hadm7qW2LmAJQlWXAwfrKTIU+MhXdivQV0S6AmOBHaqaqarlwEvABYGK0xhjGmrxhr10iYsgrW/9vZfaCifbIL4CLgIQkbFAHyAZ6Al8V6tclm+bMcY46oudB5g2qEu7eLwEziaIB4GOIrIeuBVYB1Q19iQiMkdE0kUkPS8vr7ljNMYYAA4Wl3O4pIIBXWOdDqXFODZQTlULgGsAxDvEcCeQCUQBvWoVTQb21HOep4GnAdLS0upfxskYY5ooM68IgH5JMQ5H0nIcq0GISEcRCfd9nA0s9yWN1cAAEUnx7b8cWORUnMYYA5C5vxiAlESrQZwwEVkITAUSRSQLuA8IA1DVJ4EhwDMiokAG8DPfvkoRuQV4HwgB5qtqRqDiNMaYhsjMKyYsROiVEOV0KC0mYAlCVWcdZ/9KYGAd+5YASwIRlzHGNEVmXhG9O0W3+ek1ams/39QYY05A5v5i+iW1n8dLYAnCGGOOq7LKw7cHittVAzVYgjDGmOPKOlRKRZXSvx01UIMlCGOMOa7M/e2viytYgjDGmOPKzPN2cbU2CGOMMUf4Jq+YjtFhdIoJP37hNsQShDHGHMfO/UX0S2xfj5fAEoQxxhxXZl776+IKDUwQIhIjIi7fzwNF5HwRCQtsaMYY47xCdwW5hWXtroEaGl6DWA5EikhPYClwJd4FgYwxpk3b6ZuDqV876+IKDU8QoqoleNdv+LuqXgIMC1xYxhgTHL7vwWQ1iLqIiEwArgAW+7aFBCYkY4wJHpl5RbgE+nSOdjqUFtfQBDEXuAd4Q1UzRKQf8HHgwjLGmODwzf5ikhOiiQhtf38TN2g2V1X9BPgEwNdYvV9VbwtkYMYYEwy8PZja3+MlaHgvphdFpIOIxACbgM0iMi+woRljjLM8HvWNgWh/DdTQ8EdMQ32rvf0QeBdIwduTyRhj2qx9BW7cFR6rQRxHmG/cww+BRapaAdj6z8aYNq0992CChieIp4BdQAywXET6AAWBCsoYY4JB9Syu/dvhKGpoYIJQ1cdVtaeqnqNe3wLT6jtGROaLSK6IbKpjf7yIvC0iX4lIhohcU2tflYis970WNeobGWNMM8nMKyYmPIQucRFOh+KIhjZSx4vIwyKS7nv9GW9toj4LgJn17L8Z2Kyqo4CpwJ9FpHqqxFJVTfW9zm9IjMYY09y+ySuiX1IsIuJ0KI5o6COm+UAhcKnvVQD8u74DVHU5cLC+IkCceO98rK9sZQPjMcaYgGvPXVyhgeMggP6qenGtzw+IyPoTvPZfgUVANhAHXKaqHt++SBFJx5swHlTVN+s6iYjMAeYA9O7d+wRDMsYYL3dFFdn5pfRL7OV0KI5paA2iVEQmVX8QkYlA6QleewawHugBpAJ/FZEOvn19VDUN+DHwqIj0r+skqvq0qqapalpSUtIJhmSMMV479xej2n57MEHDaxA3AM+KSLzv8yHgqhO89jV4awcK7BCRncBgYJWq7gFQ1UwRWQaMBr45wesZY0yDVc/imtIOFwqq1tBeTF/5GpNHAiNVdTRw+gleezdwBoCIdAUGAZkikiAiEb7ticBEYPMJXssYYxolM8/bxdVqEA3kG01d7Q7g0brKishCvL2TEkUkC7gPCPOd50ngt8ACEdkICHCXqu4XkVOBp0TEgzeBPaiqliCMMS0qM6+Y7vGRRIc36tdkm3Ii37zefl+qOus4+7OB6X62rwBGnEBcxhhzwr7Z3757MMGJrUltU20YY9okVSUzr/1O0let3hqEiBTiPxEIEBWQiIwxxmH7i8opdFe2+xpEvQlCVeNaKhBjjAkW3zdQt+8axIk8YjLGmDYp09fFtV877uIKliCMMeYYmXlFRIS66NmxfT9JtwRhjDFH2bm/mJTEGFyu9jlJXzVLEMYYc5T2PklfNUsQxhhTS0WVh90HS9p9F1ewBGGMMUf4Jq+ISo9yUhdLEJYgjDGmls3Z3hmFhvXocJySbZ8lCGOMqSUju4DIMFe7HwMBliCMMeYIm7MLGNytAyHtvAcTWIIwxpgaqkpGdj5D7fESYAnCGGNqZB0qpcBdae0PPpYgjDHGZ/NebwP10O6WIMAShDHG1MjILsAlMLibJQiwBGGMMTU2Z+fTLymWqPAQp0MJCgFNECIyX0RyRWRTHfvjReRtEflKRDJE5Jpa+64Ske2+11WBjNMYY8Dbg8naH74X6BrEAmBmPftvBjar6ii861f/WUTCRaQT3jWsxwFjgftEJCHAsRpj2rFDxeVk57stQdQS0AShqsuBg/UVAeJERIBYX9lKYAbwgaoeVNVDwAfUn2iMMeaEZGRXN1DHOxxJ8HC6DeKvwBAgG9gI3K6qHqAn8F2tclm+bccQkTkiki4i6Xl5eYGO1xjTRm3emw9gYyBqcTpBzADWAz2AVOCvItKo/zqq+rSqpqlqWlJSUiBiNMa0AxnZBXSPj6RTTLjToQQNpxPENcDr6rUD2AkMBvYAvWqVS/ZtM8aYgLAG6mM5nSB2A2cAiEhXYBCQCbwPTBeRBF/j9HTfNmOMaXal5VV8k1fE0B7W/lBbaCBPLiIL8fZOShSRLLw9k8IAVPVJ4LfAAhHZCAhwl6ru9x37W2C171S/UdX6GruNMabJtu4rwKM2gvpoAU0QqjrrOPuz8dYO/O2bD8wPRFzGGFNb9RQb9ojpSE4/YjLGGMdlZBfQITKU5IQop0MJKpYgjDHtXkZ2AUN7dMA7JMtUswRhjGnXKqs8bN1bYAPk/LAEYYxp13buL6as0mPtD35YgjDGtGs1DdQ9LUEcLaC9mNqiKo+SX1rBoZJy8ksryC+p4HBpue+9gtLyKhJjI+gWH0n3+Ei6xUfStUMkYSGWi40JRhnZBYSHuuifFOt0KEHHEgSw6Ktsissqfa8qissrKSqrpKSskgJ3JQeLyzlcUs6hkgoK3BWo1n2u8BAX5VWeI7aJQOeYCMJCBI8qquBRAMWjEBYiRIWFEBkWQlR4CNHhIUSFhSAilFd6KKusorzSQ3mVh/JKD5VVSohLCHEJoSFCiMtFqO9zZFgIkaEuonzniPS9EqLDSIyNIDEugsTYcJJiI+gUE06oJS7TzmVk5zOoa5z9EeeHJQjg7tc2UFJeVfM5MsxFbEQo0eGhdIgKJSE6nF6dokmIDqNjdDgJ0WEkRIcTHx1GxyjvtvioMDpEhhLiEgrclezLd7M3v9T37ia30E2VRxEElwtEBMGbPCqrlJLyKkorqnBXVFFSXsWh4go8qkSEuggPdREdHkrHUBfhIS5CfYmmskqp8iiVHu97RZWH/NIKciu85yqtdc6KqmOzmgj0S4zh1P6JnNq/M+P7dSbB5qEx7Yiqsjm7gOlDuzkdSlCyBAEsvm0yUWEhREeEEBPu/SV/IuKjwoiPCmNQt7hmivDEqCpFZZXsLypnf1EZ+wvL2F9URl5hGRv25PPa2iye++JbRGBItw6c2r8zI5Lj6RAZRmxkKHGRocRGhBIXGUZsxInfH2OCxd58N4dKKqz9oQ6WIICUxBinQwgoESEuMoy4yDC/37WiysOGrMOs/OYAK745wLNffEt5pcfPmaBDZCjzZg7mirG9cVmiMK1c9RoQ1oPJP0sQhrAQF2P6dGJMn07ccvoA3BVVZB0qpdBdQVFZJYXuSgrdFRS6K/loay6/fnMTb63bwx8uGsGArsFRSzKmKTZnFyACg7tZgvDHEoQ5RmRYCCd18d+j42eTUnh1TRa/X7KFcx7/lJumnsRN0/oTEWqLvJvgpap+R0lnZOeT0jmGmAj7VeiP3RXTKCLCJWm9mDa4C795ezOP/Xc7izfu5bcXDCc5IYrDvm6/h0sqOFxSToG7ktG9OjKhf2ebxsC0uANFZdz12kY+3Z7H5AFJ/GBkd84Y0oW4yDDA+4gptXdHh6MMXpYgTJMkxkbw+KzRXHhyT+59YxOz/vFFveVTEmP48djeXDwm2VbsMi3i0+153PHKV+SXVvCDEd1ZmXmAD7fkEB7qYsrAJM4Y3IU9h0u5Ynxvp0MNWpYgzAmZNqgLS39+Gm9/lU2IS+gYHU7H6DASosOIjwonMszFh1tyeOGL3fx+yRYeen8b54zoxo/H9eGUvglWqzDNrrzSw5+WbuPp5ZkM6BLLs9eOZUj3Dng8yrrvDvHOhr28u3EfH2zOAWwNiPqI1jfqq5VJS0vT9PR0p8Mwddi2r5AXv/yW19ftodBdyYie8dw1czCTBiQ27kRTp3rfly1r7hBNK5eZV8RtL61j054CfjK+N786ZyhR4ce2j3k8ytrdh8jILuCKcb3b9YBREVmjqml+91mCMC2tpLySReuz+ctHO9hzuJRJJyVy18zBjEhu4GyaliCMH29/lc1dr20gPNTFHy8eyfRhNvitIepLEAFLmyIyX0RyRWRTHfvnich632uTiFSJSCffvuO8DY0AABPASURBVF0istG3z37jtzHR4aFcPrY3H/1iCr/+wVAysvM576+fccuLa9m1v9jp8EwrtHjDXm5/aR3De8Tz3u2nWXJoJoGsVy0AZta1U1UfUtVUVU0F7gE+OWrd6Wm+/X4zm2n9IkJD+NmkFD75n2ncevpJ/HdLLmc+/Am/eXszZZVVxz+BMcBHW3O4/aV1pPXpxDPXjqVbfKTTIbUZAUsQqrocOHjcgl6zgIWBisUEtw6RYdw5fRCfzJvKJWm9mP/5Tq74x5fsLypzOjQT5Fbs2M8Nz69laI8O/OvqNL/tDabpHG+ZEZFovDWN12ptVmCpiKwRkTnORGZaWpcOkfzhohH8ZdZoNmXnc/5fPiMjO9/psEyQWvPtIWY/m05K5xieuWZszdgG03wcTxDAecDnRz1emqSqJwNnAzeLyGl1HSwic0QkXUTS8/LyAh2raQHnjerBf64/FQUufmIFizfsdTokE2Q27cnn6n+voktcBM/NHmuzEAdIMCSIyznq8ZKq7vG95wJvAGPrOlhVn1bVNFVNS0pKCmigpuWMSI7nrVsmMrR7B25+cS0PL92Gx9N2etyZptuRW8hP568iLiKU52ePo0uctTkEiqMJQkTigSnAW7W2xYhIXPXPwHTAb08o07Z1iYtk4ZzxXDImmcc/2sENz6+xxut2zuNRfvZMOi4RXrhuPMkJ0U6H1KYFbCS1iCwEpgKJIpIF3AeEAajqk75iFwJLVbV238auwBu+EbahwIuq+l6g4jTBLSI0hD/+aCSDusXxu8VbuPeNTfwRsPHX7dP6rMN8e6CEhy8d1ean6Q8GAUsQqjqrAWUW4O0OW3tbJjAqMFGZ1khEmD25HwXuSh7/73buyHfT3boytkvvb9pHWIhwxpCuTofSLgRDG4QxDTL3jAGcPbwb3x4o5lBJhdPhmBamqry7aR+n9k8kPsp6LLUESxCm1XC5hD9fOoro8FB25BayPafQ6ZBMC9qyt5DdB0uYOdxGSbcUSxCmVYkOD2VQtzhcIsx+Np1DxeVOh2RayHsZ+3AJnDXUHi+1FEsQptWJCHUxsGscew+7ufGFNVRU+V8/27Qt72/axyl9O5EYG+F0KO2GrQdhWqW4yFAevHgEd7zyFf/7VgY/Gd+brEOlfHewhKxDpWQdKmHPYTdXju/Dj8fZgjCtXWZeEdtyCrnvvKFOh9KuWIIwrdZFJyfzdU4RT37yDQtX7a7ZHhMeQq9O0ZRXerh/UQaje3dkiC0K06q9l7EPgBk2S2uLsgRhWrV5MwYxoqd3HYlenaLolRBNx+gwRIQDRWXMePRTfv7yet68eSKRYTaRW2v1/qZ9jEqOp0fHKKdDaVesDcK0aiEu4dyR3Tl3ZHdGJnckISa8ZhnTzrERPPSjkWzdV8if3t/mcKSmqfYcLuWrrHxmDu/udCjtjiUI06ZNG9yFK8f34Z+f7eTzHfudDsc0wdKax0vWe6mlWYIwbd4vzxlCv6QY7nzlK/JtgF2r8+6mfQzqGke/pFinQ2l3LEGYNi8qPIRHL0tlf1EZ975l8z62JnmFZazeddAGxznEEoRpF0Ymd2TumQN4+6ts3lq/x+lwTAN9uCUHVSxBOMQShGk3bpx6Eml9Erj3zU3sOVzqdDimAd7btI8+naMZ3C3O6VDaJevmatqNEJfwyGWpzHx0Oaf98WOiwkKIDHMRERpCRPV7qIvIMJdvX/XLRUx4KClJMQzsGsfALnHER9tkcYGWX1rBim/2c+3ElJqeaaZlWYIw7UqvTtE8+7NxfLglh7IKD+7KKsoqPJRVVuGuea9if1E57ooq3L7the4K3BXfT+nRtUMEA7vGMbhbHLMn96NrB5t+vLl9tDWHiiq1x0sOsgRh2p0xfRIY0yehUcd4PMqew6Vszy3k65wivs4pZHtOEc+s+JZPt+/nPzdMIC6y/dQq1u4+RHJCVECX+3xv0z66dYhkVHLHgF3D1M8ShDEN4HIJvTpF06tTNKcP/r4//qfb87j636u5beE6/vHTNEJD2n6zXvbhUn70xAo6x0Yw/6pTGJEc3+zXKHBX8MnXeVyW1guXyx4vOaXt/2s2JoAmD0jigfOH8fG2PH63eIvT4bSIV9K/w6MQ6hIufWolH2zOafZr3PdWBhVVyiVpvZr93KbhApYgRGS+iOSKiN+O5yIyT0TW+16bRKRKRDr59s0UkW0iskNE7g5UjMY0h5+M78O1E1NYsGIXz63c5XQ4AVXlUV5e/R2TBySy6JZJDOway5zn0pn/2c5mu8Zb6/fwxro93Hb6AIb3bP7aiWm4QNYgFgAz69qpqg+paqqqpgL3AJ+o6kERCQH+BpwNDAVmiYjN8WuC2q/OHcIZg7tw/9ub+eTrPKfDCZhPvs5lb76bH4/tTVJcBC/NmcD0oV35zTubue+tTVSe4Noc3x0s4d43NjGmTwI3T+vfTFGbpgpYglDV5cDBBhafBSz0/TwW2KGqmapaDrwEXBCAEI1pNiEu4bFZoxnYNY5bXljL1210OdQXv/yOxNgIzvSt6hYVHsITV4xhzmn9eGblt8x5bg0Hi8spq6zC49FGnbvKo9zxynoUePSy1HbRnhPsHG+kFpFovDWNW3ybegLf1SqSBYyr5/g5wByA3r1tYRjjnNiIUP51VRo//NvnXLtgNW/ePLFNrX62L9/NR1tzuH5Kf8Jq/fJ2uYRfnjOE3p2iuW9RBif/9oOafSEuIdQlhIW4vPvPG8q4fp39nv+JZTtYvesQD186il6dogP+fczxOZ4ggPOAz1W1obWNI6jq08DTAGlpaY37k8WYZtajYxT/vCqNS59ayZxn03nxuvFtZh2K6sbpy0/x33D8k/F9GNK9A2u+PUhFlVJR5aHS915e5eGDzTlc9vQXzBrbi7vPHkJ81Pfdgtd/d5hHP9zOeaN6cOHoni31lcxxBEOCuJzvHy8B7AFq/wtM9m0zplUYmdyRRy5N5cYX1nL3axt45LLUVj8SuLpxetJJifTpHFNnufrGmMybMYhHPviaf322kw+35PLA+cM4e3g3SsqrmPvSOrp2iOR3Pxze6u9VW+LoQz4RiQemAG/V2rwaGCAiKSISjjeBLHIiPmOa6uwR3fnF9IG8uT6bvy/7xulwAMgvqaDA3bTpzpdvz2PP4VJmjW36Y9zo8FB+de5Q3rp5El3iIrjphbVc9+wa7n59I98eLOHPl446olZhnBewGoSILASmAokikgXcB4QBqOqTvmIXAktVtbj6OFWtFJFbgPeBEGC+qmYEKk5jAuXmaSexI7eIh97fRr/EGM4e0TIrouUVlrHm24Nk7i9mZ14xO/d7XweKywkPdXHxyclcNzmlUesrLPxyN51jwjlr6Ikv2jMiOZ63bp7Ivz7bySMffo27wsNNU/szvo62CeOcgCUIVZ3VgDIL8HaHPXr7EmBJ80dlTMsRER68eCS7D5bw81fWk5wQHZBRx7XtyC3ikidXcMi3MFKXuAhSEmOYPqwrKYkx7DpQwqtrsnhp9W7OGtKV66f0Y0yfTvWeM7fAzX+35jJ7cgrhoc3z0CE0xMX1U/pz9vDufLwt94RqJiZwgqENwpg2KzIshKeu9PZsmv3sahbdMilgE/vtOVzKlf/6khCXi1eun8DQHh2IjTj2f/E7zhrIsyt28ewX37J0cw5j+iRww5T+nDmki9/n//9Zk0WVR7n8lOb/Jd67czRXndq32c9rmod1NDYmwJLiIvjnVWkUuSuZ/Uw6peVVzX6NA0VlXPmvLykqq+TZa8cyNqWT3+QAkBgbwR3TB7Hi7tO5/7yh5BS4ue7ZdC55ciXrvzt8RFmPR1m4ajen9u9MSmLdjdOmbbIEYUwLGNK9A49dPppN2fnc8PwaDhSVNdu5C90VXP3v1WQfLmX+1acwtEeHBh0XHR7K1RNTWPaLqfzhohHsOlDMD//2OXNfWlezoNJnO/aTdejEGqdN62UJwpgWcubQrvz+hyNY8c1+pj+ynMUb9p7wOd0VVcx5dg1b9hbwxBVjOKVv/e0J/oSGuJg1tjfL5k3j5mn9WbJpH6f/aRkPvb+VBSt20SkmnOnDTrxx2rQ+liCMaUE/Htebd26dTM+EKG5+cS03vbCG/U2sTVRWebh14TpWZh7gz5eOYtrgLicUW2xEKPNmDOajO6cwc3g3/vbxN3y0NZcfjUkmIrRtDPYzjWMJwpgWNqhbHK/feCr/M3MQH27OZfojy3n7q2xUGz4RwOGScn7xn6/4YHMOD5w/jAtSm2/0cXJCNI9dPpo3b57IrLG9mT0ppdnObVoX68VkjANCQ1zcNPUkzhrSlV+8uoFbF67jzXV7uOyUXpw2MKnO6Tn2HC7ln59m8vLq7ygpr+LOswYGrBdQaq+OpPay1dzaM0sQxjhoQNc4XrthAv/8bCdPLPuG/27NJSY8hGmDu3DOiO5MHZREdHgoW/YW8PTyTBZ9lY0A56f24PrT+jOoW5zTX8G0YZYgjHFYaIiLG6b052eTUvgi8wBLNu5jacY+3tmwl8gwFwO7xrEhK5/o8BCuPrUvP5uUQo+OUU6HbdoBSxDGBImwEBeTByQxeUASv/vhcFbtPMi7m/aybvdhfjF9IFeO70t8tM1VZFqOJQhjglCIS5jQvzMT+tv8RMY51ovJGGOMX5YgjDHG+GUJwhhjjF+WIIwxxvhlCcIYY4xfliCMMcb4ZQnCGGOMX5YgjDHG+CWNmUEy2IlIHvBtA4rGA/knWK6uff62H72tvs+JwP4GxNZYDf3OTTnG7lP7u091xdYcx9h9atn7NEBV/S+Wrqrt7gU8faLl6trnb/vR2+r7DKQ7+Z3tPtl9cvpe2X1y9j7VfrXXR0xvN0O5uvb52370tuN9DoSmXMPuU/Me05buU1Ov05Bj7D45e59qtKlHTG2BiKSraprTcQQ7u08NY/epYew++ddeaxDB7GmnA2gl7D41jN2nhrH75IfVIIwxxvhlNQhjjDF+WYIwxhjjlyUIY4wxflmCaEVEZKqIfCoiT4rIVKfjCWYiEiMi6SLyA6djCVYiMsT3b+lVEbnR6XiClYj8UET+ISIvi8h0p+NpSZYgWoiIzBeRXBHZdNT2mSKyTUR2iMjdxzmNAkVAJJAVqFid1Ez3CeAu4JXAROm85rhPqrpFVW8ALgUmBjJepzTTfXpTVa8DbgAuC2S8wcZ6MbUQETkN7y/3Z1V1uG9bCPA1cBbeX/irgVlACPCHo05xLbBfVT0i0hV4WFWvaKn4W0oz3adRQGe8iXS/qr7TMtG3nOa4T6qaKyLnAzcCz6nqiy0Vf0tprvvkO+7PwAuquraFwndcqNMBtBequlxE+h61eSywQ1UzAUTkJeACVf0DUN+jkUNARCDidFpz3Cff47cYYChQKiJLVNUTyLhbWnP9e1LVRcAiEVkMtLkE0Uz/ngR4EHi3PSUHsAThtJ7Ad7U+ZwHj6iosIhcBM4COwF8DG1pQadR9UtVfAYjI1fhqXQGNLng09t/TVOAivH9sLAloZMGlUfcJuBU4E4gXkZNU9clABhdMLEG0Iqr6OvC603G0Fqq6wOkYgpmqLgOWORxG0FPVx4HHnY7DCdZI7aw9QK9an5N928yR7D41jN2nhrH71ECWIJy1GhggIikiEg5cDixyOKZgZPepYew+NYzdpwayBNFCRGQhsBIYJCJZIvIzVa0EbgHeB7YAr6hqhpNxOs3uU8PYfWoYu08nxrq5GmOM8ctqEMYYY/yyBGGMMcYvSxDGGGP8sgRhjDHGL0sQxhhj/LIEYYwxxi9LEKbNE5GiFr7eiha+XkcRuaklr2naB0sQxjSSiNQ7h5mqntrC1+wIWIIwzc4ShGmXRKS/iLwnImt8q/QN9m0/T0S+FJF1IvKhb+0NROR+EXlORD4HnvN9ni8iy0QkU0Ruq3XuIt/7VN/+V0Vkq4i84Js6GhE5x7dtjYg8LiLHrFkhIleLyCIR+Qj4r4jEish/RWStiGwUkQt8RR8E+ovIehF5yHfsPBFZLSIbROSBQN5L03bZbK6mvXoauEFVt4vIOODvwOnAZ8B4VVURmQ38D3Cn75ihwCRVLRWR+4HBwDQgDtgmIk+oasVR1xkNDAOygc+BiSKSDjwFnKaqO33TQdTlZGCkqh701SIuVNUCEUkEvhCRRcDdwHBVTQUQ77KYA/CueyB413s4TVWXN/lumXbJEoRpd0QkFjgV+I/vD3r4fgGmZOBlEekOhAM7ax26SFVLa31erKplQJmI5AJdOXYp2FWqmuW77nqgL94VzjJVtfrcC4E5dYT7gaoerA4d+D/fKmkevOsadPVzzHTfa53vcyzehGEJwjSKJQjTHrmAw9V/cR/lL3iXc13kW1Dn/lr7io8qW1br5yr8///UkDL1qX3NK4AkYIyqVojILrzLqh5NgD+o6lONvJYxR7A2CNPuqGoBsFNELgHvkpIiMsq3O57v1wa4KkAhbAP61VoK87IGHhcP5PqSwzSgj297Id7HXNXeB6711ZQQkZ4i0uWEozbtjtUgTHsQLSK1H/08jPev8SdE5F4gDHgJ+ApvjeE/InII+AhIae5gfG0YNwHviUgx3vUJGuIF4G0R2QikA1t95zsgIp+LyCa86ybPE5EhwErfI7Qi4CdAbnN/F9O22XTfxjhARGJVtcjXq+lvwHZVfcTpuIypzR4xGeOM63yN1hl4Hx1Ze4EJOlaDMMYY45fVIIwxxvhlCcIYY4xfliCMMcb4ZQnCGGOMX5YgjDHG+GUJwhhjjF//H5KGgimKy9CzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(<matplotlib.axes._subplots.AxesSubplot at 0x7ff986e53190>,\n",
       " 0.036492170789302746)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_finder.plot(show_lr=lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZhHutCseBxjJ"
   },
   "source": [
    "## Training the Emotion Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "q3FiLr3LBrjs",
    "outputId": "cda9d5ff-144a-4bcb-a404-b313ddf102de"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/modeling_auto.py:882: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
      "  FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "hparams = Namespace(\n",
    "    train_path=train_path,\n",
    "    val_path=val_path,\n",
    "    test_path=test_path,\n",
    "    batch_size=32,\n",
    "    warmup_steps=100,\n",
    "    epochs=1,\n",
    "    lr=lr,\n",
    "    accumulate_grad_batches=1\n",
    ")\n",
    "module = TrainingModule(hparams)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "N8Jv_U25B37g"
   },
   "outputs": [],
   "source": [
    "## garbage collection\n",
    "import gc; gc.collect()\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 867,
     "referenced_widgets": [
      "163afdc0b9264042b077d18e0eb52a6b",
      "1cb6a608fb424ebba37963363dd0bfbe",
      "8da4e895f98845e08267b25d32eeef50",
      "5193bb81089f4b0f8e4edd5b3f5217f3",
      "b0f51d2ab93c4eba9a43a5a05935600c",
      "8724d5cfe6a6463f83e9ad5f5954607b",
      "179400f1591c4b42980ad2a0b3fc785e",
      "2a7a08f6d7484ff1a3174d68ca167b8e",
      "096cd19c21354a7dac1c45daa1af460d",
      "a157658902f34eb3a7c63cbfcb3e53c7",
      "24b055e798604635abe3c346845cdacc",
      "7ab54ba7ccc9484a8c8b2605d35a6a9c",
      "55690e54a4ce492bbd234fd380be63ee",
      "85007e2fd3204764b63fbf2277c1005d",
      "84201a6f02fb405ea11b7716ab204a04",
      "7c368c0d395f492dbf31beadd460e281",
      "f4a53c58c5c64fd094d30aa26cd4d1d9",
      "f73a02a5670a4d66a3dc8e386a2f56e5",
      "db921fe611024aa9adcc3caa4dc89a39",
      "91dae13977c347079c8c2421acae1e67",
      "af78532809f14e63856e426d601e807e",
      "4b9f99f4c5ad44fa8f6c841bc00d8868"
     ]
    },
    "id": "oRnl4HXvB5-T",
    "outputId": "b3c07d45-af92-4226-f68a-bb97cc1c215e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/callback_connector.py:97: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=10)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.\n",
      "  f\"Setting `Trainer(progress_bar_refresh_rate={progress_bar_refresh_rate})` is deprecated in v1.5 and\"\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "Missing logger folder: /content/lightning_logs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:309: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  FutureWarning,\n",
      "\n",
      "  | Name  | Type             | Params\n",
      "-------------------------------------------\n",
      "0 | model | EmoModel         | 82.1 M\n",
      "1 | loss  | CrossEntropyLoss | 0     \n",
      "-------------------------------------------\n",
      "82.1 M    Trainable params\n",
      "0         Non-trainable params\n",
      "82.1 M    Total params\n",
      "328.492   Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "163afdc0b9264042b077d18e0eb52a6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Sanity Checking: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ab54ba7ccc9484a8c8b2605d35a6a9c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "RuntimeError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-29-9d9e678cd6b9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m                      accumulate_grad_batches=hparams.accumulate_grad_batches)\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)\u001b[0m\n\u001b[1;32m    770\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstrategy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    771\u001b[0m         self._call_and_handle_interrupt(\n\u001b[0;32m--> 772\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit_impl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_dataloaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mval_dataloaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdatamodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mckpt_path\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    773\u001b[0m         )\n\u001b[1;32m    774\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_call_and_handle_interrupt\u001b[0;34m(self, trainer_fn, *args, **kwargs)\u001b[0m\n\u001b[1;32m    722\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstrategy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlauncher\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlaunch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrainer_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrainer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    723\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 724\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mtrainer_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    725\u001b[0m         \u001b[0;31m# TODO: treat KeyboardInterrupt as BaseException (delete the code below) in v1.7\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    726\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyboardInterrupt\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexception\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_fit_impl\u001b[0;34m(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)\u001b[0m\n\u001b[1;32m    810\u001b[0m             \u001b[0mckpt_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel_provided\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel_connected\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlightning_module\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    811\u001b[0m         )\n\u001b[0;32m--> 812\u001b[0;31m         \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mckpt_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mckpt_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    813\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    814\u001b[0m         \u001b[0;32massert\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstopped\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, model, ckpt_path)\u001b[0m\n\u001b[1;32m   1235\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_checkpoint_connector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresume_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1236\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1237\u001b[0;31m         \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_stage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1238\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1239\u001b[0m         \u001b[0mlog\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetail\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{self.__class__.__name__}: trainer tearing down\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_run_stage\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1322\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredicting\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1323\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_predict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1324\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_train\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1325\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1326\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_pre_training_routine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_run_train\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1352\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1353\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_detect_anomaly\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_detect_anomaly\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1354\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1355\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1356\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_run_evaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0m_EVALUATE_OUTPUT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    202\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_start\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    205\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    206\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_restarting\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/fit_loop.py\u001b[0m in \u001b[0;36madvance\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    267\u001b[0m         )\n\u001b[1;32m    268\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"run_training_epoch\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 269\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mepoch_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_data_fetcher\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    270\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    271\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    202\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_start\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    205\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    206\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_restarting\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py\u001b[0m in \u001b[0;36madvance\u001b[0;34m(self, data_fetcher)\u001b[0m\n\u001b[1;32m    206\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    207\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"run_training_batch\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 208\u001b[0;31m                 \u001b[0mbatch_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    209\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    210\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch_progress\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mincrement_processed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    202\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_start\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    205\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    206\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_restarting\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/batch/training_batch_loop.py\u001b[0m in \u001b[0;36madvance\u001b[0;34m(self, batch, batch_idx)\u001b[0m\n\u001b[1;32m     86\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlightning_module\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautomatic_optimization\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m             \u001b[0moptimizers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_get_active_optimizers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_frequencies\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m             \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msplit_batch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     90\u001b[0m             \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmanual_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msplit_batch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    202\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_start\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    205\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    206\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_restarting\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36madvance\u001b[0;34m(self, batch, *args, **kwargs)\u001b[0m\n\u001b[1;32m    205\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_batch_idx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    206\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptim_progress\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_position\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 207\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_idx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    208\u001b[0m         )\n\u001b[1;32m    209\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36m_run_optimization\u001b[0;34m(self, split_batch, batch_idx, optimizer, opt_idx)\u001b[0m\n\u001b[1;32m    254\u001b[0m         \u001b[0;31m# gradient update with accumulated gradients\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    255\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 256\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizer_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mopt_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    257\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    258\u001b[0m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconsume_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36m_optimizer_step\u001b[0;34m(self, optimizer, opt_idx, batch_idx, train_step_and_backward_closure)\u001b[0m\n\u001b[1;32m    376\u001b[0m             \u001b[0mon_tpu\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccelerator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTPUAccelerator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    377\u001b[0m             \u001b[0musing_native_amp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mamp_backend\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mAMPType\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNATIVE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 378\u001b[0;31m             \u001b[0musing_lbfgs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_lbfgs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    379\u001b[0m         )\n\u001b[1;32m    380\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_call_lightning_module_hook\u001b[0;34m(self, hook_name, pl_module, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1594\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1595\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"[LightningModule]{pl_module.__class__.__name__}.{hook_name}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1596\u001b[0;31m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1597\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1598\u001b[0m         \u001b[0;31m# restore current_fx when nested context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/lightning.py\u001b[0m in \u001b[0;36moptimizer_step\u001b[0;34m(self, epoch, batch_idx, optimizer, optimizer_idx, optimizer_closure, on_tpu, using_native_amp, using_lbfgs)\u001b[0m\n\u001b[1;32m   1623\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1624\u001b[0m         \"\"\"\n\u001b[0;32m-> 1625\u001b[0;31m         \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptimizer_closure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1626\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1627\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0moptimizer_zero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepoch\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer_idx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/optimizer.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure, **kwargs)\u001b[0m\n\u001b[1;32m    166\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    167\u001b[0m         \u001b[0;32massert\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_strategy\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 168\u001b[0;31m         \u001b[0mstep_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_strategy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizer_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    169\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    170\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_on_after_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/strategies/strategy.py\u001b[0m in \u001b[0;36moptimizer_step\u001b[0;34m(self, optimizer, opt_idx, closure, model, **kwargs)\u001b[0m\n\u001b[1;32m    191\u001b[0m         \"\"\"\n\u001b[1;32m    192\u001b[0m         \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlightning_module\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 193\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprecision_plugin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mopt_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    194\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    195\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_setup_model_and_optimizers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mModule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizers\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOptimizer\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mModule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOptimizer\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/precision/precision_plugin.py\u001b[0m in \u001b[0;36moptimizer_step\u001b[0;34m(self, model, optimizer, optimizer_idx, closure, **kwargs)\u001b[0m\n\u001b[1;32m    153\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLightningModule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    154\u001b[0m             \u001b[0mclosure\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpartial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_wrap_closure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 155\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    156\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    157\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_track_grad_norm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrainer\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"pl.Trainer\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     63\u001b[0m                 \u001b[0minstance\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_step_count\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     64\u001b[0m                 \u001b[0mwrapped\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__get__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minstance\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 65\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mwrapped\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     66\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     67\u001b[0m             \u001b[0;31m# Note that the returned function here is no longer a bound method,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     86\u001b[0m                 \u001b[0mprofile_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Optimizer.step#{}.step\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m                 \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecord_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprofile_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m                     \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/optimization.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure)\u001b[0m\n\u001b[1;32m    330\u001b[0m         \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    331\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mclosure\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 332\u001b[0;31m             \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    333\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    334\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mgroup\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparam_groups\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/precision/precision_plugin.py\u001b[0m in \u001b[0;36m_wrap_closure\u001b[0;34m(self, model, optimizer, optimizer_idx, closure)\u001b[0m\n\u001b[1;32m    138\u001b[0m         \u001b[0mconsistent\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mthe\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mPrecisionPlugin\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0msubclasses\u001b[0m \u001b[0mthat\u001b[0m \u001b[0mcannot\u001b[0m \u001b[0;32mpass\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0mdirectly\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    139\u001b[0m         \"\"\"\n\u001b[0;32m--> 140\u001b[0;31m         \u001b[0mclosure_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    141\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_after_closure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    142\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mclosure_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    146\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    147\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 148\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    149\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    150\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36mclosure\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    132\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    133\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mClosureResult\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m         \u001b[0mstep_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_step_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    135\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mstep_output\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclosure_loss\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36m_training_step\u001b[0;34m(self, split_batch, batch_idx, opt_idx)\u001b[0m\n\u001b[1;32m    425\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    426\u001b[0m         \u001b[0;31m# manually capture logged metrics\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 427\u001b[0;31m         \u001b[0mtraining_step_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_strategy_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"training_step\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mstep_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    428\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstrategy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpost_training_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    429\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_call_strategy_hook\u001b[0;34m(self, hook_name, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1764\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1765\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"[Strategy]{self.strategy.__class__.__name__}.{hook_name}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1766\u001b[0;31m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1767\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1768\u001b[0m         \u001b[0;31m# restore current_fx when nested context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/strategies/strategy.py\u001b[0m in \u001b[0;36mtraining_step\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    331\u001b[0m         \"\"\"\n\u001b[1;32m    332\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprecision_plugin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_step_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 333\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtraining_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    334\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    335\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mpost_training_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-25-f7c84d7e269b>\u001b[0m in \u001b[0;36mtraining_step\u001b[0;34m(self, batch, batch_idx)\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mtraining_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"train\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     25\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mvalidation_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-25-f7c84d7e269b>\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, batch, step_name)\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstep_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"train\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m         \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m         \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m         \u001b[0mloss_key\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mf\"{step_name}_loss\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0mtensorboard_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mloss_key\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-25-f7c84d7e269b>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, X, *args)\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mtraining_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-13-c355932196ab>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_, *args)\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     21\u001b[0m         \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput_\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbase_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_mask\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattention_mask\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m         \u001b[0;31m# maybe do some pooling / RNNs... go crazy here!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m    858\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    859\u001b[0m             \u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 860\u001b[0;31m             \u001b[0mreturn_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreturn_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    861\u001b[0m         )\n\u001b[1;32m    862\u001b[0m         \u001b[0msequence_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mencoder_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m    531\u001b[0m                     \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    532\u001b[0m                     \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 533\u001b[0;31m                     \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    534\u001b[0m                 )\n\u001b[1;32m    535\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    415\u001b[0m             \u001b[0mhead_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    416\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 417\u001b[0;31m             \u001b[0mpast_key_value\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself_attn_past_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    418\u001b[0m         )\n\u001b[1;32m    419\u001b[0m         \u001b[0mattention_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself_attention_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    344\u001b[0m             \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    345\u001b[0m             \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 346\u001b[0;31m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    347\u001b[0m         )\n\u001b[1;32m    348\u001b[0m         \u001b[0mattention_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    239\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    240\u001b[0m         \u001b[0;31m# Take the dot product between \"query\" and \"key\" to get the raw attention scores.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 241\u001b[0;31m         \u001b[0mattention_scores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery_layer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey_layer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    242\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    243\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mposition_embedding_type\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"relative_key\"\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mposition_embedding_type\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"relative_key_query\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 11.17 GiB total capacity; 10.52 GiB already allocated; 13.81 MiB free; 10.56 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF"
     ]
    }
   ],
   "source": [
    "## train roughly for about 10-15 minutes with GPU enabled.\n",
    "trainer = pl.Trainer(gpus=1, max_epochs=hparams.epochs, progress_bar_refresh_rate=10,\n",
    "                     accumulate_grad_batches=hparams.accumulate_grad_batches)\n",
    "\n",
    "trainer.fit(module)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "id": "Y8kzE1AeB_ij",
    "outputId": "932f2f1f-fb96-4a4e-a169-85f2150afa80"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "________________________________________________________________________________\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     sadness   0.961872  0.955250  0.958549       581\n",
      "         joy   0.958580  0.932374  0.945295       695\n",
      "        love   0.806818  0.893082  0.847761       159\n",
      "       anger   0.936567  0.912727  0.924494       275\n",
      "        fear   0.886364  0.870536  0.878378       224\n",
      "    surprise   0.674699  0.848485  0.751678        66\n",
      "\n",
      "    accuracy                       0.923500      2000\n",
      "   macro avg   0.870817  0.902076  0.884359      2000\n",
      "weighted avg   0.926988  0.923500  0.924647      2000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    progress = [\"/\", \"-\", \"\\\\\", \"|\", \"/\", \"-\", \"\\\\\", \"|\"]\n",
    "    module.eval()\n",
    "    true_y, pred_y = [], []\n",
    "    for i, batch_ in enumerate(module.test_dataloader()):\n",
    "        (X, attn), y = batch_\n",
    "        batch = (X.cuda(), attn.cuda())\n",
    "        print(progress[i % len(progress)], end=\"\\r\")\n",
    "        y_pred = torch.argmax(module(batch), dim=1)\n",
    "        true_y.extend(y.cpu())\n",
    "        pred_y.extend(y_pred.cpu())\n",
    "print(\"\\n\" + \"_\" * 80)\n",
    "print(classification_report(true_y, pred_y, target_names=label2int.keys(), digits=len(emotions)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "U0_Z_4Pkl3fc",
    "outputId": "a45a771a-1566-4ef7-f935-52633d499c4b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sat Apr  2 02:35:31 2022       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
      "| N/A   50C    P0    61W / 149W |  11259MiB / 11441MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ifER7sn-Htge"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "RoBERTa_Fine_Tuning_Emotion_classification.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}