{ "cells": [ { "cell_type": "raw", "metadata": {}, "source": [ "---\n", "title: 12 Emotion Classification with Fine-tuned BERT\n", "description: Emotion classification using fine-tuned BERT model\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"Colab\"" ] }, { "cell_type": "markdown", "metadata": { "id": "Wj6eoKzotv5I" }, "source": [ "## Emotion Classification using Fine-tuned BERT model\n", "\n", "In this tutorial, I will show to fine-tune a language model (LM) for emotion classification with code adapted from this [tutorial](https://zablo.net/blog/post/custom-classifier-on-bert-model-guide-polemo2-sentiment-analysis/) by MARCIN ZABŁOCKI. I adapted his tutorial and modified the code to suit the emotion classification task using a different BERT model. Please refer to his tutorial for more detailed explanations for each code block. I really liked his tutorial because of the attention to detail and the use of high-level libraries to take care of certain parts of the model such as training and finding a good learning rate. \n", "\n", "Before you get started, make sure to enable `GPU` in the runtime and be sure to \n", "restart the runtime in this environment after installing the `pytorch-lr-finder` library.\n", "\n", "This tutorial is in a rough draft so if you find any issues with this tutorial or have any further questions reach out to me via [Twitter](https://twitter.com/omarsar0). \n", "\n", "Note that the notebook was created a little while back so if something break it's because the code is not compatible with the library changes.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "G2tokZqttmTA" }, "outputs": [], "source": [ "%%capture\n", "!pip install transformers tokenizers pytorch-lightning" ] }, { "cell_type": "markdown", "metadata": { "id": "I0jZnNegGhZj" }, "source": [ "Note: you need to Restart runtime after running this code segment" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "k9ZKIIGvuW5m" }, "outputs": [], "source": [ "%%capture\n", "!git clone https://github.com/davidtvs/pytorch-lr-finder.git && cd pytorch-lr-finder && python setup.py install" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "id": "qqRRWe4UuuIh", "outputId": "479b5e60-10b5-4d84-c8fc-291ec32feceb" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'1.10.0+cu111'" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import torch\n", "from torch import nn\n", "from typing import List\n", "import torch.nn.functional as F\n", "from transformers import DistilBertTokenizer, AutoTokenizer, AutoModelWithLMHead, DistilBertForSequenceClassification, AdamW, get_linear_schedule_with_warmup\n", "import logging\n", "import os\n", "from functools import lru_cache\n", "from tokenizers import ByteLevelBPETokenizer\n", "from tokenizers.processors import BertProcessing\n", "import pytorch_lightning as pl\n", "from torch.utils.data import DataLoader, Dataset\n", "import pandas as pd\n", "from argparse import Namespace\n", "from sklearn.metrics import classification_report\n", "torch.__version__" ] }, { "cell_type": "markdown", "metadata": { "id": "_whSBDujRiga" }, "source": [ "## Load the Pretrained Language Model\n", "We are first going to look at pretrained language model provided by HuggingFace models. We will use a variant of BERT, called DistilRoBERTa base. The `base` model has less parameters than the `larger` model. \n", "\n", "[RoBERTa](https://arxiv.org/abs/1907.11692) is a variant of of BERT which \"*modifies key hyperparameters, removing the next-sentence pretraining objective and training with much larger mini-batches and learning rates*\".\n", "\n", "Knowledge distillation help to train smaller LMs with similar performance and potential." ] }, { "cell_type": "markdown", "metadata": { "id": "BvHNcMckSR4M" }, "source": [ "First, let's load the tokenizer for this model:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 145, "referenced_widgets": [ "bfa062ba95014af8993654adebeed30b", "494133cceef14cae8fa7efa5d18bfb45", "522e5a6c954c4297b21f8e0cea44a171", "269ff42456ee47bba72aca0923542c29", "511c63b5846447f1a98f99efa7b414ef", "9f2624b1ec3a4de3aeac12b403131745", "55662051a2f845c2a9de8fc27eea8485", "b93fa5f05abe46f8a15cd4b93f743243", "6b01657a3a0f47569757d62cff6970bd", "8fcfdc4bfbf74c73a38aa13175269565", "352725603b554637bceb08047c134523", "7b89a5097be146cb94844ff9af75bb77", "dd64efe54437454c9d919530cceadff7", "ccdae4645e744fd1a083d857d0e56a0a", "9404f7978dba4adfb4b757b8446bfae5", "43fdffaac68940beb54c21e0f5d21135", "0e3236fcfc07425ab83f1b158e3c0f00", "cb4074b47b464adb863bf9d8048ff10a", "bc402b2b5ac64749abee5ef047a78022", "b5cc3132707b4ee5885bc3bab6b384f4", "4f896b64ee4c4102aa5826f3927ed297", "eb8e48d71030482286bba58939a3ba25", "2ab5305315964703bf834e87f1881aaf", "aa7dcbcebf7e450da1d472da998b93ab", "728bb27db36d49319bcd129245ce2ade", "f8c4c65ab8eb4602b9cc67a18f17a80f", "52f517a67ec549d8bb3d952bc5c8471e", "a065514ff5ab45418085d2b5450c81ea", "09efbdba1ef1407aab615418a44a6571", "ae1118ec19ec4fb095c1e5616c366ee2", "a67576e0883e45de9f7c6e25d32601e4", "f80c7217e70b408684909176198e8720", "e894699ebf924a978689b1754e28449e", "93f62d912f43409ab8c9bd245b137bde", "0df8766d48974f799cf4a253ebfd8206", "a6f26948e4fe449b99e58a584be39492", "d1519847727e42a6b6a46fd1b425068c", "843b224c4f454a2695bbc99869987d80", "3f5081a276b04d38aca64760de0a078e", "4f51720e43824ae59e343090b4718f15", "697378036251425d965f8e52b36cfa60", "e940a875278a4ce297c7fe35be8014f8", "7306f377658349a5a36123251c68be41", "0a8b8f25fda04d358c8eb4575b5998fd" ] }, "id": "BPbTd5lmuzQn", "outputId": "68c9cb51-f420-45a7-e500-235fd96c2038" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "bfa062ba95014af8993654adebeed30b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading: 0%| | 0.00/480 [00:00Elvis is the king of rock\n", "Length: 9\n", "torch.Size([9, 768])\n" ] } ], "source": [ "t = \"Elvis is the king of rock\"\n", "enc = tokenizer.encode_plus(t)\n", "token_representations = base_model(torch.tensor(enc[\"input_ids\"]).unsqueeze(0))[0][0]\n", "print(enc[\"input_ids\"])\n", "print(tokenizer.decode(enc[\"input_ids\"]))\n", "print(f\"Length: {len(enc['input_ids'])}\")\n", "print(token_representations.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "9RFifOoY7Hsc" }, "source": [ "## Building Custom Classification head on top of LM base model" ] }, { "cell_type": "markdown", "metadata": { "id": "vSUMm4Oq7nvR" }, "source": [ "Use Mish activiation function as in the one proposed in the original tutorial" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tCEDXLxq628O" }, "outputs": [], "source": [ "# from https://github.com/digantamisra98/Mish/blob/b5f006660ac0b4c46e2c6958ad0301d7f9c59651/Mish/Torch/mish.py\n", "@torch.jit.script\n", "def mish(input):\n", " return input * torch.tanh(F.softplus(input))\n", " \n", "class Mish(nn.Module):\n", " def forward(self, input):\n", " return mish(input)" ] }, { "cell_type": "markdown", "metadata": { "id": "C6Ln6KWm74ku" }, "source": [ "The model we will use to do the fine-tuning" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "9VDRSRsc71H2" }, "outputs": [], "source": [ "class EmoModel(nn.Module):\n", " def __init__(self, base_model, n_classes, base_model_output_size=768, dropout=0.05):\n", " super().__init__()\n", " self.base_model = base_model\n", " \n", " self.classifier = nn.Sequential(\n", " nn.Dropout(dropout),\n", " nn.Linear(base_model_output_size, base_model_output_size),\n", " Mish(),\n", " nn.Dropout(dropout),\n", " nn.Linear(base_model_output_size, n_classes)\n", " )\n", " \n", " for layer in self.classifier:\n", " if isinstance(layer, nn.Linear):\n", " layer.weight.data.normal_(mean=0.0, std=0.02)\n", " if layer.bias is not None:\n", " layer.bias.data.zero_()\n", "\n", " def forward(self, input_, *args):\n", " X, attention_mask = input_\n", " hidden_states = self.base_model(X, attention_mask=attention_mask)\n", " \n", " # maybe do some pooling / RNNs... go crazy here!\n", " \n", " # use the representation\n", " return self.classifier(hidden_states[0][:, 0, :])" ] }, { "cell_type": "markdown", "metadata": { "id": "wjgME-3O8Yfo" }, "source": [ "### Pretest the model with dummy text\n", "We want to ensure that the model is returing the right information back." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Y6H9eF8A8XeV", "outputId": "4cc0eccb-40bb-420d-f679-7fa3b548b9a6" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/modeling_auto.py:882: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n", " FutureWarning,\n" ] } ], "source": [ "classifier = EmoModel(AutoModelWithLMHead.from_pretrained(\"distilroberta-base\").base_model, 3)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "-sjfHJ_L9iNH" }, "outputs": [], "source": [ "X = torch.tensor(enc[\"input_ids\"]).unsqueeze(0).to('cpu')\n", "attn = torch.tensor(enc[\"attention_mask\"]).unsqueeze(0).to('cpu')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "o6QhCuEC-y2z", "outputId": "2dd22943-cb2c-4235-faae-a9024cbf69ec" }, "outputs": [ { "data": { "text/plain": [ "tensor([[ 0.0115, -0.1552, 0.0227]], grad_fn=)" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "classifier((X, attn))" ] }, { "cell_type": "markdown", "metadata": { "id": "I-N7WSY7Cb7v" }, "source": [ "## Prepare your dataset for fine-tuning" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jDWkjaLV-5tj" }, "outputs": [], "source": [ "!mkdir -p tokenizer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wMMm5Ye1Db-m", "outputId": "0babb6a0-f763-4684-82f9-55819619a807" }, "outputs": [ { "data": { "text/plain": [ "('tokenizer/tokenizer_config.json',\n", " 'tokenizer/special_tokens_map.json',\n", " 'tokenizer/vocab.json',\n", " 'tokenizer/merges.txt',\n", " 'tokenizer/added_tokens.json',\n", " 'tokenizer/tokenizer.json')" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## load pretrained tokenizer information\n", "tokenizer.save_pretrained(\"tokenizer\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3FVtbmrzDkF8", "outputId": "bcbae7d2-f35a-4118-81f4-10f7638f7e50" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "merges.txt\t\t tokenizer_config.json\tvocab.json\n", "special_tokens_map.json tokenizer.json\n" ] } ], "source": [ "!ls tokenizer" ] }, { "cell_type": "markdown", "metadata": { "id": "BhTEgIaLEDRo" }, "source": [ "Implement CollateFN using fast tokenizers.\n", "This function basically takes care of proper tokenization and batches of sequences. This way you don't need to create your batches manually. Find out more about Tokenizers [here](https://github.com/huggingface/tokenizers/tree/master/bindings/python)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "3SCLBZsMDn4s" }, "outputs": [], "source": [ "class TokenizersCollateFn:\n", " def __init__(self, max_tokens=512):\n", "\n", " ## RoBERTa uses BPE tokenizer similar to GPT\n", " t = ByteLevelBPETokenizer(\n", " \"tokenizer/vocab.json\",\n", " \"tokenizer/merges.txt\"\n", " )\n", " t._tokenizer.post_processor = BertProcessing(\n", " (\"\", t.token_to_id(\"\")),\n", " (\"\", t.token_to_id(\"\")),\n", " )\n", " t.enable_truncation(max_tokens)\n", " t.enable_padding(length=max_tokens, pad_id=t.token_to_id(\"\"))\n", " self.tokenizer = t\n", "\n", " def __call__(self, batch):\n", " encoded = self.tokenizer.encode_batch([x[0] for x in batch])\n", " sequences_padded = torch.tensor([enc.ids for enc in encoded])\n", " attention_masks_padded = torch.tensor([enc.attention_mask for enc in encoded])\n", " labels = torch.tensor([x[1] for x in batch])\n", " \n", " return (sequences_padded, attention_masks_padded), labels" ] }, { "cell_type": "markdown", "metadata": { "id": "4hu70Ng0Eqls" }, "source": [ "## Getting the Data and Preview it\n", "Below we are going to load the data and show you how to create the splits. However, we don't need to split the data manually becuase I have already created the splits and stored those files seperately which you can quickly download below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "JZ3SoJH3fUsq", "outputId": "4b4adfed-68b5-4673-f010-96ac2e578804" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2022-04-02 02:29:43-- https://www.dropbox.com/s/ikkqxfdbdec3fuj/test.txt\n", "Resolving www.dropbox.com (www.dropbox.com)... 162.125.80.18, 2620:100:601b:18::a27d:812\n", "Connecting to www.dropbox.com (www.dropbox.com)|162.125.80.18|:443... connected.\n", "HTTP request sent, awaiting response... 301 Moved Permanently\n", "Location: /s/raw/ikkqxfdbdec3fuj/test.txt [following]\n", "--2022-04-02 02:29:44-- https://www.dropbox.com/s/raw/ikkqxfdbdec3fuj/test.txt\n", "Reusing existing connection to www.dropbox.com:443.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: https://uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com/cd/0/inline/BindkPg73vrwbieJ5UPi7q0aVj4zSdWDIAJEXTLje7fzw332Q4is5tqTFH6-p-vAaxn9i19935V16q5W7VLMnWygO4NIy8JPhtF_og-e53ggh1bjKDWOubFHkfDLdqFeUpBy_deZU3Mq24B26W7AuDV2n-mg0iFL1CUofID0gxW3Kw/file# [following]\n", "--2022-04-02 02:29:44-- https://uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com/cd/0/inline/BindkPg73vrwbieJ5UPi7q0aVj4zSdWDIAJEXTLje7fzw332Q4is5tqTFH6-p-vAaxn9i19935V16q5W7VLMnWygO4NIy8JPhtF_og-e53ggh1bjKDWOubFHkfDLdqFeUpBy_deZU3Mq24B26W7AuDV2n-mg0iFL1CUofID0gxW3Kw/file\n", "Resolving uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com (uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com)... 162.125.8.15, 2620:100:601c:15::a27d:60f\n", "Connecting to uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com (uc47a044c3484cacd77f1998a9af.dl.dropboxusercontent.com)|162.125.8.15|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 206760 (202K) [text/plain]\n", "Saving to: ‘test.txt’\n", "\n", "test.txt 100%[===================>] 201.91K --.-KB/s in 0.05s \n", "\n", "2022-04-02 02:29:45 (3.82 MB/s) - ‘test.txt’ saved [206760/206760]\n", "\n", "--2022-04-02 02:29:45-- https://www.dropbox.com/s/1pzkadrvffbqw6o/train.txt\n", "Resolving www.dropbox.com (www.dropbox.com)... 162.125.80.18, 2620:100:601b:18::a27d:812\n", "Connecting to www.dropbox.com (www.dropbox.com)|162.125.80.18|:443... connected.\n", "HTTP request sent, awaiting response... 301 Moved Permanently\n", "Location: /s/raw/1pzkadrvffbqw6o/train.txt [following]\n", "--2022-04-02 02:29:46-- https://www.dropbox.com/s/raw/1pzkadrvffbqw6o/train.txt\n", "Reusing existing connection to www.dropbox.com:443.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: https://uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com/cd/0/inline/BilN3fBwXofFrI1cRcd9FtYqHNPbVbTn1_yppqDJ7X-LrJJhV_knWzdwIaw03J-6tBnJCcH5rd6QvOFO2_WQ0FpfkezzCc0a1OfVMsJ2J0VvVEYH-893SFHLPPHpK-vTGrLX_Pq136GsnSKOvUD4_j6IcG29LQZIegH-zv_h6dgUbw/file# [following]\n", "--2022-04-02 02:29:46-- https://uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com/cd/0/inline/BilN3fBwXofFrI1cRcd9FtYqHNPbVbTn1_yppqDJ7X-LrJJhV_knWzdwIaw03J-6tBnJCcH5rd6QvOFO2_WQ0FpfkezzCc0a1OfVMsJ2J0VvVEYH-893SFHLPPHpK-vTGrLX_Pq136GsnSKOvUD4_j6IcG29LQZIegH-zv_h6dgUbw/file\n", "Resolving uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com (uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com)... 162.125.3.15, 2620:100:601c:15::a27d:60f\n", "Connecting to uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com (uc28e6937dfec019d31854dd7ae5.dl.dropboxusercontent.com)|162.125.3.15|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 1658616 (1.6M) [text/plain]\n", "Saving to: ‘train.txt’\n", "\n", "train.txt 100%[===================>] 1.58M --.-KB/s in 0.07s \n", "\n", "2022-04-02 02:29:46 (24.0 MB/s) - ‘train.txt’ saved [1658616/1658616]\n", "\n", "--2022-04-02 02:29:46-- https://www.dropbox.com/s/2mzialpsgf9k5l3/val.txt\n", "Resolving www.dropbox.com (www.dropbox.com)... 162.125.80.18, 2620:100:601b:18::a27d:812\n", "Connecting to www.dropbox.com (www.dropbox.com)|162.125.80.18|:443... connected.\n", "HTTP request sent, awaiting response... 301 Moved Permanently\n", "Location: /s/raw/2mzialpsgf9k5l3/val.txt [following]\n", "--2022-04-02 02:29:47-- https://www.dropbox.com/s/raw/2mzialpsgf9k5l3/val.txt\n", "Reusing existing connection to www.dropbox.com:443.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: https://uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com/cd/0/inline/BinDvaaKEwfVAZXuoMsG0dUJTaMOqqlvpcEUBGJ6T-1huQAZIZlOy5DP5WiDm60aNF3zascBKEBnyXQsBIdcOfUvxkpMqw_NNwWfEUYlRRc4D9qEYnm4DYLIfhW2TJtRKNBzB1bPC2KNpOCZTFbkMkeQHGaVEZ4Tk214eeFK-seb1g/file# [following]\n", "--2022-04-02 02:29:48-- https://uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com/cd/0/inline/BinDvaaKEwfVAZXuoMsG0dUJTaMOqqlvpcEUBGJ6T-1huQAZIZlOy5DP5WiDm60aNF3zascBKEBnyXQsBIdcOfUvxkpMqw_NNwWfEUYlRRc4D9qEYnm4DYLIfhW2TJtRKNBzB1bPC2KNpOCZTFbkMkeQHGaVEZ4Tk214eeFK-seb1g/file\n", "Resolving uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com (uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com)... 162.125.3.15, 2620:100:6022:15::a27d:420f\n", "Connecting to uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com (uc33b67c4e8a3d54070f7e57d2c2.dl.dropboxusercontent.com)|162.125.3.15|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 204240 (199K) [text/plain]\n", "Saving to: ‘val.txt’\n", "\n", "val.txt 100%[===================>] 199.45K --.-KB/s in 0.03s \n", "\n", "2022-04-02 02:29:48 (5.57 MB/s) - ‘val.txt’ saved [204240/204240]\n", "\n" ] } ], "source": [ "!wget https://www.dropbox.com/s/ikkqxfdbdec3fuj/test.txt\n", "!wget https://www.dropbox.com/s/1pzkadrvffbqw6o/train.txt\n", "!wget https://www.dropbox.com/s/2mzialpsgf9k5l3/val.txt" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "r_03fxufWX_G" }, "outputs": [], "source": [ "## export the datasets as txt files\n", "## EXERCISE: Change this to an address\n", "\n", "train_path = \"train.txt\"\n", "test_path = \"test.txt\"\n", "val_path = \"val.txt\"\n", "\n", "## emotion labels\n", "label2int = {\n", " \"sadness\": 0,\n", " \"joy\": 1,\n", " \"love\": 2,\n", " \"anger\": 3,\n", " \"fear\": 4,\n", " \"surprise\": 5\n", "}\n", "\n", "emotions = [ \"sadness\", \"joy\", \"love\", \"anger\", \"fear\", \"surprise\"]" ] }, { "cell_type": "markdown", "metadata": { "id": "-FJ-wN1_zmkV" }, "source": [ "### A Quick Look at the dataset\n", "Below is a few code sniphets to get a good idea of the dataset we are using here. You can skip this whole subsection if you like." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "t23zHggkEpc-", "outputId": "60c12470-7ff0-43fb-a867-321fd0266862" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2022-04-02 01:07:36-- https://www.dropbox.com/s/607ptdakxuh5i4s/merged_training.pkl\n", "Resolving www.dropbox.com (www.dropbox.com)... 162.125.3.18, 2620:100:6018:18::a27d:312\n", "Connecting to www.dropbox.com (www.dropbox.com)|162.125.3.18|:443... connected.\n", "HTTP request sent, awaiting response... 301 Moved Permanently\n", "Location: /s/raw/607ptdakxuh5i4s/merged_training.pkl [following]\n", "--2022-04-02 01:07:36-- https://www.dropbox.com/s/raw/607ptdakxuh5i4s/merged_training.pkl\n", "Reusing existing connection to www.dropbox.com:443.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: https://uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com/cd/0/inline/Biks1uZUpPg-4hPIU8S6gBvKR2bKR8WtT5cwtOf4Kc8EbggGQBsIjoyL2n3m3mrxFeoFYX6uWurmaLJRYsVqqWRzGyzLF_JBk6frRedoLHUAC4BoZMNUV624AW9XwRGyXvyYa0W4_P6I0lHGmx9xgcGpqkS0C4_J99RwktDpqp8BuQ/file# [following]\n", "--2022-04-02 01:07:37-- https://uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com/cd/0/inline/Biks1uZUpPg-4hPIU8S6gBvKR2bKR8WtT5cwtOf4Kc8EbggGQBsIjoyL2n3m3mrxFeoFYX6uWurmaLJRYsVqqWRzGyzLF_JBk6frRedoLHUAC4BoZMNUV624AW9XwRGyXvyYa0W4_P6I0lHGmx9xgcGpqkS0C4_J99RwktDpqp8BuQ/file\n", "Resolving uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com (uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com)... 162.125.4.15, 2620:100:6018:15::a27d:30f\n", "Connecting to uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com (uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com)|162.125.4.15|:443... connected.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: /cd/0/inline2/Bikixw43BxlYqPt-13I6GiwXazCK74atIS0dtaeeTf_dj-Wkjl_9eczKwufjPW3iO25EISJ5q5RXb8IME7cJfu4G9vGjXM9klJeLpUoZXzNMHsECzUtKaoyOoCnmaUvUrP_r4-YJXoNwkYnJxXUeOXH-aBXaLnKBsc3cOdc_2sTUxrCVd244Mu7EIaTG3mAdy76eCK3SgNiTTyExcShQVZz7-xmtJ_qKccsnJC6sdAOuATCkS42esAwgk88MiWyOsUi5N0DvfRnOb6kioBX1cObwiZ-bwb53p-fP_Os0WeidckaaHkkh24Wij4rtPJaP68L8A2B1_wTkTvNZt4B1YOGG_-I08i-6KlrRSbz-1EnbIjXfS-_589CBiZDbAjJ-tbH61_RKak64LiE9BPo7FHJtlO3vmIsFMyjW1Pg_N7EU1A/file [following]\n", "--2022-04-02 01:07:37-- https://uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com/cd/0/inline2/Bikixw43BxlYqPt-13I6GiwXazCK74atIS0dtaeeTf_dj-Wkjl_9eczKwufjPW3iO25EISJ5q5RXb8IME7cJfu4G9vGjXM9klJeLpUoZXzNMHsECzUtKaoyOoCnmaUvUrP_r4-YJXoNwkYnJxXUeOXH-aBXaLnKBsc3cOdc_2sTUxrCVd244Mu7EIaTG3mAdy76eCK3SgNiTTyExcShQVZz7-xmtJ_qKccsnJC6sdAOuATCkS42esAwgk88MiWyOsUi5N0DvfRnOb6kioBX1cObwiZ-bwb53p-fP_Os0WeidckaaHkkh24Wij4rtPJaP68L8A2B1_wTkTvNZt4B1YOGG_-I08i-6KlrRSbz-1EnbIjXfS-_589CBiZDbAjJ-tbH61_RKak64LiE9BPo7FHJtlO3vmIsFMyjW1Pg_N7EU1A/file\n", "Reusing existing connection to uc85825e9ae09b5cee10fd3c90a5.dl.dropboxusercontent.com:443.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 49991846 (48M) [application/octet-stream]\n", "Saving to: ‘merged_training.pkl’\n", "\n", "merged_training.pkl 100%[===================>] 47.68M 51.9MB/s in 0.9s \n", "\n", "2022-04-02 01:07:38 (51.9 MB/s) - ‘merged_training.pkl’ saved [49991846/49991846]\n", "\n" ] } ], "source": [ "!wget https://www.dropbox.com/s/607ptdakxuh5i4s/merged_training.pkl" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PQrMSUTRF06B" }, "outputs": [], "source": [ "import pickle\n", "\n", "## helper function\n", "def load_from_pickle(directory):\n", " return pickle.load(open(directory,\"rb\"))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 315 }, "id": "XGz89mNSHaYM", "outputId": "1340de4b-868b-4f86-882c-11bd810d4cf6" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEZCAYAAAB7HPUdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAZbUlEQVR4nO3df7RdZX3n8ffHRH5olQS5zcIkGNQMDo4K9BZodc0olBBACVagOCpZNDazHPzVdqZEp5YpaBfWjo50KmOUaHApEGktKaCYRmkrlB8JMlBAmgvCIimQaGKwUn76mT/2c+UQ7s09F849OznP57XWXWfvZz/nnO+Gm8/Z99nP3ke2iYiIOryg7QIiIqJ/EvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERWZMPQlHSTplo6fhyV9WNK+ktZI2lAeZ5b+knS+pBFJt0o6rOO1Fpf+GyQtnsodi4iIZ9Nk5ulLmgZsAo4AzgS22j5P0jJgpu2zJB0PfAA4vvT7rO0jJO0LrAOGAQPrgV+xva2nexQREeOaPsn+RwN3275P0iLgzaV9JXANcBawCLjIzafJ9ZJmSNq/9F1jeyuApDXAQuDi8d5sv/3287x58yZZYkRE3davX/8j20NjbZts6J/G0yE9y/YDZflBYFZZng3c3/GcjaVtvPZnkLQUWApwwAEHsG7dukmWGBFRN0n3jbet6xO5kvYATgS+vuO2clTfk/s52F5ue9j28NDQmB9UERHxHE1m9s5xwM22HyrrD5VhG8rj5tK+CZjb8bw5pW289oiI6JPJhP47eeb4+2pgdAbOYuDyjvbTyyyeI4HtZRjoamCBpJllps+C0hYREX3S1Zi+pBcDxwD/paP5PGCVpCXAfcCppf0qmpk7I8AjwBkAtrdKOhe4qfQ7Z/SkbkRE9Mekpmz22/DwsHMiNyJiciSttz081rZckRsRUZGEfkRERRL6EREVmezFWbuFecuu7Ov73XveCX19v4iI5ypH+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUpKvQlzRD0mWSfiDpTkm/JmlfSWskbSiPM0tfSTpf0oikWyUd1vE6i0v/DZIWT9VORUTE2Lo90v8s8C3brwHeANwJLAPW2p4PrC3rAMcB88vPUuACAEn7AmcDRwCHA2ePflBERER/TBj6kvYB/iNwIYDtx23/BFgErCzdVgInleVFwEVuXA/MkLQ/cCywxvZW29uANcDCnu5NRETsVDdH+gcCW4AvSfq+pC9KejEwy/YDpc+DwKyyPBu4v+P5G0vbeO3PIGmppHWS1m3ZsmVyexMRETvVTehPBw4DLrB9KPAznh7KAcC2AfeiINvLbQ/bHh4aGurFS0ZERNFN6G8ENtq+oaxfRvMh8FAZtqE8bi7bNwFzO54/p7SN1x4REX0yYejbfhC4X9JBpelo4A5gNTA6A2cxcHlZXg2cXmbxHAlsL8NAVwMLJM0sJ3AXlLaIiOiT6V32+wDwVUl7APcAZ9B8YKyStAS4Dzi19L0KOB4YAR4pfbG9VdK5wE2l3zm2t/ZkLyIioitdhb7tW4DhMTYdPUZfA2eO8zorgBWTKTAiInqn2yP92IXMW3ZlX9/v3vNO6Ov7RcTUyW0YIiIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIq0lXoS7pX0m2SbpG0rrTtK2mNpA3lcWZpl6TzJY1IulXSYR2vs7j03yBp8dTsUkREjGcyR/pvsX2I7eGyvgxYa3s+sLasAxwHzC8/S4ELoPmQAM4GjgAOB84e/aCIiIj+eD7DO4uAlWV5JXBSR/tFblwPzJC0P3AssMb2VtvbgDXAwufx/hERMUndhr6Bb0taL2lpaZtl+4Gy/CAwqyzPBu7veO7G0jZe+zNIWippnaR1W7Zs6bK8iIjoxvQu+73J9iZJvwyskfSDzo22Lcm9KMj2cmA5wPDwcE9eMyIiGl0d6dveVB43A9+gGZN/qAzbUB43l+6bgLkdT59T2sZrj4iIPpkw9CW9WNJLRpeBBcA/AauB0Rk4i4HLy/Jq4PQyi+dIYHsZBroaWCBpZjmBu6C0RUREn3QzvDML+Iak0f5fs/0tSTcBqyQtAe4DTi39rwKOB0aAR4AzAGxvlXQucFPpd47trT3bk4iImNCEoW/7HuANY7T/GDh6jHYDZ47zWiuAFZMvMyIieiFX5EZEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRka5DX9I0Sd+XdEVZP1DSDZJGJF0qaY/SvmdZHynb53W8xkdK+12Sju31zkRExM5N5kj/Q8CdHeufBD5j+9XANmBJaV8CbCvtnyn9kHQwcBrwWmAh8DlJ055f+RERMRldhb6kOcAJwBfLuoCjgMtKl5XASWV5UVmnbD+69F8EXGL7Mds/BEaAw3uxExER0Z1uj/T/N/AHwM/L+suAn9h+sqxvBGaX5dnA/QBl+/bS/xftYzznFyQtlbRO0rotW7ZMYlciImIiE4a+pLcCm22v70M92F5ue9j28NDQUD/eMiKiGtO76PNG4ERJxwN7AS8FPgvMkDS9HM3PATaV/puAucBGSdOBfYAfd7SP6nxORET0wYRH+rY/YnuO7Xk0J2K/Y/tdwHeBk0u3xcDlZXl1Wads/45tl/bTyuyeA4H5wI0925OIiJhQN0f64zkLuETSx4HvAxeW9guBr0gaAbbSfFBg+3ZJq4A7gCeBM20/9TzePyIiJmlSoW/7GuCasnwPY8y+sf0ocMo4z/8E8InJFhkREb2RK3IjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqMiEoS9pL0k3Svp/km6X9Mel/UBJN0gakXSppD1K+55lfaRsn9fxWh8p7XdJOnaqdioiIsbWzZH+Y8BRtt8AHAIslHQk8EngM7ZfDWwDlpT+S4Btpf0zpR+SDgZOA14LLAQ+J2laL3cmIiJ2bsLQd+Nfy+oLy4+Bo4DLSvtK4KSyvKisU7YfLUml/RLbj9n+ITACHN6TvYiIiK50NaYvaZqkW4DNwBrgbuAntp8sXTYCs8vybOB+gLJ9O/CyzvYxntP5XkslrZO0bsuWLZPfo4iIGFdXoW/7KduHAHNojs5fM1UF2V5ue9j28NDQ0FS9TURElSY1e8f2T4DvAr8GzJA0vWyaA2wqy5uAuQBl+z7Ajzvbx3hORET0QTezd4YkzSjLewPHAHfShP/Jpdti4PKyvLqsU7Z/x7ZL+2llds+BwHzgxl7tSERETGz6xF3YH1hZZtq8AFhl+wpJdwCXSPo48H3gwtL/QuArkkaArTQzdrB9u6RVwB3Ak8CZtp/q7e5ERMTOTBj6tm8FDh2j/R7GmH1j+1HglHFe6xPAJyZfZkRE9EKuyI2IqEhCPyKiIt2M6Uf01bxlV/b1/e4974S+vl9Em3KkHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFchuGiD7LbSaiTTnSj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiCf2IiIpMGPqS5kr6rqQ7JN0u6UOlfV9JayRtKI8zS7sknS9pRNKtkg7reK3Fpf8GSYunbrciImIs3RzpPwn8vu2DgSOBMyUdDCwD1tqeD6wt6wDHAfPLz1LgAmg+JICzgSOAw4GzRz8oIiKiPyYMfdsP2L65LP8UuBOYDSwCVpZuK4GTyvIi4CI3rgdmSNofOBZYY3ur7W3AGmBhT/cmIiJ2alJj+pLmAYcCNwCzbD9QNj0IzCrLs4H7O562sbSN177jeyyVtE7Sui1btkymvIiImEDXoS/pl4C/BD5s++HObbYNuBcF2V5ue9j28NDQUC9eMiIiiq5CX9ILaQL/q7b/qjQ/VIZtKI+bS/smYG7H0+eUtvHaIyKiT7qZvSPgQuBO25/u2LQaGJ2Bsxi4vKP99DKL50hgexkGuhpYIGlmOYG7oLRFRESfdHNr5TcC7wFuk3RLafsocB6wStIS4D7g1LLtKuB4YAR4BDgDwPZWSecCN5V+59je2pO9iIiIrkwY+ra/B2iczUeP0d/AmeO81gpgxWQKjIiI3skVuRERFUnoR0RUJKEfEVGRhH5EREUS+hERFelmymZERFfmLbuyr+9373kn9PX9BkGO9CMiKpLQj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiCf2IiIpMGPqSVkjaLOmfOtr2lbRG0obyOLO0S9L5kkYk3SrpsI7nLC79N0haPDW7ExERO9PNkf6XgYU7tC0D1tqeD6wt6wDHAfPLz1LgAmg+JICzgSOAw4GzRz8oIiKifyYMfdt/D2zdoXkRsLIsrwRO6mi/yI3rgRmS9geOBdbY3mp7G7CGZ3+QRETEFHuuY/qzbD9Qlh8EZpXl2cD9Hf02lrbx2p9F0lJJ6ySt27Jly3MsLyIixvK8T+TaNuAe1DL6esttD9seHhoa6tXLRkQEzz30HyrDNpTHzaV9EzC3o9+c0jZee0RE9NFzDf3VwOgMnMXA5R3tp5dZPEcC28sw0NXAAkkzywncBaUtIiL6aPpEHSRdDLwZ2E/SRppZOOcBqyQtAe4DTi3drwKOB0aAR4AzAGxvlXQucFPpd47tHU8OR0TEFJsw9G2/c5xNR4/R18CZ47zOCmDFpKqLiIieyhW5EREVSehHRFQkoR8RUZEJx/QjIqIxb9mVfX2/e887oeevmSP9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIirS99CXtFDSXZJGJC3r9/tHRNSsr6EvaRrwF8BxwMHAOyUd3M8aIiJq1u8j/cOBEdv32H4cuARY1OcaIiKqJdv9ezPpZGCh7feW9fcAR9h+f0efpcDSsnoQcFffCoT9gB/18f36Lfu3exvk/RvkfYP+798rbA+NtWF6H4voiu3lwPI23lvSOtvDbbx3P2T/dm+DvH+DvG+wa+1fv4d3NgFzO9bnlLaIiOiDfof+TcB8SQdK2gM4DVjd5xoiIqrV1+Ed209Kej9wNTANWGH79n7WMIFWhpX6KPu3exvk/RvkfYNdaP/6eiI3IiLalStyIyIqktCPiKhIQj8ioiJVh76kt0mq+r/B7kqNuRP3jIhOtQfebwEbJP2ppNe0XcxUkzRT0uvbrqMX3MxAuKrtOqaKpGmSftB2HVNN0isk/UZZ3lvSS9quqVckzZJ0oaRvlvWDJS1pu66qQ9/2u4FDgbuBL0v6R0lLB+wX7xpJL5W0L3Az8AVJn267rh65WdKvtl3EVLD9FHCXpAParmWqSPod4DLg86VpDvDX7VXUc1+mmZ7+8rL+z8CHW6umqDr0AWw/TPOLdwmwP/B2mjD5QKuF9c4+ZR9/E7jI9hHAb7RcU68cAfyjpLsl3SrpNkm3tl1UD80Ebpe0VtLq0Z+2i+qhM4E3Ag8D2N4A/HKrFfXWfrZXAT+H5jol4Kl2S9oF773TT5JOBM4AXg1cBBxue7OkFwF3AH/eZn09Ml3S/sCpwP9ou5geO7btAqbYx9ouYIo9ZvtxSQBImg4M0oVDP5P0Mso+SToS2N5uSZWHPvAO4DO2/76z0fYju8LYW4+cQ/Mn5vds3yTplcCGlmvqCdv3SXoTMN/2lyQNAb/Udl29Yvvv2q5hiv2dpI8Ce0s6BvivwN+0XFMv/R7NbWZeJelaYAg4ud2SckUukmYBo+PCN9re3GY90T1JZwPDwEG2/52klwNft/3GlkvriXJk+OfAvwf2oLl1yc9sv7TVwnqkzJxbAiwARHNw8kUPUCiVv14Ootm/u2w/0XJJdY/pSzoFuBE4hWb444Zyz/+BUWYmvVTSC8vY8BZJ7267rh55O3Ai8DMA2/8CDMxJeOD/AO+k+ctsb+C9NN88NyhOojnPdIrtk21/YcAC/xRg73J/sZOASyUd1nJZdYc+8IfAr9pebPt0mm/2GrRx1AXlRO5bgXtpzl/891Yr6p3HS0iMjpm+uOV6es72CDDN9lO2vwQsbLumHnob8M+SviLpreWoeJB8zPZPyxDk0cCFwAUt11R96L9gh+GcHzN4/01G/yGdQDP00fqJpB5aJenzwIwy/e9vgS+0XFMvPVJuQX5L+Yvtdxmg30/bo5Movk7zF83dkr7YblU9NTpT5wTgC7avpBmma9WgfbJO1rckXQ1cXNZPA77ZYj1T4Ypykc+/Ae8rJzsfbbmmnrD9Z+UE4MM046Z/ZHtNy2X10ntoQv79wO/SfAHRO1qtqMdsP1EuXjLNENZJNMNYg2BTOSg5BvikpD3ZBT60cyJX+k2aucIA/2B7kC4OAaBcmLXd9lNlCOQlth9su66YmKS9gQNs9/O7ovtC0nE0V8W/GbgGWAV8u8xn3+2Vqd8LgdtsbyhTp19n+9ut1lVj6Ev6nu03SfopzRGGOjb/HNgKfMr251opsIfKL97v0QTHUknzaWa7XNFyac9bx/+/TtuBdcDv276n/1X1jqS3AX8G7GH7QEmHAOfYPrHl0npC0sXApcA3bT/Wdj29Iumlth8uB1vPYntrv2vqVGXoT6RcUHGd7YParuX5knQpsB443fZ/KB8C19k+pOXSnjdJ5wIbga/RfHCfBryK5nYT77P95vaqe/4krQeOAq6xfWhpu83269qtrHcGccq0pCtsv1XSD3n2QaVtv7Kl0oBdYHxpV2T7xzR/cg6CV9n+U+AJaC4845m/hLuzE21/3vZPbT9sezlwrO1LaW5hsLt7YowT7wNzlDaoU6ZL4Av4T7ZfafvAjp9WAx9yIndcth9ou4YeebyMC49Oa3wVMCh/Sj8i6VSaeydBc7Xj6EnqQQjH2yX9Z2BaGZb7IHBdyzX10uiU6c0AZZLB3/L0/8/dlm1LuhLY5f4qy5H+4Dsb+BYwV9JXgbXAH7RbUs+8i2aGy2bgobL87vIh9/42C3s+JH2lLN4NvJbmQ/pimllKrd+lsYcGfcr0LnkX2IzpV6CcoziSZljnets/armk2AlJd9DcCfWbwFt23N72icBekfQp4PU8PWX6t4BbbZ/VXlW9U6ZKvxq4j+aqcdH8EdDqd1ok9CsgaTbwCjqG83a8ydzuqAwH/A4wj2fu22+3VVMvSPog8D7glcCmzk3sAicCe0nSO3jmlOlvtFlPL0l6xVjttu/rdy2dEvoDTtInaY6gbqfc15smOHb7aX+SrgP+gWZ20i/uU277L1srqockXWD7fW3XEc9dudfOm2jOMV1r++aWS0roDzpJdwGvH6R50KMk3TIIU09rM871FfD0XzKDchfRP6KZmfRXpekkmluhfLy9qhL6A69c4n6K7X9tu5Zek/RxmmsOBva7cmP3VQ643mD70bK+N3BL29f/ZMrm4HuE5oZda+mYqmn7g+2V1DMfAj4q6TGa6xAG6kgxdnv/AuzF09OI9+SZ52hakdAffKvLz8Cx/ZJyqft8mn9cEbuS7TTXWqyhGc46BrhR0vnQ3oFXhndityXpvTRH+3OAW2impV5n++hWC4sAJC3e2XbbK/tVS6cc6Q8oSbexk6tS254r3CMforlvy/W23yLpNcCftFxTBJKm0XyB0bvarmVHCf3B9dbyeGZ5HL3K890Mxi0KAB61/agkJO1p+weSdvub5MXur9zG/BWS9rD9eNv1dEroD6jRC0AkHTN6h8biLEk3A8vaqaynNkqaAfw1sEbSNpqrHyN2BfcA10paTfkeZwDbn26vpIR+DSTpjbavLSu/zoDc38T228vi/5T0XWAfmvsMRewK7i4/LwBe0nItv5ATuQNO0q8AK2gCUcA24Ld3hSsDI6L/EvqVkLQPwIB9MXrELqv89fmsgLV9VAvl/EKGdyog6QSaW/Tu1Xy3A9g+p9WiIgbff+tY3ovmS+1b//7fhP6Ak/R/gRfR3KL3izRfNHJjq0VFVMD2+h2arpXU+r+9hP7g+3Xbr5d0q+0/lvS/aO7THhFTaIcvRn8BMExzbq1VCf3BN3rfj0ckvRzYCuzfYj0RtVjP01+M/gRwL7CkzYJgQKbuxU79TZnL/ingZuCHwNfaLSmiCmcBh9g+kObiyJ/R3ACxVQn9wfcD4KnyxSJ/AVxPczFTREytP7T9sKQ3AUfRnFO7oOWaEvoV+Jjtn+5qv3gRFRj9NrcTgC/YvhLYo8V6gIR+DXbJX7yICmyS9Hmaryu9StKe7AKZm4uzBpykK2i+uOEY4DDg34Abbb+h1cIiBpykFwELgdtsb5C0P/A6299uta6E/mDbVX/xIqIdCf2IiIq0Pr4UERH9k9CPiKhIQj8ioiIJ/YiIivx/xQTNrHTszP8AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "data = load_from_pickle(directory=\"merged_training.pkl\")\n", "\n", "## using a sample\n", "data= data[data[\"emotions\"].isin(emotions)]\n", "\n", "\n", "data = data.sample(n=20000);\n", "\n", "data.emotions.value_counts().plot.bar()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Comaf36-Hb6X", "outputId": "bd2ef6e3-5a10-4443-d5ea-2add82ba033f" }, "outputs": [ { "data": { "text/plain": [ "text 20000\n", "emotions 20000\n", "dtype: int64" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.count()" ] }, { "cell_type": "markdown", "metadata": { "id": "jYxc8fx_H3ad" }, "source": [ "Data has been preprocessed already, using technique from this paper: https://www.aclweb.org/anthology/D18-1404/" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 206 }, "id": "gYKK7ujRHfRt", "outputId": "04993b5a-7487-4175-8638-8860020e194d" }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textemotions
35968i feel fantastic and i m still alive pagetitle...joy
30417i were asked recently about making a lightweig...love
49194i was small i always feel jealous of my brothe...anger
5172i am feeling hopeless and this is my therapysadness
77433i know how you feel i was physically abused as...sadness
\n", "
\n", " \n", " \n", " \n", "\n", " \n", "
\n", "
\n", " " ], "text/plain": [ " text emotions\n", "35968 i feel fantastic and i m still alive pagetitle... joy\n", "30417 i were asked recently about making a lightweig... love\n", "49194 i was small i always feel jealous of my brothe... anger\n", "5172 i am feeling hopeless and this is my therapy sadness\n", "77433 i know how you feel i was physically abused as... sadness" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "JXovcl56NFPp" }, "outputs": [], "source": [ "## reset index\n", "data.reset_index(drop=True, inplace=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "pSzoz9InH0Ta", "outputId": "1dec6c28-d407-408d-dc82-aa9c4370e05c" }, "outputs": [ { "data": { "text/plain": [ "array(['joy', 'love', 'anger', 'sadness', 'fear', 'surprise'],\n", " dtype=object)" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## check unique emotions in the dataset\n", "data.emotions.unique()" ] }, { "cell_type": "markdown", "metadata": { "id": "rJm31gKShQus" }, "source": [ "## Split the data and store into individual text files\n", "\n", "If you are using your own dataset and want to split it for training, you can uncomment the code below. Otherwise, just skip it. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6ooNxSnPiztL" }, "outputs": [], "source": [ "## uncomment the code below to generate the text files for your train, val, and test datasets.\n", "\n", "'''\n", "from sklearn.model_selection import train_test_split\n", "import numpy as np\n", "\n", "# Creating training and validation sets using an 80-20 split\n", "input_train, input_val, target_train, target_val = train_test_split(data.text.to_numpy(), \n", " data.emotions.to_numpy(), \n", " test_size=0.2)\n", "\n", "# Split the validataion further to obtain a holdout dataset (for testing) -- split 50:50\n", "input_val, input_test, target_val, target_test = train_test_split(input_val, target_val, test_size=0.5)\n", "\n", "\n", "## create a dataframe for each dataset\n", "train_dataset = pd.DataFrame(data={\"text\": input_train, \"class\": target_train})\n", "val_dataset = pd.DataFrame(data={\"text\": input_val, \"class\": target_val})\n", "test_dataset = pd.DataFrame(data={\"text\": input_test, \"class\": target_test})\n", "final_dataset = {\"train\": train_dataset, \"val\": val_dataset , \"test\": test_dataset }\n", "\n", "train_dataset.to_csv(train_path, sep=\";\",header=False, index=False)\n", "val_dataset.to_csv(test_path, sep=\";\",header=False, index=False)\n", "test_dataset.to_csv(val_path, sep=\";\",header=False, index=False)\n", "'''" ] }, { "cell_type": "markdown", "metadata": { "id": "rAD1J6c0dLp8" }, "source": [ "## Create the Dataset object" ] }, { "cell_type": "markdown", "metadata": { "id": "aOOI69vwIYcN" }, "source": [ "Create the Dataset object that will be used to load the different datasets." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Ktr6xeMuISin" }, "outputs": [], "source": [ "class EmoDataset(Dataset):\n", " def __init__(self, path):\n", " super().__init__()\n", " self.data_column = \"text\"\n", " self.class_column = \"class\"\n", " self.data = pd.read_csv(path, sep=\";\", header=None, names=[self.data_column, self.class_column],\n", " engine=\"python\")\n", "\n", " def __getitem__(self, idx):\n", " return self.data.loc[idx, self.data_column], label2int[self.data.loc[idx, self.class_column]]\n", "\n", " def __len__(self):\n", " return self.data.shape[0]" ] }, { "cell_type": "markdown", "metadata": { "id": "9EYQRq3qJH7n" }, "source": [ "Sanity check" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "uGWw4wGEJGhJ", "outputId": "e0983e49-770d-495f-a4d4-50d499ff4011" }, "outputs": [ { "data": { "text/plain": [ "('i now feel compromised and skeptical of the value of every unit of work i put in',\n", " 4)" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = EmoDataset(train_path)\n", "ds[19]" ] }, { "cell_type": "markdown", "metadata": { "id": "0h6tTn9hd6v8" }, "source": [ "## Training with PyTorchLightning\n", "\n", "[PyTorchLightning](https://www.pytorchlightning.ai/) is a library that abstracts the complexity of training neural networks with PyTorch. It is built on top of PyTorch and simplifies training.\n", "\n", "![](https://pytorch-lightning.readthedocs.io/en/latest/_images/pt_to_pl.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "RJHhNRcZK7sV" }, "outputs": [], "source": [ "## Methods required by PyTorchLightning\n", "\n", "class TrainingModule(pl.LightningModule):\n", " def __init__(self, hparams):\n", " super().__init__()\n", " self.model = EmoModel(AutoModelWithLMHead.from_pretrained(\"distilroberta-base\").base_model, len(emotions))\n", " self.loss = nn.CrossEntropyLoss() ## combines LogSoftmax() and NLLLoss()\n", " #self.hparams = hparams\n", " self.hparams.update(vars(hparams))\n", "\n", " def step(self, batch, step_name=\"train\"):\n", " X, y = batch\n", " loss = self.loss(self.forward(X), y)\n", " loss_key = f\"{step_name}_loss\"\n", " tensorboard_logs = {loss_key: loss}\n", "\n", " return { (\"loss\" if step_name == \"train\" else loss_key): loss, 'log': tensorboard_logs,\n", " \"progress_bar\": {loss_key: loss}}\n", "\n", " def forward(self, X, *args):\n", " return self.model(X, *args)\n", "\n", " def training_step(self, batch, batch_idx):\n", " return self.step(batch, \"train\")\n", " \n", " def validation_step(self, batch, batch_idx):\n", " return self.step(batch, \"val\")\n", "\n", " def validation_end(self, outputs: List[dict]):\n", " loss = torch.stack([x[\"val_loss\"] for x in outputs]).mean()\n", " return {\"val_loss\": loss}\n", " \n", " def test_step(self, batch, batch_idx):\n", " return self.step(batch, \"test\")\n", " \n", " def train_dataloader(self):\n", " return self.create_data_loader(self.hparams.train_path, shuffle=True)\n", "\n", " def val_dataloader(self):\n", " return self.create_data_loader(self.hparams.val_path)\n", "\n", " def test_dataloader(self):\n", " return self.create_data_loader(self.hparams.test_path)\n", " \n", " def create_data_loader(self, ds_path: str, shuffle=False):\n", " return DataLoader(\n", " EmoDataset(ds_path),\n", " batch_size=self.hparams.batch_size,\n", " shuffle=shuffle,\n", " collate_fn=TokenizersCollateFn()\n", " )\n", " \n", " @lru_cache()\n", " def total_steps(self):\n", " return len(self.train_dataloader()) // self.hparams.accumulate_grad_batches * self.hparams.epochs\n", "\n", " def configure_optimizers(self):\n", " ## use AdamW optimizer -- faster approach to training NNs\n", " ## read: https://www.fast.ai/2018/07/02/adam-weight-decay/\n", " optimizer = AdamW(self.model.parameters(), lr=self.hparams.lr)\n", " lr_scheduler = get_linear_schedule_with_warmup(\n", " optimizer,\n", " num_warmup_steps=self.hparams.warmup_steps,\n", " num_training_steps=self.total_steps(),\n", " )\n", " return [optimizer], [{\"scheduler\": lr_scheduler, \"interval\": \"step\"}]" ] }, { "cell_type": "markdown", "metadata": { "id": "OGc7Vw1moHxr" }, "source": [ "## Finding Learning rate for the model\n", "\n", "The code below aims to obtain valuable information about the optimal learning rate during a pretraining run. Determine boundary and increase the leanring rate linearly or exponentially.\n", "\n", "More: https://github.com/davidtvs/pytorch-lr-finder" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 474, "referenced_widgets": [ "69c22be4a46740149cf5ea6823a5eed9", "7c615a5ac40b43ad80f49e5b520c7cd5", "b4038fd3724d429fb0026753c504c40d", "c8b4362c1387438d8bff0a98838c62f6", "3c8b0bd72bb34489ba4f7336b04127be", "4599afc67ec241fb879ac419d92cb01d", "abb8e33775974bdb8718d09b1ba6875b", "66f1cd46a86e44558cf2657469a63c62", "e0d5d58dd5584898a7fe092a9fb371fb", "fd994e14a67a4adf80a480b6ee20afa8", "2e849223bca94e42a426180c54ecd8fb" ] }, "id": "xL4lNPDFoFyU", "outputId": "aa34657a-62da-4718-8495-768b30c9849e" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/modeling_auto.py:882: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n", " FutureWarning,\n", "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:309: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n", " FutureWarning,\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "69c22be4a46740149cf5ea6823a5eed9", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/100 [00:00" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "lr=0.1 ## uper bound LR\n", "from torch_lr_finder import LRFinder\n", "hparams_tmp = Namespace(\n", " train_path=train_path,\n", " val_path=val_path,\n", " test_path=test_path,\n", " batch_size=16,\n", " warmup_steps=100,\n", " epochs=1,\n", " lr=lr,\n", " accumulate_grad_batches=1,\n", ")\n", "module = TrainingModule(hparams_tmp)\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = AdamW(module.parameters(), lr=5e-7) ## lower bound LR\n", "lr_finder = LRFinder(module, optimizer, criterion, device=\"cuda\")\n", "lr_finder.range_test(module.train_dataloader(), end_lr=100, num_iter=100, accumulation_steps=hparams_tmp.accumulate_grad_batches)\n", "lr_finder.plot()\n", "lr_finder.reset()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "YdqP56M1oXav", "outputId": "f6c0b138-6248-4543-b987-6d7430c015f1" }, "outputs": [ { "data": { "text/plain": [ "0.0001" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr = 1e-4 \n", "lr" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 352 }, "id": "vMab6vu0Bow0", "outputId": "d3da4b83-d2ea-4f28-988a-61e796ae210b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LR suggestion: steepest gradient\n", "Suggested LR: 3.65E-02\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU5dn48e892TdCIGENS0D2LUhkERBwAdSqVetCrXUp4q5Uy6u29lW7vPVXW7cuLm0prqh1RUFFq4gKCmERCItgQAyBJGzZJ9vcvz9mEgNMQhIyOZPk/lzXXJM55znn3HPE3HnOs4mqYowxxhzN5XQAxhhjgpMlCGOMMX5ZgjDGGOOXJQhjjDF+WYIwxhjjlyUIY4wxfoU6HUBzSkxM1L59+zodhgm0bdu874MGORuHMW3AmjVr9qtqkr99bSpB9O3bl/T0dKfDMIE2dar3fdkyJ6Mwpk0QkW/r2mePmIwxxvhlCcIYY4xfAUsQItJLRD4Wkc0ikiEit/spIyLyuIjsEJENInJyrX1VIrLe91oUqDiNMcb4F8g2iErgTlVdKyJxwBoR+UBVN9cqczYwwPcaBzzhewcoVdXUEw2ioqKCrKws3G73iZ7KBIv77vO+b9lyxObIyEiSk5MJCwtzIChj2p6AJQhV3Qvs9f1cKCJbgJ5A7QRxAfCsemcM/EJEOopId9+xzSIrK4u4uDj69u2LiDTXaY2TXL6Kb61eTKrKgQMHyMrKIiUlxaHAjGlbWqQNQkT6AqOBL4/a1RP4rtbnLN82gEgRSReRL0Tkh/Wce46vXHpeXt4x+91uN507d7bk0MaJCJ07d7aaojHNKOAJQkRigdeAuapa0IhD+6hqGvBj4FER6e+vkKo+rappqpqWlOS3K68lh3bC/jubtmDbvkLcFVVOhwEEOEGISBje5PCCqr7up8geoFetz8m+bahq9XsmsAxvDSTwVOGLL+CNN7zvAVov49FHH6WkpCQg526ow4cP8/e//73Frte3b1/2798PwKmnntrk8yx4/XWyc3KaKyxjgoa7oorz/voZd7yy3ulQgMD2YhLgX8AWVX24jmKLgJ/6ejONB/JVda+IJIhIhO88icBEjmy7CIwlS6B3bzjrLLj6au97797e7c2srSSIysrKJh23YsWKJl9zwRtvkJ2b2+TjjQlWeYVllFd6WLJxH+9ubLam2CYLZA1iInAlcHqt7qrniMgNInKDr8wSIBPYAfwDuMm3fQiQLiJfAR8DDx7V+6n5LVkCP/oRZGVBUREUFHjfs7K825uYJIqLizn33HMZNWoUw4cP5+WXX+bxxx8nOzubadOmMW3aNACWLl3KhAkTOPnkk7nkkksoKioCYM2aNUyZMoUxY8YwY8YM9u71/qOZOnUqt99+O6mpqQwfPpxVq1bVXO/aa69l7NixjB49mrfeeguAjIwMxo4dS2pqKiNHjmT79u3cfffdfPPNN6SmpjJv3rxjYv/tb3/LoEGDmDRpErNmzeJPf/pTzbXnzp1LWloajz32GG+//Tbjxo1j9OjRnHnmmeT4/ro/cOAA06dPZ9iwYcyePZvaqxfGxsbW/PzQQw9xyimnMHLkSO7z9VDatWsXQ4YM4brrrmPYsGFMnz6d0tJSXn31VdIzMrhi3jxSU1MpLS1t0n8XY4JRToG3DS0mPIRfv5XBoeJyZwNS1TbzGjNmjB5t8+bNx2w7hsej2rOnqveBkv9XcrK3XCO9+uqrOnv27JrPhw8fVlXVPn36aF5enqqq5uXl6eTJk7WoqEhVVR988EF94IEHtLy8XCdMmKC5ubmqqvrSSy/pNddco6qqU6ZMqTnvJ598osOGDVNV1XvuuUefe+45VVU9dOiQDhgwQIuKivSWW27R559/XlVVy8rKtKSkRHfu3Flz3NFWrVqlo0aN0tLSUi0oKNCTTjpJH3rooZpr33jjjTVlDx48qB7fvfnHP/6hd9xxh6qq3nrrrfrAAw+oquo777yjQM13jomJUVXV999/X6+77jr1eDxaVVWl5557rn7yySe6c+dODQkJ0XXr1qmq6iWXXFLzvaaccoqu/s9//MbdoP/exgSpxRuytc9d7+hra77T/vcs1p+/tC7g1wTStY7fqW1qLqYm+/JLyM+vv8zhw7BqFYwbV3+5o4wYMYI777yTu+66ix/84AdMnjz5mDJffPEFmzdvZuLEiQCUl5czYcIEtm3bxqZNmzjrrLMAqKqqonv37jXHzZo1C4DTTjuNgoICDh8+zNKlS1m0aFHNX/tut5vdu3czYcIEfv/735OVlcVFF13EgAED6o37888/54ILLiAyMpLIyEjOO++8I/ZfdtllNT9nZWVx2WWXsXfvXsrLy2u6mS5fvpzXX/c2PZ177rkkJCQcc52lS5eydOlSRo/2NjEVFRWxfft2evfuTUpKCqmp3qEwY8aMYdeuXfXGbExrV12DmDqoCzdNO4nH/7ud80b1YNrgLo7EYwkCYO/e7/vW18XlguzsRp964MCBrF27liVLlnDvvfdyxhln8L//+79HlFFVzjrrLBYuXHjE9o0bNzJs2DBWrlzp99xH99oREVSV1157jUFHzXQ6ZMgQxo0bx+LFiznnnHN46qmn6NevX6O/T7WYmJian2+99VbuuOMOzj//fJYtW8b999/f4POoKvfccw/XX3/9Edt37dpFREREzeeQkBB7nGTavJyCMsJChIToMG6ZdhLvbdrLL9/YyPtzJ9Phq7Xe31Xdu3v/UG2BXns2FxN4b7jHU38Zjwd69Gj0qbOzs4mOjuYnP/kJ8+bNY+3atQDExcVRWFgIwPjx4/n888/ZsWMH4G1H+Prrrxk0aBB5eXk1CaKiooKMjIyac7/88ssAfPbZZ8THxxMfH8+MGTP4y1/+UvO8f926dQBkZmbSr18/brvtNi644AI2bNhwRAxHmzhxIm+//TZut5uioiLeeeedOr9jfn4+PXt6h68888wzNdtPO+00XnzxRQDeffddDh06dMyxM2bMYP78+TVtLnv27CH3OA3QcTExFBYX11vGmNYot8BNl7hIRITwUBd//NEohqz7FE+vluk8czSrQYA3G8fHexul69KxI4wd2+hTb9y4kXnz5uFyuQgLC+OJJ54AYM6cOcycOZMePXrw8ccfs2DBAmbNmkVZWRkAv/vd7xg4cCCvvvoqt912G/n5+VRWVjJ37lyGDRsGeKeWGD16NBUVFcyfPx+AX//618ydO5eRI0fi8XhISUnhnXfe4ZVXXuG5554jLCyMbt268ctf/pJOnToxceJEhg8fztlnn81DDz1UE/cpp5zC+eefz8iRI+natSsjRowgPj7e73e8//77ueSSS0hISOD0009n586dANx3333MmjWLYcOGceqpp9K7d+9jjp0+fTpbtmxhwoQJgLfx+vnnnyckJKTOe3r1hRdyw/33E/XII6xcuZKoqKjG/mcxJijlFpbRpcP3NefUjSt46q3/R1j5UQNAi4q8nWdefRXOOSdwAdXVONEaX01upFZVXbxYNSrKfwN1VJR3fxCZMmWKrl69OqDXKCwsVFXV4uJiHTNmjK5Zsyag12uwrVu9Lz+skdq0Zmf+eZle/2y690MAO8/URj2N1PaIqdo553izcXIyxMZChw7e9+TkwGfpIDVnzhxSU1M5+eSTufjiizn55JOPf5AxpslyCtzf1yAa03kmQOwRU23nnAO7d3tveHa2t81h7NgWaQxqrGUtsJpadfuBMSbw3BVVFLgr6doh0rshgJ1nGsoSxNFEGt2V1RhjTlRugbf9sUucrwYRwM4zDdUuHjGpBmY+JRNc7L+zac1yCr0N0TU1iOrOM/VpYueZhmrzCSIyMpIDBw7YL482TtW7HkRkZKTToRjTJNWD5GraIETg6aehjl56ZeEReJ54MqCPwNv8I6bk5GSysrLwt1aEaaX27fO+H1X9rl5RzpjWKMf3iKlrXK0/cqo7z1x/vbdB2uUCj4eiqFhumXI9Y2MH1UxgFwhtPkGEhYXZCmNtzY03et9boKHemJaSW+gmPMRFx+ijlsz103km5pRTiH1pPX96fxsje3Zk0oDEgMTU5hOEMca0BrkF3kFyfhe+OqrzjAD/7+KRbNtXyG0vreOdWyfRo2PzDxht820QxhjTGuQUuL/vwdQAMRGhPHnlGMorPdz4wlrKKpt/FTpLEMYYEwRyCtzf92BqoP5JsfzpklGM7tURofkbq+0RkzHGBIHcwjImD0hq9HEzh3dj5vBuAYjIahDGGOO4kvJKCt2VR0zUFwwCuSZ1LxH5WEQ2i0iGiNzup4yIyOMiskNENojIybX2XSUi232vqwIVpzHGOO37UdTBNY4nkI+YKoE7VXWtiMQBa0TkAz1ybemzgQG+1zjgCWCciHQC7gPSAPUdu0hVj11QwBhjWrncQt8YiPZSg1DVvaq61vdzIbAF6HlUsQuAZ32zzn4BdBSR7sAM4ANVPehLCh8AMwMVqzHGOKl6FHVjG6kDrUXaIESkLzAa+PKoXT2B72p9zvJtq2u7Mca0OTXTbDSim2tLCHiCEJFY4DVgrqoWBOD8c0QkXUTSbToNY0xrlFtYRnioi/iosOMXbkEBTRAiEoY3Obygqq/7KbIH6FXrc7JvW13bj6GqT6tqmqqmJSU1vouYMcY4LbfATde6RlE7KJC9mAT4F7BFVR+uo9gi4Ke+3kzjgXxV3Qu8D0wXkQQRSQCm+7YZY0ybk1NQduQkfUEikL2YJgJXAhtFZL1v2y+B3gCq+iSwBDgH2AGUANf49h0Ukd8Cq33H/UZVDwYwVmOMcUxOoZvB3eKcDuMYAUsQqvoZ1D/227dg9s117JsPzA9AaMYYE1RyC8o4rQmjqAPNRlIbY4yDissqKSqrDLourmAJwhhjHBWsg+TAEoQxxjjq+zEQVoMwxhhTy/ejqK0GYYwxppY83yOmLtYGYYwxpracAjeRYS46RAbf8jyWIIwxxkE5BWV0iYsMulHUYAnCGGMcleObZiMYWYIwxhgH5RWWBWX7A1iCMMYYR+UUuINyHiawBGGMMY4pKqukuLwq6NairmYJwhhjHBLMYyDAEoQxxjgmt8A3zYY9YjLGGFNbbqFvmg1rpDbGGFNbzTxM9ojJGGNMbTkFZUSFhRAXEXyjqMEShDHGOCa3sCwo16KuZgnCGGMcklPgDtr2BwhgghCR+SKSKyKb6tifICJviMgGEVklIsNr7dslIhtFZL2IpAcqRmOMcVJugZsuccHZ/gCBrUEsAGbWs/+XwHpVHQn8FHjsqP3TVDVVVdMCFJ8xxjhGVckpKAvKpUarBSxBqOpy4GA9RYYCH/nKbgX6ikjXQMVjjDHBpKisktKKqqAdJAfOtkF8BVwEICJjgT5Asm+fAktFZI2IzKnvJCIyR0TSRSQ9Ly8voAEbY0xzyakeJNceaxAN8CDQUUTWA7cC64Aq375JqnoycDZws4icVtdJVPVpVU1T1bSkpKSAB22MMc0h1zcGIimI2yAc63yrqgXANQDi7eO1E8j07dvje88VkTeAscByh0I1xphml1NYPQ+T1SCOISIdRSTc93E2sFxVC0QkRkTifGVigOmA355QxhjTWuW2gkdMAatBiMhCYCqQKCJZwH1AGICqPgkMAZ4REQUygJ/5Du0KvOEbOBIKvKiq7wUqTmOMcUJOQRkx4SHEBukoaghgglDVWcfZvxIY6Gd7JjAqUHEZY0wwyCkM7kFyYCOpjTHGEcE+SA4sQRhjzAn7vyVbuPDvn1NR5WnwMd55mKwGYYwxbVZ5pYeXVu1m3e7DzP9sZ4OO8Y6ithqEMca0aZ/tyKPAXUnPjlE8+uF2sg6VHPeYAncl7gqP1SCMMaYte2fDXuIiQ3lh9jgA7l+UgarWe0ywLxRUzRKEMcY0UVllFR9k5DBjWDf6Jsbw87MG8OGWXJZuzqnzGHdFFb9+cxMhLmFYjw4tGG3jWYIwxpgm+vTr/RSWVfKDkd0BuGZiCoO7xXH/ogyKyyqPKV9e6eGG59ewatdBHr50FCd1iWvpkBvFEoQxxjTR4o17iY8KY+JJiQCEhbj4/YUj2Jvv5pEPvj6ibGWVh7kvr2PZtjz+78IRXJDa04mQG8UShDHGNIG7oooPNucwc1g3wkK+/1U6pk8Cs8b25t8rdpGRnQ+Ax6Pc/fpGlmzcx73nDmHW2N5Ohd0oliCMMaYJPvk6j6KySs71PV6q7a6Zg+gYFcav3thElUf5zTubeXVNFnPPHMDsyf0ciLZpLEEYY0wTLN6wl4ToMCb073zMvo7R4fzq3CGs/+4wlz21kgUrdjF7Ugq3nzHAgUibzhKEMcY0kruiig+35DBz+JGPl2q7cHRPJvTrTPq3h5g1the/OncIvklIW43gnUbQGGOC1LJtuZSUV/GDkT3qLCMiPHZ5Kh9vy+VHY3q1uuQAliCMMabR3tmwl84x4YxL6VRvuS4dIrnslNbRIO2PPWIyxphGKCmv5L9bcpk5vBuhdTxeaiva9rczxphm9vHWPEorqvz2XmprLEEYY0wjLN6YTWJsOONSju291NYELEGIyHwRyRURv+tJi0iCiLwhIhtEZJWIDK+1b6aIbBORHSJyd6BiNMaYxiguq+SjrbmcPbw7Ia7W1+jcWIGsQSwAZtaz/5fAelUdCfwUeAxAREKAvwFnA0OBWSIyNIBxGmNMg3y0NRd3hadm7qW2LmAJQlWXAwfrKTIU+MhXdivQV0S6AmOBHaqaqarlwEvABYGK0xhjGmrxhr10iYsgrW/9vZfaCifbIL4CLgIQkbFAHyAZ6Al8V6tclm+bMcY46oudB5g2qEu7eLwEziaIB4GOIrIeuBVYB1Q19iQiMkdE0kUkPS8vr7ljNMYYAA4Wl3O4pIIBXWOdDqXFODZQTlULgGsAxDvEcCeQCUQBvWoVTQb21HOep4GnAdLS0upfxskYY5ooM68IgH5JMQ5H0nIcq0GISEcRCfd9nA0s9yWN1cAAEUnx7b8cWORUnMYYA5C5vxiAlESrQZwwEVkITAUSRSQLuA8IA1DVJ4EhwDMiokAG8DPfvkoRuQV4HwgB5qtqRqDiNMaYhsjMKyYsROiVEOV0KC0mYAlCVWcdZ/9KYGAd+5YASwIRlzHGNEVmXhG9O0W3+ek1ams/39QYY05A5v5i+iW1n8dLYAnCGGOOq7LKw7cHittVAzVYgjDGmOPKOlRKRZXSvx01UIMlCGOMOa7M/e2viytYgjDGmOPKzPN2cbU2CGOMMUf4Jq+YjtFhdIoJP37hNsQShDHGHMfO/UX0S2xfj5fAEoQxxhxXZl776+IKDUwQIhIjIi7fzwNF5HwRCQtsaMYY47xCdwW5hWXtroEaGl6DWA5EikhPYClwJd4FgYwxpk3b6ZuDqV876+IKDU8QoqoleNdv+LuqXgIMC1xYxhgTHL7vwWQ1iLqIiEwArgAW+7aFBCYkY4wJHpl5RbgE+nSOdjqUFtfQBDEXuAd4Q1UzRKQf8HHgwjLGmODwzf5ikhOiiQhtf38TN2g2V1X9BPgEwNdYvV9VbwtkYMYYEwy8PZja3+MlaHgvphdFpIOIxACbgM0iMi+woRljjLM8HvWNgWh/DdTQ8EdMQ32rvf0QeBdIwduTyRhj2qx9BW7cFR6rQRxHmG/cww+BRapaAdj6z8aYNq0992CChieIp4BdQAywXET6AAWBCsoYY4JB9Syu/dvhKGpoYIJQ1cdVtaeqnqNe3wLT6jtGROaLSK6IbKpjf7yIvC0iX4lIhohcU2tflYis970WNeobGWNMM8nMKyYmPIQucRFOh+KIhjZSx4vIwyKS7nv9GW9toj4LgJn17L8Z2Kyqo4CpwJ9FpHqqxFJVTfW9zm9IjMYY09y+ySuiX1IsIuJ0KI5o6COm+UAhcKnvVQD8u74DVHU5cLC+IkCceO98rK9sZQPjMcaYgGvPXVyhgeMggP6qenGtzw+IyPoTvPZfgUVANhAHXKaqHt++SBFJx5swHlTVN+s6iYjMAeYA9O7d+wRDMsYYL3dFFdn5pfRL7OV0KI5paA2iVEQmVX8QkYlA6QleewawHugBpAJ/FZEOvn19VDUN+DHwqIj0r+skqvq0qqapalpSUtIJhmSMMV479xej2n57MEHDaxA3AM+KSLzv8yHgqhO89jV4awcK7BCRncBgYJWq7gFQ1UwRWQaMBr45wesZY0yDVc/imtIOFwqq1tBeTF/5GpNHAiNVdTRw+gleezdwBoCIdAUGAZkikiAiEb7ticBEYPMJXssYYxolM8/bxdVqEA3kG01d7Q7g0brKishCvL2TEkUkC7gPCPOd50ngt8ACEdkICHCXqu4XkVOBp0TEgzeBPaiqliCMMS0qM6+Y7vGRRIc36tdkm3Ii37zefl+qOus4+7OB6X62rwBGnEBcxhhzwr7Z3757MMGJrUltU20YY9okVSUzr/1O0let3hqEiBTiPxEIEBWQiIwxxmH7i8opdFe2+xpEvQlCVeNaKhBjjAkW3zdQt+8axIk8YjLGmDYp09fFtV877uIKliCMMeYYmXlFRIS66NmxfT9JtwRhjDFH2bm/mJTEGFyu9jlJXzVLEMYYc5T2PklfNUsQxhhTS0WVh90HS9p9F1ewBGGMMUf4Jq+ISo9yUhdLEJYgjDGmls3Z3hmFhvXocJySbZ8lCGOMqSUju4DIMFe7HwMBliCMMeYIm7MLGNytAyHtvAcTWIIwxpgaqkpGdj5D7fESYAnCGGNqZB0qpcBdae0PPpYgjDHGZ/NebwP10O6WIMAShDHG1MjILsAlMLibJQiwBGGMMTU2Z+fTLymWqPAQp0MJCgFNECIyX0RyRWRTHfvjReRtEflKRDJE5Jpa+64Ske2+11WBjNMYY8Dbg8naH74X6BrEAmBmPftvBjar6ii861f/WUTCRaQT3jWsxwFjgftEJCHAsRpj2rFDxeVk57stQdQS0AShqsuBg/UVAeJERIBYX9lKYAbwgaoeVNVDwAfUn2iMMeaEZGRXN1DHOxxJ8HC6DeKvwBAgG9gI3K6qHqAn8F2tclm+bccQkTkiki4i6Xl5eYGO1xjTRm3emw9gYyBqcTpBzADWAz2AVOCvItKo/zqq+rSqpqlqWlJSUiBiNMa0AxnZBXSPj6RTTLjToQQNpxPENcDr6rUD2AkMBvYAvWqVS/ZtM8aYgLAG6mM5nSB2A2cAiEhXYBCQCbwPTBeRBF/j9HTfNmOMaXal5VV8k1fE0B7W/lBbaCBPLiIL8fZOShSRLLw9k8IAVPVJ4LfAAhHZCAhwl6ru9x37W2C171S/UdX6GruNMabJtu4rwKM2gvpoAU0QqjrrOPuz8dYO/O2bD8wPRFzGGFNb9RQb9ojpSE4/YjLGGMdlZBfQITKU5IQop0MJKpYgjDHtXkZ2AUN7dMA7JMtUswRhjGnXKqs8bN1bYAPk/LAEYYxp13buL6as0mPtD35YgjDGtGs1DdQ9LUEcLaC9mNqiKo+SX1rBoZJy8ksryC+p4HBpue+9gtLyKhJjI+gWH0n3+Ei6xUfStUMkYSGWi40JRhnZBYSHuuifFOt0KEHHEgSw6Ktsissqfa8qissrKSqrpKSskgJ3JQeLyzlcUs6hkgoK3BWo1n2u8BAX5VWeI7aJQOeYCMJCBI8qquBRAMWjEBYiRIWFEBkWQlR4CNHhIUSFhSAilFd6KKusorzSQ3mVh/JKD5VVSohLCHEJoSFCiMtFqO9zZFgIkaEuonzniPS9EqLDSIyNIDEugsTYcJJiI+gUE06oJS7TzmVk5zOoa5z9EeeHJQjg7tc2UFJeVfM5MsxFbEQo0eGhdIgKJSE6nF6dokmIDqNjdDgJ0WEkRIcTHx1GxyjvtvioMDpEhhLiEgrclezLd7M3v9T37ia30E2VRxEElwtEBMGbPCqrlJLyKkorqnBXVFFSXsWh4go8qkSEuggPdREdHkrHUBfhIS5CfYmmskqp8iiVHu97RZWH/NIKciu85yqtdc6KqmOzmgj0S4zh1P6JnNq/M+P7dSbB5qEx7Yiqsjm7gOlDuzkdSlCyBAEsvm0yUWEhREeEEBPu/SV/IuKjwoiPCmNQt7hmivDEqCpFZZXsLypnf1EZ+wvL2F9URl5hGRv25PPa2iye++JbRGBItw6c2r8zI5Lj6RAZRmxkKHGRocRGhBIXGUZsxInfH2OCxd58N4dKKqz9oQ6WIICUxBinQwgoESEuMoy4yDC/37WiysOGrMOs/OYAK745wLNffEt5pcfPmaBDZCjzZg7mirG9cVmiMK1c9RoQ1oPJP0sQhrAQF2P6dGJMn07ccvoA3BVVZB0qpdBdQVFZJYXuSgrdFRS6K/loay6/fnMTb63bwx8uGsGArsFRSzKmKTZnFyACg7tZgvDHEoQ5RmRYCCd18d+j42eTUnh1TRa/X7KFcx7/lJumnsRN0/oTEWqLvJvgpap+R0lnZOeT0jmGmAj7VeiP3RXTKCLCJWm9mDa4C795ezOP/Xc7izfu5bcXDCc5IYrDvm6/h0sqOFxSToG7ktG9OjKhf2ebxsC0uANFZdz12kY+3Z7H5AFJ/GBkd84Y0oW4yDDA+4gptXdHh6MMXpYgTJMkxkbw+KzRXHhyT+59YxOz/vFFveVTEmP48djeXDwm2VbsMi3i0+153PHKV+SXVvCDEd1ZmXmAD7fkEB7qYsrAJM4Y3IU9h0u5Ynxvp0MNWpYgzAmZNqgLS39+Gm9/lU2IS+gYHU7H6DASosOIjwonMszFh1tyeOGL3fx+yRYeen8b54zoxo/H9eGUvglWqzDNrrzSw5+WbuPp5ZkM6BLLs9eOZUj3Dng8yrrvDvHOhr28u3EfH2zOAWwNiPqI1jfqq5VJS0vT9PR0p8Mwddi2r5AXv/yW19ftodBdyYie8dw1czCTBiQ27kRTp3rfly1r7hBNK5eZV8RtL61j054CfjK+N786ZyhR4ce2j3k8ytrdh8jILuCKcb3b9YBREVmjqml+91mCMC2tpLySReuz+ctHO9hzuJRJJyVy18zBjEhu4GyaliCMH29/lc1dr20gPNTFHy8eyfRhNvitIepLEAFLmyIyX0RyRWRTHfvnich632uTiFSJSCffvuO8DY0AABPASURBVF0istG3z37jtzHR4aFcPrY3H/1iCr/+wVAysvM576+fccuLa9m1v9jp8EwrtHjDXm5/aR3De8Tz3u2nWXJoJoGsVy0AZta1U1UfUtVUVU0F7gE+OWrd6Wm+/X4zm2n9IkJD+NmkFD75n2ncevpJ/HdLLmc+/Am/eXszZZVVxz+BMcBHW3O4/aV1pPXpxDPXjqVbfKTTIbUZAUsQqrocOHjcgl6zgIWBisUEtw6RYdw5fRCfzJvKJWm9mP/5Tq74x5fsLypzOjQT5Fbs2M8Nz69laI8O/OvqNL/tDabpHG+ZEZFovDWN12ptVmCpiKwRkTnORGZaWpcOkfzhohH8ZdZoNmXnc/5fPiMjO9/psEyQWvPtIWY/m05K5xieuWZszdgG03wcTxDAecDnRz1emqSqJwNnAzeLyGl1HSwic0QkXUTS8/LyAh2raQHnjerBf64/FQUufmIFizfsdTokE2Q27cnn6n+voktcBM/NHmuzEAdIMCSIyznq8ZKq7vG95wJvAGPrOlhVn1bVNFVNS0pKCmigpuWMSI7nrVsmMrR7B25+cS0PL92Gx9N2etyZptuRW8hP568iLiKU52ePo0uctTkEiqMJQkTigSnAW7W2xYhIXPXPwHTAb08o07Z1iYtk4ZzxXDImmcc/2sENz6+xxut2zuNRfvZMOi4RXrhuPMkJ0U6H1KYFbCS1iCwEpgKJIpIF3AeEAajqk75iFwJLVbV238auwBu+EbahwIuq+l6g4jTBLSI0hD/+aCSDusXxu8VbuPeNTfwRsPHX7dP6rMN8e6CEhy8d1ean6Q8GAUsQqjqrAWUW4O0OW3tbJjAqMFGZ1khEmD25HwXuSh7/73buyHfT3boytkvvb9pHWIhwxpCuTofSLgRDG4QxDTL3jAGcPbwb3x4o5lBJhdPhmBamqry7aR+n9k8kPsp6LLUESxCm1XC5hD9fOoro8FB25BayPafQ6ZBMC9qyt5DdB0uYOdxGSbcUSxCmVYkOD2VQtzhcIsx+Np1DxeVOh2RayHsZ+3AJnDXUHi+1FEsQptWJCHUxsGscew+7ufGFNVRU+V8/27Qt72/axyl9O5EYG+F0KO2GrQdhWqW4yFAevHgEd7zyFf/7VgY/Gd+brEOlfHewhKxDpWQdKmHPYTdXju/Dj8fZgjCtXWZeEdtyCrnvvKFOh9KuWIIwrdZFJyfzdU4RT37yDQtX7a7ZHhMeQq9O0ZRXerh/UQaje3dkiC0K06q9l7EPgBk2S2uLsgRhWrV5MwYxoqd3HYlenaLolRBNx+gwRIQDRWXMePRTfv7yet68eSKRYTaRW2v1/qZ9jEqOp0fHKKdDaVesDcK0aiEu4dyR3Tl3ZHdGJnckISa8ZhnTzrERPPSjkWzdV8if3t/mcKSmqfYcLuWrrHxmDu/udCjtjiUI06ZNG9yFK8f34Z+f7eTzHfudDsc0wdKax0vWe6mlWYIwbd4vzxlCv6QY7nzlK/JtgF2r8+6mfQzqGke/pFinQ2l3LEGYNi8qPIRHL0tlf1EZ975l8z62JnmFZazeddAGxznEEoRpF0Ymd2TumQN4+6ts3lq/x+lwTAN9uCUHVSxBOMQShGk3bpx6Eml9Erj3zU3sOVzqdDimAd7btI8+naMZ3C3O6VDaJevmatqNEJfwyGWpzHx0Oaf98WOiwkKIDHMRERpCRPV7qIvIMJdvX/XLRUx4KClJMQzsGsfALnHER9tkcYGWX1rBim/2c+3ElJqeaaZlWYIw7UqvTtE8+7NxfLglh7IKD+7KKsoqPJRVVuGuea9if1E57ooq3L7the4K3BXfT+nRtUMEA7vGMbhbHLMn96NrB5t+vLl9tDWHiiq1x0sOsgRh2p0xfRIY0yehUcd4PMqew6Vszy3k65wivs4pZHtOEc+s+JZPt+/nPzdMIC6y/dQq1u4+RHJCVECX+3xv0z66dYhkVHLHgF3D1M8ShDEN4HIJvTpF06tTNKcP/r4//qfb87j636u5beE6/vHTNEJD2n6zXvbhUn70xAo6x0Yw/6pTGJEc3+zXKHBX8MnXeVyW1guXyx4vOaXt/2s2JoAmD0jigfOH8fG2PH63eIvT4bSIV9K/w6MQ6hIufWolH2zOafZr3PdWBhVVyiVpvZr93KbhApYgRGS+iOSKiN+O5yIyT0TW+16bRKRKRDr59s0UkW0iskNE7g5UjMY0h5+M78O1E1NYsGIXz63c5XQ4AVXlUV5e/R2TBySy6JZJDOway5zn0pn/2c5mu8Zb6/fwxro93Hb6AIb3bP7aiWm4QNYgFgAz69qpqg+paqqqpgL3AJ+o6kERCQH+BpwNDAVmiYjN8WuC2q/OHcIZg7tw/9ub+eTrPKfDCZhPvs5lb76bH4/tTVJcBC/NmcD0oV35zTubue+tTVSe4Noc3x0s4d43NjGmTwI3T+vfTFGbpgpYglDV5cDBBhafBSz0/TwW2KGqmapaDrwEXBCAEI1pNiEu4bFZoxnYNY5bXljL1210OdQXv/yOxNgIzvSt6hYVHsITV4xhzmn9eGblt8x5bg0Hi8spq6zC49FGnbvKo9zxynoUePSy1HbRnhPsHG+kFpFovDWNW3ybegLf1SqSBYyr5/g5wByA3r1tYRjjnNiIUP51VRo//NvnXLtgNW/ePLFNrX62L9/NR1tzuH5Kf8Jq/fJ2uYRfnjOE3p2iuW9RBif/9oOafSEuIdQlhIW4vPvPG8q4fp39nv+JZTtYvesQD186il6dogP+fczxOZ4ggPOAz1W1obWNI6jq08DTAGlpaY37k8WYZtajYxT/vCqNS59ayZxn03nxuvFtZh2K6sbpy0/x33D8k/F9GNK9A2u+PUhFlVJR5aHS915e5eGDzTlc9vQXzBrbi7vPHkJ81Pfdgtd/d5hHP9zOeaN6cOHoni31lcxxBEOCuJzvHy8B7AFq/wtM9m0zplUYmdyRRy5N5cYX1nL3axt45LLUVj8SuLpxetJJifTpHFNnufrGmMybMYhHPviaf322kw+35PLA+cM4e3g3SsqrmPvSOrp2iOR3Pxze6u9VW+LoQz4RiQemAG/V2rwaGCAiKSISjjeBLHIiPmOa6uwR3fnF9IG8uT6bvy/7xulwAMgvqaDA3bTpzpdvz2PP4VJmjW36Y9zo8FB+de5Q3rp5El3iIrjphbVc9+wa7n59I98eLOHPl446olZhnBewGoSILASmAokikgXcB4QBqOqTvmIXAktVtbj6OFWtFJFbgPeBEGC+qmYEKk5jAuXmaSexI7eIh97fRr/EGM4e0TIrouUVlrHm24Nk7i9mZ14xO/d7XweKywkPdXHxyclcNzmlUesrLPxyN51jwjlr6Ikv2jMiOZ63bp7Ivz7bySMffo27wsNNU/szvo62CeOcgCUIVZ3VgDIL8HaHPXr7EmBJ80dlTMsRER68eCS7D5bw81fWk5wQHZBRx7XtyC3ikidXcMi3MFKXuAhSEmOYPqwrKYkx7DpQwqtrsnhp9W7OGtKV66f0Y0yfTvWeM7fAzX+35jJ7cgrhoc3z0CE0xMX1U/pz9vDufLwt94RqJiZwgqENwpg2KzIshKeu9PZsmv3sahbdMilgE/vtOVzKlf/6khCXi1eun8DQHh2IjTj2f/E7zhrIsyt28ewX37J0cw5j+iRww5T+nDmki9/n//9Zk0WVR7n8lOb/Jd67czRXndq32c9rmod1NDYmwJLiIvjnVWkUuSuZ/Uw6peVVzX6NA0VlXPmvLykqq+TZa8cyNqWT3+QAkBgbwR3TB7Hi7tO5/7yh5BS4ue7ZdC55ciXrvzt8RFmPR1m4ajen9u9MSmLdjdOmbbIEYUwLGNK9A49dPppN2fnc8PwaDhSVNdu5C90VXP3v1WQfLmX+1acwtEeHBh0XHR7K1RNTWPaLqfzhohHsOlDMD//2OXNfWlezoNJnO/aTdejEGqdN62UJwpgWcubQrvz+hyNY8c1+pj+ynMUb9p7wOd0VVcx5dg1b9hbwxBVjOKVv/e0J/oSGuJg1tjfL5k3j5mn9WbJpH6f/aRkPvb+VBSt20SkmnOnDTrxx2rQ+liCMaUE/Htebd26dTM+EKG5+cS03vbCG/U2sTVRWebh14TpWZh7gz5eOYtrgLicUW2xEKPNmDOajO6cwc3g3/vbxN3y0NZcfjUkmIrRtDPYzjWMJwpgWNqhbHK/feCr/M3MQH27OZfojy3n7q2xUGz4RwOGScn7xn6/4YHMOD5w/jAtSm2/0cXJCNI9dPpo3b57IrLG9mT0ppdnObVoX68VkjANCQ1zcNPUkzhrSlV+8uoFbF67jzXV7uOyUXpw2MKnO6Tn2HC7ln59m8vLq7ygpr+LOswYGrBdQaq+OpPay1dzaM0sQxjhoQNc4XrthAv/8bCdPLPuG/27NJSY8hGmDu3DOiO5MHZREdHgoW/YW8PTyTBZ9lY0A56f24PrT+jOoW5zTX8G0YZYgjHFYaIiLG6b052eTUvgi8wBLNu5jacY+3tmwl8gwFwO7xrEhK5/o8BCuPrUvP5uUQo+OUU6HbdoBSxDGBImwEBeTByQxeUASv/vhcFbtPMi7m/aybvdhfjF9IFeO70t8tM1VZFqOJQhjglCIS5jQvzMT+tv8RMY51ovJGGOMX5YgjDHG+GUJwhhjjF+WIIwxxvhlCcIYY4xfliCMMcb4ZQnCGGOMX5YgjDHG+CWNmUEy2IlIHvBtA4rGA/knWK6uff62H72tvs+JwP4GxNZYDf3OTTnG7lP7u091xdYcx9h9atn7NEBV/S+Wrqrt7gU8faLl6trnb/vR2+r7DKQ7+Z3tPtl9cvpe2X1y9j7VfrXXR0xvN0O5uvb52370tuN9DoSmXMPuU/Me05buU1Ov05Bj7D45e59qtKlHTG2BiKSraprTcQQ7u08NY/epYew++ddeaxDB7GmnA2gl7D41jN2nhrH75IfVIIwxxvhlNQhjjDF+WYIwxhjjlyUIY4wxflmCaEVEZKqIfCoiT4rIVKfjCWYiEiMi6SLyA6djCVYiMsT3b+lVEbnR6XiClYj8UET+ISIvi8h0p+NpSZYgWoiIzBeRXBHZdNT2mSKyTUR2iMjdxzmNAkVAJJAVqFid1Ez3CeAu4JXAROm85rhPqrpFVW8ALgUmBjJepzTTfXpTVa8DbgAuC2S8wcZ6MbUQETkN7y/3Z1V1uG9bCPA1cBbeX/irgVlACPCHo05xLbBfVT0i0hV4WFWvaKn4W0oz3adRQGe8iXS/qr7TMtG3nOa4T6qaKyLnAzcCz6nqiy0Vf0tprvvkO+7PwAuquraFwndcqNMBtBequlxE+h61eSywQ1UzAUTkJeACVf0DUN+jkUNARCDidFpz3Cff47cYYChQKiJLVNUTyLhbWnP9e1LVRcAiEVkMtLkE0Uz/ngR4EHi3PSUHsAThtJ7Ad7U+ZwHj6iosIhcBM4COwF8DG1pQadR9UtVfAYjI1fhqXQGNLng09t/TVOAivH9sLAloZMGlUfcJuBU4E4gXkZNU9clABhdMLEG0Iqr6OvC603G0Fqq6wOkYgpmqLgOWORxG0FPVx4HHnY7DCdZI7aw9QK9an5N928yR7D41jN2nhrH71ECWIJy1GhggIikiEg5cDixyOKZgZPepYew+NYzdpwayBNFCRGQhsBIYJCJZIvIzVa0EbgHeB7YAr6hqhpNxOs3uU8PYfWoYu08nxrq5GmOM8ctqEMYYY/yyBGGMMcYvSxDGGGP8sgRhjDHGL0sQxhhj/LIEYYwxxi9LEKbNE5GiFr7eiha+XkcRuaklr2naB0sQxjSSiNQ7h5mqntrC1+wIWIIwzc4ShGmXRKS/iLwnImt8q/QN9m0/T0S+FJF1IvKhb+0NROR+EXlORD4HnvN9ni8iy0QkU0Ruq3XuIt/7VN/+V0Vkq4i84Js6GhE5x7dtjYg8LiLHrFkhIleLyCIR+Qj4r4jEish/RWStiGwUkQt8RR8E+ovIehF5yHfsPBFZLSIbROSBQN5L03bZbK6mvXoauEFVt4vIOODvwOnAZ8B4VVURmQ38D3Cn75ihwCRVLRWR+4HBwDQgDtgmIk+oasVR1xkNDAOygc+BiSKSDjwFnKaqO33TQdTlZGCkqh701SIuVNUCEUkEvhCRRcDdwHBVTQUQ77KYA/CueyB413s4TVWXN/lumXbJEoRpd0QkFjgV+I/vD3r4fgGmZOBlEekOhAM7ax26SFVLa31erKplQJmI5AJdOXYp2FWqmuW77nqgL94VzjJVtfrcC4E5dYT7gaoerA4d+D/fKmkevOsadPVzzHTfa53vcyzehGEJwjSKJQjTHrmAw9V/cR/lL3iXc13kW1Dn/lr7io8qW1br5yr8///UkDL1qX3NK4AkYIyqVojILrzLqh5NgD+o6lONvJYxR7A2CNPuqGoBsFNELgHvkpIiMsq3O57v1wa4KkAhbAP61VoK87IGHhcP5PqSwzSgj297Id7HXNXeB6711ZQQkZ4i0uWEozbtjtUgTHsQLSK1H/08jPev8SdE5F4gDHgJ+ApvjeE/InII+AhIae5gfG0YNwHviUgx3vUJGuIF4G0R2QikA1t95zsgIp+LyCa86ybPE5EhwErfI7Qi4CdAbnN/F9O22XTfxjhARGJVtcjXq+lvwHZVfcTpuIypzR4xGeOM63yN1hl4Hx1Ze4EJOlaDMMYY45fVIIwxxvhlCcIYY4xfliCMMcb4ZQnCGGOMX5YgjDHG+GUJwhhjjF//H5KGgimKy9CzAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/plain": [ "(,\n", " 0.036492170789302746)" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr_finder.plot(show_lr=lr)" ] }, { "cell_type": "markdown", "metadata": { "id": "ZhHutCseBxjJ" }, "source": [ "## Training the Emotion Classifier" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "q3FiLr3LBrjs", "outputId": "cda9d5ff-144a-4bcb-a404-b313ddf102de" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/modeling_auto.py:882: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n", " FutureWarning,\n" ] } ], "source": [ "hparams = Namespace(\n", " train_path=train_path,\n", " val_path=val_path,\n", " test_path=test_path,\n", " batch_size=32,\n", " warmup_steps=100,\n", " epochs=1,\n", " lr=lr,\n", " accumulate_grad_batches=1\n", ")\n", "module = TrainingModule(hparams)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "N8Jv_U25B37g" }, "outputs": [], "source": [ "## garbage collection\n", "import gc; gc.collect()\n", "torch.cuda.empty_cache()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 867, "referenced_widgets": [ "163afdc0b9264042b077d18e0eb52a6b", "1cb6a608fb424ebba37963363dd0bfbe", "8da4e895f98845e08267b25d32eeef50", "5193bb81089f4b0f8e4edd5b3f5217f3", "b0f51d2ab93c4eba9a43a5a05935600c", "8724d5cfe6a6463f83e9ad5f5954607b", "179400f1591c4b42980ad2a0b3fc785e", "2a7a08f6d7484ff1a3174d68ca167b8e", "096cd19c21354a7dac1c45daa1af460d", "a157658902f34eb3a7c63cbfcb3e53c7", "24b055e798604635abe3c346845cdacc", "7ab54ba7ccc9484a8c8b2605d35a6a9c", "55690e54a4ce492bbd234fd380be63ee", "85007e2fd3204764b63fbf2277c1005d", "84201a6f02fb405ea11b7716ab204a04", "7c368c0d395f492dbf31beadd460e281", "f4a53c58c5c64fd094d30aa26cd4d1d9", "f73a02a5670a4d66a3dc8e386a2f56e5", "db921fe611024aa9adcc3caa4dc89a39", "91dae13977c347079c8c2421acae1e67", "af78532809f14e63856e426d601e807e", "4b9f99f4c5ad44fa8f6c841bc00d8868" ] }, "id": "oRnl4HXvB5-T", "outputId": "b3c07d45-af92-4226-f68a-bb97cc1c215e" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/callback_connector.py:97: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=10)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.\n", " f\"Setting `Trainer(progress_bar_refresh_rate={progress_bar_refresh_rate})` is deprecated in v1.5 and\"\n", "GPU available: True, used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "Missing logger folder: /content/lightning_logs\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:309: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n", " FutureWarning,\n", "\n", " | Name | Type | Params\n", "-------------------------------------------\n", "0 | model | EmoModel | 82.1 M\n", "1 | loss | CrossEntropyLoss | 0 \n", "-------------------------------------------\n", "82.1 M Trainable params\n", "0 Non-trainable params\n", "82.1 M Total params\n", "328.492 Total estimated model params size (MB)\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "163afdc0b9264042b077d18e0eb52a6b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Sanity Checking: 0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "7ab54ba7ccc9484a8c8b2605d35a6a9c", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Training: 0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "ename": "RuntimeError", "evalue": "ignored", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 3\u001b[0m accumulate_grad_batches=hparams.accumulate_grad_batches)\n\u001b[1;32m 4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)\u001b[0m\n\u001b[1;32m 770\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstrategy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 771\u001b[0m self._call_and_handle_interrupt(\n\u001b[0;32m--> 772\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit_impl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_dataloaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mval_dataloaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdatamodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mckpt_path\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 773\u001b[0m )\n\u001b[1;32m 774\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_call_and_handle_interrupt\u001b[0;34m(self, trainer_fn, *args, **kwargs)\u001b[0m\n\u001b[1;32m 722\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstrategy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlauncher\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlaunch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrainer_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrainer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 723\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 724\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mtrainer_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 725\u001b[0m \u001b[0;31m# TODO: treat KeyboardInterrupt as BaseException (delete the code below) in v1.7\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 726\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mKeyboardInterrupt\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexception\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_fit_impl\u001b[0;34m(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)\u001b[0m\n\u001b[1;32m 810\u001b[0m \u001b[0mckpt_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel_provided\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel_connected\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlightning_module\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 811\u001b[0m )\n\u001b[0;32m--> 812\u001b[0;31m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mckpt_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mckpt_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 813\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 814\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstopped\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, model, ckpt_path)\u001b[0m\n\u001b[1;32m 1235\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_checkpoint_connector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresume_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1236\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1237\u001b[0;31m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_stage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1238\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1239\u001b[0m \u001b[0mlog\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetail\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{self.__class__.__name__}: trainer tearing down\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_run_stage\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1322\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredicting\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1323\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_predict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1324\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_train\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1325\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1326\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_pre_training_routine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_run_train\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1352\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1353\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_detect_anomaly\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_detect_anomaly\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1354\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1355\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1356\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_run_evaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0m_EVALUATE_OUTPUT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 202\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 203\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_start\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 205\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 206\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_restarting\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/fit_loop.py\u001b[0m in \u001b[0;36madvance\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 267\u001b[0m )\n\u001b[1;32m 268\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"run_training_epoch\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 269\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mepoch_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_data_fetcher\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 270\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 271\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 202\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 203\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_start\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 205\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 206\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_restarting\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py\u001b[0m in \u001b[0;36madvance\u001b[0;34m(self, data_fetcher)\u001b[0m\n\u001b[1;32m 206\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 207\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"run_training_batch\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 208\u001b[0;31m \u001b[0mbatch_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 209\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 210\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch_progress\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mincrement_processed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 202\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 203\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_start\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 205\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 206\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_restarting\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/batch/training_batch_loop.py\u001b[0m in \u001b[0;36madvance\u001b[0;34m(self, batch, batch_idx)\u001b[0m\n\u001b[1;32m 86\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlightning_module\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautomatic_optimization\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 87\u001b[0m \u001b[0moptimizers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_get_active_optimizers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_frequencies\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msplit_batch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 89\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 90\u001b[0m \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmanual_loop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msplit_batch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 202\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 203\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_start\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madvance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 205\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_advance_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 206\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_restarting\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36madvance\u001b[0;34m(self, batch, *args, **kwargs)\u001b[0m\n\u001b[1;32m 205\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_batch_idx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 206\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptim_progress\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_position\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 207\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_idx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 208\u001b[0m )\n\u001b[1;32m 209\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36m_run_optimization\u001b[0;34m(self, split_batch, batch_idx, optimizer, opt_idx)\u001b[0m\n\u001b[1;32m 254\u001b[0m \u001b[0;31m# gradient update with accumulated gradients\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 255\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 256\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizer_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mopt_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 257\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 258\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconsume_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36m_optimizer_step\u001b[0;34m(self, optimizer, opt_idx, batch_idx, train_step_and_backward_closure)\u001b[0m\n\u001b[1;32m 376\u001b[0m \u001b[0mon_tpu\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccelerator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTPUAccelerator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 377\u001b[0m \u001b[0musing_native_amp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mamp_backend\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mAMPType\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNATIVE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 378\u001b[0;31m \u001b[0musing_lbfgs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_lbfgs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 379\u001b[0m )\n\u001b[1;32m 380\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_call_lightning_module_hook\u001b[0;34m(self, hook_name, pl_module, *args, **kwargs)\u001b[0m\n\u001b[1;32m 1594\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1595\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"[LightningModule]{pl_module.__class__.__name__}.{hook_name}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1596\u001b[0;31m \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1597\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1598\u001b[0m \u001b[0;31m# restore current_fx when nested context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/lightning.py\u001b[0m in \u001b[0;36moptimizer_step\u001b[0;34m(self, epoch, batch_idx, optimizer, optimizer_idx, optimizer_closure, on_tpu, using_native_amp, using_lbfgs)\u001b[0m\n\u001b[1;32m 1623\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1624\u001b[0m \"\"\"\n\u001b[0;32m-> 1625\u001b[0;31m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptimizer_closure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1626\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1627\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0moptimizer_zero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepoch\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer_idx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/optimizer.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure, **kwargs)\u001b[0m\n\u001b[1;32m 166\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 167\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_strategy\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 168\u001b[0;31m \u001b[0mstep_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_strategy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizer_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 169\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 170\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_on_after_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/strategies/strategy.py\u001b[0m in \u001b[0;36moptimizer_step\u001b[0;34m(self, optimizer, opt_idx, closure, model, **kwargs)\u001b[0m\n\u001b[1;32m 191\u001b[0m \"\"\"\n\u001b[1;32m 192\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlightning_module\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 193\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprecision_plugin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mopt_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 194\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 195\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_setup_model_and_optimizers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mModule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizers\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOptimizer\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mModule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOptimizer\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/precision/precision_plugin.py\u001b[0m in \u001b[0;36moptimizer_step\u001b[0;34m(self, model, optimizer, optimizer_idx, closure, **kwargs)\u001b[0m\n\u001b[1;32m 153\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLightningModule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 154\u001b[0m \u001b[0mclosure\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpartial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_wrap_closure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 155\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 156\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 157\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_track_grad_norm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrainer\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"pl.Trainer\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 63\u001b[0m \u001b[0minstance\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_step_count\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[0mwrapped\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__get__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minstance\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 65\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mwrapped\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 66\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 67\u001b[0m \u001b[0;31m# Note that the returned function here is no longer a bound method,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 86\u001b[0m \u001b[0mprofile_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Optimizer.step#{}.step\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 87\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecord_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprofile_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 89\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/optimization.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure)\u001b[0m\n\u001b[1;32m 330\u001b[0m \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 331\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mclosure\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 332\u001b[0;31m \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 333\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 334\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mgroup\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparam_groups\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/precision/precision_plugin.py\u001b[0m in \u001b[0;36m_wrap_closure\u001b[0;34m(self, model, optimizer, optimizer_idx, closure)\u001b[0m\n\u001b[1;32m 138\u001b[0m \u001b[0mconsistent\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mthe\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mPrecisionPlugin\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0msubclasses\u001b[0m \u001b[0mthat\u001b[0m \u001b[0mcannot\u001b[0m \u001b[0;32mpass\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0mdirectly\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 139\u001b[0m \"\"\"\n\u001b[0;32m--> 140\u001b[0;31m \u001b[0mclosure_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 141\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_after_closure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 142\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mclosure_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 146\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 147\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 148\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclosure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 149\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_result\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 150\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36mclosure\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 132\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 133\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mclosure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mClosureResult\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m \u001b[0mstep_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_step_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 135\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 136\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mstep_output\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclosure_loss\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py\u001b[0m in \u001b[0;36m_training_step\u001b[0;34m(self, split_batch, batch_idx, opt_idx)\u001b[0m\n\u001b[1;32m 425\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 426\u001b[0m \u001b[0;31m# manually capture logged metrics\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 427\u001b[0;31m \u001b[0mtraining_step_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_strategy_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"training_step\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mstep_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 428\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstrategy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpost_training_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 429\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py\u001b[0m in \u001b[0;36m_call_strategy_hook\u001b[0;34m(self, hook_name, *args, **kwargs)\u001b[0m\n\u001b[1;32m 1764\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1765\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"[Strategy]{self.strategy.__class__.__name__}.{hook_name}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1766\u001b[0;31m \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1767\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1768\u001b[0m \u001b[0;31m# restore current_fx when nested context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pytorch_lightning/strategies/strategy.py\u001b[0m in \u001b[0;36mtraining_step\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 331\u001b[0m \"\"\"\n\u001b[1;32m 332\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprecision_plugin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_step_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 333\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtraining_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 334\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 335\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mpost_training_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mtraining_step\u001b[0;34m(self, batch, batch_idx)\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 23\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtraining_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"train\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 25\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 26\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvalidation_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, batch, step_name)\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstep_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"train\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 14\u001b[0m \u001b[0mloss_key\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mf\"{step_name}_loss\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0mtensorboard_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mloss_key\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, X, *args)\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 20\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 23\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtraining_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_, *args)\u001b[0m\n\u001b[1;32m 20\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 21\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput_\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbase_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_mask\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattention_mask\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 24\u001b[0m \u001b[0;31m# maybe do some pooling / RNNs... go crazy here!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m 858\u001b[0m \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 859\u001b[0m \u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 860\u001b[0;31m \u001b[0mreturn_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreturn_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 861\u001b[0m )\n\u001b[1;32m 862\u001b[0m \u001b[0msequence_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mencoder_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m 531\u001b[0m \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 532\u001b[0m \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 533\u001b[0;31m \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 534\u001b[0m )\n\u001b[1;32m 535\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m 415\u001b[0m \u001b[0mhead_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 416\u001b[0m \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 417\u001b[0;31m \u001b[0mpast_key_value\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself_attn_past_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 418\u001b[0m )\n\u001b[1;32m 419\u001b[0m \u001b[0mattention_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself_attention_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m 344\u001b[0m \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 345\u001b[0m \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 346\u001b[0;31m \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 347\u001b[0m )\n\u001b[1;32m 348\u001b[0m \u001b[0mattention_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m 1100\u001b[0m if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m 1101\u001b[0m or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1103\u001b[0m \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1104\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m 239\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 240\u001b[0m \u001b[0;31m# Take the dot product between \"query\" and \"key\" to get the raw attention scores.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 241\u001b[0;31m \u001b[0mattention_scores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery_layer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey_layer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 242\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 243\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mposition_embedding_type\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"relative_key\"\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mposition_embedding_type\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"relative_key_query\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mRuntimeError\u001b[0m: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 11.17 GiB total capacity; 10.52 GiB already allocated; 13.81 MiB free; 10.56 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF" ] } ], "source": [ "## train roughly for about 10-15 minutes with GPU enabled.\n", "trainer = pl.Trainer(gpus=1, max_epochs=hparams.epochs, progress_bar_refresh_rate=10,\n", " accumulate_grad_batches=hparams.accumulate_grad_batches)\n", "\n", "trainer.fit(module)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 272 }, "id": "Y8kzE1AeB_ij", "outputId": "932f2f1f-fb96-4a4e-a169-85f2150afa80" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "________________________________________________________________________________\n", " precision recall f1-score support\n", "\n", " sadness 0.961872 0.955250 0.958549 581\n", " joy 0.958580 0.932374 0.945295 695\n", " love 0.806818 0.893082 0.847761 159\n", " anger 0.936567 0.912727 0.924494 275\n", " fear 0.886364 0.870536 0.878378 224\n", " surprise 0.674699 0.848485 0.751678 66\n", "\n", " accuracy 0.923500 2000\n", " macro avg 0.870817 0.902076 0.884359 2000\n", "weighted avg 0.926988 0.923500 0.924647 2000\n", "\n" ] } ], "source": [ "with torch.no_grad():\n", " progress = [\"/\", \"-\", \"\\\\\", \"|\", \"/\", \"-\", \"\\\\\", \"|\"]\n", " module.eval()\n", " true_y, pred_y = [], []\n", " for i, batch_ in enumerate(module.test_dataloader()):\n", " (X, attn), y = batch_\n", " batch = (X.cuda(), attn.cuda())\n", " print(progress[i % len(progress)], end=\"\\r\")\n", " y_pred = torch.argmax(module(batch), dim=1)\n", " true_y.extend(y.cpu())\n", " pred_y.extend(y_pred.cpu())\n", "print(\"\\n\" + \"_\" * 80)\n", "print(classification_report(true_y, pred_y, target_names=label2int.keys(), digits=len(emotions)))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "U0_Z_4Pkl3fc", "outputId": "a45a771a-1566-4ef7-f935-52633d499c4b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sat Apr 2 02:35:31 2022 \n", "+-----------------------------------------------------------------------------+\n", "| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |\n", "|-------------------------------+----------------------+----------------------+\n", "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", "| | | MIG M. |\n", "|===============================+======================+======================|\n", "| 0 Tesla K80 Off | 00000000:00:04.0 Off | 0 |\n", "| N/A 50C P0 61W / 149W | 11259MiB / 11441MiB | 0% Default |\n", "| | | N/A |\n", "+-------------------------------+----------------------+----------------------+\n", " \n", "+-----------------------------------------------------------------------------+\n", "| Processes: |\n", "| GPU GI CI PID Type Process name GPU Memory |\n", "| ID ID Usage |\n", "|=============================================================================|\n", "+-----------------------------------------------------------------------------+\n" ] } ], "source": [ "!nvidia-smi" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ifER7sn-Htge" }, "outputs": [], "source": [] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "name": "RoBERTa_Fine_Tuning_Emotion_classification.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" } }, "nbformat": 4, "nbformat_minor": 1 }