01_PyTorch_Hello_World.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "title: 02 PyTorch Hello World!\n",
    "description: Build a simple neural network and train it\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/drive/1ac0K9_aa46c77XEeYtaMAfSOfmH1Bl9L?usp=sharing\" target=\"_blank\"><img align=\"left\" alt=\"Colab\" title=\"Open in Colab\" src=\"https://colab.research.google.com/assets/colab-badge.svg\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H7gQFbUxOQtb"
   },
   "source": [
    "# A First Shot at Deep Learning with PyTorch\n",
    "\n",
    "In this notebook, we are going to take a baby step into the world of deep learning using PyTorch. There are a ton of notebooks out there that teach you the fundamentals of deep learning and PyTorch, so here the idea is to give you some basic introduction to deep learning and PyTorch at a very high level. Therefore, this notebook is targeting beginners but it can also serve as a review for more experienced developers.\n",
    "\n",
    "After completion of this notebook, you are expected to know the basic components of training a basic neural network with PyTorch. I have also left a couple of exercises towards the end with the intention of encouraging more research and practise of your deep learning skills. \n",
    "\n",
    "---\n",
    "\n",
    "**Author:** Elvis Saravia([Twitter](https://twitter.com/omarsar0) | [LinkedIn](https://www.linkedin.com/in/omarsar/))\n",
    "\n",
    "**Complete Code Walkthrough:** [Blog post](https://medium.com/dair-ai/a-first-shot-at-deep-learning-with-pytorch-4a8252d30c75)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CkzttrQCwaSQ"
   },
   "source": [
    "## Importing the libraries\n",
    "\n",
    "Like with any other programming exercise, the first step is to import the necessary libraries. As we are going to be using Google Colab to program our neural network, we need to install and import the necessary PyTorch libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FuhJIaeXO2W9",
    "outputId": "bf494471-115e-45a8-c7cb-15a26f12154a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.10.0+cu111\n"
     ]
    }
   ],
   "source": [
    "## The usual imports\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "## print out the pytorch version used\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0a2C_nneO_wp"
   },
   "source": [
    "## The Neural Network\n",
    "\n",
    "![alt text](https://drive.google.com/uc?export=view&id=1Lpi4VPBfAV3JkOLopcsGK4L8dyxmPF1b)\n",
    "\n",
    "Before building and training a neural network the first step is to process and prepare the data. In this notebook, we are going to use syntethic data (i.e., fake data) so we won't be using any real world data. \n",
    "\n",
    "For the sake of simplicity, we are going to use the following input and output pairs converted to tensors, which is how data is typically represented in the world of deep learning. The x values represent the input of dimension `(6,1)` and the y values represent the output of similar dimension. The example is taken from this [tutorial](https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%202%20-%20Lesson%202%20-%20Notebook.ipynb). \n",
    "\n",
    "The objective of the neural network model that we are going to build and train is to automatically learn patterns that better characterize the relationship between the `x` and `y` values. Essentially, the model learns the relationship that exists between inputs and outputs which can then be used to predict the corresponding `y` value for any given input `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JWFtgUX85iwO"
   },
   "outputs": [],
   "source": [
    "## our data in tensor form\n",
    "x = torch.tensor([[-1.0],  [0.0], [1.0], [2.0], [3.0], [4.0]], dtype=torch.float)\n",
    "y = torch.tensor([[-3.0], [-1.0], [1.0], [3.0], [5.0], [7.0]], dtype=torch.float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NcQUjR_95z5J",
    "outputId": "6db5df38-6f9d-454e-87d6-cee0c29dccb3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 1])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## print size of the input tensor\n",
    "x.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9CJXO5WX1QtQ"
   },
   "source": [
    "## The Neural Network Components\n",
    "As said earlier, we are going to first define and build out the components of our neural network before training the model.\n",
    "\n",
    "### Model\n",
    "\n",
    "Typically, when building a neural network model, we define the layers and weights which form the basic components of the model. Below we show an example of how to define a hidden layer named `layer1` with size `(1, 1)`. For the purpose of this tutorial, we won't explicitly define the `weights` and allow the built-in functions provided by PyTorch to handle that part for us. By the way, the `nn.Linear(...)` function applies a linear transformation ($y = xA^T + b$) to the data that was provided as its input. We ignore the bias for now by setting `bias=False`.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "N1Ii5JRz3Jud"
   },
   "outputs": [],
   "source": [
    "## Neural network with 1 hidden layer\n",
    "layer1 = nn.Linear(1,1, bias=False)\n",
    "model = nn.Sequential(layer1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9HTWYD4aMBXQ"
   },
   "source": [
    "### Loss and Optimizer\n",
    "The loss function, `nn.MSELoss()`, is in charge of letting the model know how good it has learned the relationship between the input and output. The optimizer (in this case an `SGD`) primary role is to minimize or lower that loss value as it tunes its weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3hglFpejArxx"
   },
   "outputs": [],
   "source": [
    "## loss function\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "## optimizer algorithm\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FKj6jvZTUtGh"
   },
   "source": [
    "## Training the Neural Network Model\n",
    "We have all the components we need to train our model. Below is the code used to train our model. \n",
    "\n",
    "In simple terms, we train the model by feeding it the input and output pairs for a couple of rounds (i.e., `epoch`). After a series of forward and backward steps, the model somewhat learns the relationship between x and y values. This is notable by the decrease in the computed `loss`. For a more detailed explanation of this code check out this [tutorial](https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JeOr9i-aBzRv",
    "outputId": "299a0b60-a64c-46c4-d031-8aaf1cacbff9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 | Loss: 10.1346\n",
      "Epoch: 1 | Loss: 8.2589\n",
      "Epoch: 2 | Loss: 6.7509\n",
      "Epoch: 3 | Loss: 5.5385\n",
      "Epoch: 4 | Loss: 4.5636\n",
      "Epoch: 5 | Loss: 3.7798\n",
      "Epoch: 6 | Loss: 3.1497\n",
      "Epoch: 7 | Loss: 2.6430\n",
      "Epoch: 8 | Loss: 2.2356\n",
      "Epoch: 9 | Loss: 1.9081\n",
      "Epoch: 10 | Loss: 1.6448\n",
      "Epoch: 11 | Loss: 1.4331\n",
      "Epoch: 12 | Loss: 1.2628\n",
      "Epoch: 13 | Loss: 1.1260\n",
      "Epoch: 14 | Loss: 1.0159\n",
      "Epoch: 15 | Loss: 0.9275\n",
      "Epoch: 16 | Loss: 0.8563\n",
      "Epoch: 17 | Loss: 0.7991\n",
      "Epoch: 18 | Loss: 0.7532\n",
      "Epoch: 19 | Loss: 0.7162\n",
      "Epoch: 20 | Loss: 0.6865\n",
      "Epoch: 21 | Loss: 0.6626\n",
      "Epoch: 22 | Loss: 0.6433\n",
      "Epoch: 23 | Loss: 0.6279\n",
      "Epoch: 24 | Loss: 0.6155\n",
      "Epoch: 25 | Loss: 0.6055\n",
      "Epoch: 26 | Loss: 0.5975\n",
      "Epoch: 27 | Loss: 0.5910\n",
      "Epoch: 28 | Loss: 0.5858\n",
      "Epoch: 29 | Loss: 0.5816\n",
      "Epoch: 30 | Loss: 0.5783\n",
      "Epoch: 31 | Loss: 0.5756\n",
      "Epoch: 32 | Loss: 0.5734\n",
      "Epoch: 33 | Loss: 0.5717\n",
      "Epoch: 34 | Loss: 0.5703\n",
      "Epoch: 35 | Loss: 0.5691\n",
      "Epoch: 36 | Loss: 0.5682\n",
      "Epoch: 37 | Loss: 0.5675\n",
      "Epoch: 38 | Loss: 0.5669\n",
      "Epoch: 39 | Loss: 0.5664\n",
      "Epoch: 40 | Loss: 0.5661\n",
      "Epoch: 41 | Loss: 0.5658\n",
      "Epoch: 42 | Loss: 0.5655\n",
      "Epoch: 43 | Loss: 0.5653\n",
      "Epoch: 44 | Loss: 0.5652\n",
      "Epoch: 45 | Loss: 0.5650\n",
      "Epoch: 46 | Loss: 0.5649\n",
      "Epoch: 47 | Loss: 0.5649\n",
      "Epoch: 48 | Loss: 0.5648\n",
      "Epoch: 49 | Loss: 0.5647\n",
      "Epoch: 50 | Loss: 0.5647\n",
      "Epoch: 51 | Loss: 0.5647\n",
      "Epoch: 52 | Loss: 0.5646\n",
      "Epoch: 53 | Loss: 0.5646\n",
      "Epoch: 54 | Loss: 0.5646\n",
      "Epoch: 55 | Loss: 0.5646\n",
      "Epoch: 56 | Loss: 0.5646\n",
      "Epoch: 57 | Loss: 0.5646\n",
      "Epoch: 58 | Loss: 0.5645\n",
      "Epoch: 59 | Loss: 0.5645\n",
      "Epoch: 60 | Loss: 0.5645\n",
      "Epoch: 61 | Loss: 0.5645\n",
      "Epoch: 62 | Loss: 0.5645\n",
      "Epoch: 63 | Loss: 0.5645\n",
      "Epoch: 64 | Loss: 0.5645\n",
      "Epoch: 65 | Loss: 0.5645\n",
      "Epoch: 66 | Loss: 0.5645\n",
      "Epoch: 67 | Loss: 0.5645\n",
      "Epoch: 68 | Loss: 0.5645\n",
      "Epoch: 69 | Loss: 0.5645\n",
      "Epoch: 70 | Loss: 0.5645\n",
      "Epoch: 71 | Loss: 0.5645\n",
      "Epoch: 72 | Loss: 0.5645\n",
      "Epoch: 73 | Loss: 0.5645\n",
      "Epoch: 74 | Loss: 0.5645\n",
      "Epoch: 75 | Loss: 0.5645\n",
      "Epoch: 76 | Loss: 0.5645\n",
      "Epoch: 77 | Loss: 0.5645\n",
      "Epoch: 78 | Loss: 0.5645\n",
      "Epoch: 79 | Loss: 0.5645\n",
      "Epoch: 80 | Loss: 0.5645\n",
      "Epoch: 81 | Loss: 0.5645\n",
      "Epoch: 82 | Loss: 0.5645\n",
      "Epoch: 83 | Loss: 0.5645\n",
      "Epoch: 84 | Loss: 0.5645\n",
      "Epoch: 85 | Loss: 0.5645\n",
      "Epoch: 86 | Loss: 0.5645\n",
      "Epoch: 87 | Loss: 0.5645\n",
      "Epoch: 88 | Loss: 0.5645\n",
      "Epoch: 89 | Loss: 0.5645\n",
      "Epoch: 90 | Loss: 0.5645\n",
      "Epoch: 91 | Loss: 0.5645\n",
      "Epoch: 92 | Loss: 0.5645\n",
      "Epoch: 93 | Loss: 0.5645\n",
      "Epoch: 94 | Loss: 0.5645\n",
      "Epoch: 95 | Loss: 0.5645\n",
      "Epoch: 96 | Loss: 0.5645\n",
      "Epoch: 97 | Loss: 0.5645\n",
      "Epoch: 98 | Loss: 0.5645\n",
      "Epoch: 99 | Loss: 0.5645\n",
      "Epoch: 100 | Loss: 0.5645\n",
      "Epoch: 101 | Loss: 0.5645\n",
      "Epoch: 102 | Loss: 0.5645\n",
      "Epoch: 103 | Loss: 0.5645\n",
      "Epoch: 104 | Loss: 0.5645\n",
      "Epoch: 105 | Loss: 0.5645\n",
      "Epoch: 106 | Loss: 0.5645\n",
      "Epoch: 107 | Loss: 0.5645\n",
      "Epoch: 108 | Loss: 0.5645\n",
      "Epoch: 109 | Loss: 0.5645\n",
      "Epoch: 110 | Loss: 0.5645\n",
      "Epoch: 111 | Loss: 0.5645\n",
      "Epoch: 112 | Loss: 0.5645\n",
      "Epoch: 113 | Loss: 0.5645\n",
      "Epoch: 114 | Loss: 0.5645\n",
      "Epoch: 115 | Loss: 0.5645\n",
      "Epoch: 116 | Loss: 0.5645\n",
      "Epoch: 117 | Loss: 0.5645\n",
      "Epoch: 118 | Loss: 0.5645\n",
      "Epoch: 119 | Loss: 0.5645\n",
      "Epoch: 120 | Loss: 0.5645\n",
      "Epoch: 121 | Loss: 0.5645\n",
      "Epoch: 122 | Loss: 0.5645\n",
      "Epoch: 123 | Loss: 0.5645\n",
      "Epoch: 124 | Loss: 0.5645\n",
      "Epoch: 125 | Loss: 0.5645\n",
      "Epoch: 126 | Loss: 0.5645\n",
      "Epoch: 127 | Loss: 0.5645\n",
      "Epoch: 128 | Loss: 0.5645\n",
      "Epoch: 129 | Loss: 0.5645\n",
      "Epoch: 130 | Loss: 0.5645\n",
      "Epoch: 131 | Loss: 0.5645\n",
      "Epoch: 132 | Loss: 0.5645\n",
      "Epoch: 133 | Loss: 0.5645\n",
      "Epoch: 134 | Loss: 0.5645\n",
      "Epoch: 135 | Loss: 0.5645\n",
      "Epoch: 136 | Loss: 0.5645\n",
      "Epoch: 137 | Loss: 0.5645\n",
      "Epoch: 138 | Loss: 0.5645\n",
      "Epoch: 139 | Loss: 0.5645\n",
      "Epoch: 140 | Loss: 0.5645\n",
      "Epoch: 141 | Loss: 0.5645\n",
      "Epoch: 142 | Loss: 0.5645\n",
      "Epoch: 143 | Loss: 0.5645\n",
      "Epoch: 144 | Loss: 0.5645\n",
      "Epoch: 145 | Loss: 0.5645\n",
      "Epoch: 146 | Loss: 0.5645\n",
      "Epoch: 147 | Loss: 0.5645\n",
      "Epoch: 148 | Loss: 0.5645\n",
      "Epoch: 149 | Loss: 0.5645\n"
     ]
    }
   ],
   "source": [
    "## training\n",
    "for ITER in range(150):\n",
    "    model = model.train()\n",
    "\n",
    "    ## forward\n",
    "    output = model(x)\n",
    "    loss = criterion(output, y)\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    ## backward + update model params \n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    model.eval()\n",
    "    print('Epoch: %d | Loss: %.4f' %(ITER, loss.detach().item()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bp50Q7J0Xkiw"
   },
   "source": [
    "## Testing the Model\n",
    "After training the model we have the ability to test the model predictive capability by passing it an input. Below is a simple example of how you could achieve this with our model. The result we obtained aligns with the results obtained in this [notebook](https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%202%20-%20Lesson%202%20-%20Notebook.ipynb), which inspired this entire tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "V1odfZpGFoBi",
    "outputId": "a447b232-729e-4ccf-adc2-5aeaf79cc2ea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17.096769332885742\n"
     ]
    }
   ],
   "source": [
    "## test the model\n",
    "sample = torch.tensor([10.0], dtype=torch.float)\n",
    "predicted = model(sample)\n",
    "print(predicted.detach().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ozX4V1GhPLyr"
   },
   "source": [
    "## Final Words\n",
    "\n",
    "Congratulations! In this tutorial you learned how to train a simple neural network using PyTorch. You also learned about the basic components that make up a neural network model such as the linear transformation layer, optimizer, and loss function. We then trained the model and tested its predictive capabilities. You are well on your way to become more knowledgeable about deep learning and PyTorch. I have provided a bunch of references below if you are interested in practising and learning more. \n",
    "\n",
    "*I would like to thank Laurence Moroney for his excellent [tutorial](https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%202%20-%20Lesson%202%20-%20Notebook.ipynb) which I used as an inspiration for this tutorial.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LAABGiMHeDOr"
   },
   "source": [
    "## Exercises\n",
    "- Add more examples in the input and output tensors. In addition, try to change the dimensions of the data, say by adding an extra value in each array. What needs to be changed to successfully train the network with the new data?\n",
    "- The model converged really fast, which means it learned the relationship between x and y values after a couple of iterations. Do you think it makes sense to continue training? How would you automate the process of stopping the training after the model loss doesn't subtantially change?\n",
    "- In our example, we used a single hidden layer. Try to take a look at the PyTorch documentation to figure out what you need to do to get a model with more layers. What happens if you add more hidden layers?\n",
    "- We did not discuss the learning rate (`lr-0.001`) and the optimizer in great detail. Check out the [PyTorch documentation](https://pytorch.org/docs/stable/optim.html) to learn more about what other optimizers you can use.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4-o4w9vpPHZz"
   },
   "source": [
    "## References\n",
    "- [The Hello World of Deep Learning with Neural Networks](https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%202%20-%20Lesson%202%20-%20Notebook.ipynb)\n",
    "- [A Simple Neural Network from Scratch with PyTorch and Google Colab](https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0?source=collection_category---4------1-----------------------)\n",
    "- [PyTorch Official Docs](https://pytorch.org/docs/stable/nn.html)\n",
    "- [PyTorch 1.2 Quickstart with Google Colab](https://medium.com/dair-ai/pytorch-1-2-quickstart-with-google-colab-6690a30c38d)\n",
    "- [A Gentle Intoduction to PyTorch](https://medium.com/dair-ai/pytorch-1-2-introduction-guide-f6fa9bb7597c)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "PyTorch Hello World.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}