instructor_intr_notebook_example_training.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/adding_grading_levels_to_instructor_nb/instructor_intr_notebook_example_training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMKrKfx8_3fc"
      },
      "source": [
        "# Instructor Grading and Assessment\n",
        "This notebook executes grading of student submissions based on the examples provided in the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions) from Dr. Jesse Blocher. In this iteration, we use the Unstructured File Loader, which cannot proccess .json files (the preferred format). We are working on finding a file loader that allows .json. In this version of the notebook, the model has only been trained on Question 2 from the notebook.\n",
        "\n",
        "To train the model, we used 2 out of the three student example from each grade brack and inputted into a .pdf with clearly defined levels. Then, we used the excluded answers to test the accuracy of the model's grading."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Load and Install Neccessary Libraries"
      ],
      "metadata": {
        "id": "2UQgQSoMx4My"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install -q langchain=='0.0.229' openai gradio numpy chromadb tiktoken unstructured pdf2image pydantic==\"1.10.8\" jq"
      ],
      "metadata": {
        "id": "UJi1Oy0CyPHD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# import necessary libraries here\n",
        "from getpass import getpass\n",
        "from langchain.llms import OpenAI as openai\n",
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain.prompts import PromptTemplate\n",
        "from langchain.document_loaders import TextLoader\n",
        "from langchain.indexes import VectorstoreIndexCreator\n",
        "from langchain.text_splitter import CharacterTextSplitter\n",
        "from langchain.embeddings import OpenAIEmbeddings\n",
        "from langchain.schema import SystemMessage, HumanMessage, AIMessage\n",
        "import numpy as np\n",
        "import os\n",
        "from langchain.vectorstores import Chroma\n",
        "from langchain.document_loaders.unstructured import UnstructuredFileLoader\n",
        "from langchain.document_loaders import UnstructuredFileLoader\n",
        "from langchain.chains import VectorDBQA\n",
        "from langchain.document_loaders import JSONLoader\n",
        "import json\n",
        "from pathlib import Path\n",
        "from pprint import pprint\n",
        "\n",
        "\n",
        "from langchain.prompts.few_shot import FewShotPromptTemplate\n",
        "from langchain.prompts.prompt import PromptTemplate"
      ],
      "metadata": {
        "id": "YHytCUoExrYe"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Set up model and pass OpenAI Key\n",
        "Here we are setting up the model and using a system message to pass a persona prompt with the grading advice"
      ],
      "metadata": {
        "id": "4elyN72szz-_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# setup open AI api key\n",
        "openai_api_key = getpass()"
      ],
      "metadata": {
        "id": "jVPEFX3ixJnM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
        "openai.api_key = openai_api_key"
      ],
      "metadata": {
        "id": "obplpeB78h_M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Initiate model (model type and specify persona)\n",
        "llm = ChatOpenAI(model='gpt-3.5-turbo-16k')\n",
        "messages = [\n",
        "    SystemMessage(content=\"You are a helpful grading assistant. In grading the following questions, keep in mind the advice from the professor: one aspect of it was being specific. The poor answers have a lot of platitudes, the better answers give specific examples. Secondly, they should discuss automation and/or prediction specifically. Those are the things that ML does, it is not 'technology' broadly.\"),\n",
        "    HumanMessage(content=\"\")\n",
        "]"
      ],
      "metadata": {
        "id": "e_ZavOnS8iuE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Original route: Vector Stores from .json files\n",
        "This section uses .json files and vector stores with system/human messaging and a persona prompt tailored to Dr. Blocher's grading philosophy for the Wiki examples (only includes section 2 at this time)."
      ],
      "metadata": {
        "id": "KHS3E6PydN-2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Grading based on A, B, and C-level answers from previous students to Question 2 from the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions):\n",
        "\n",
        "**Question 2:** Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses)."
      ],
      "metadata": {
        "id": "IYCBSD_8l7uu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating .json file from case examples (Question 2)\n",
        "The purpose of this cell is to create a json file based on the previously submitted, graded work of students based on the case file provided by Dr. Blocher in the Wiki. So, here you could create your own file, or for quick demo purposes you can use the zip file in the next section heading."
      ],
      "metadata": {
        "id": "TYlGEusr64kA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "q2 = 'Question 2: Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses).'"
      ],
      "metadata": {
        "id": "DAIaKTRYxOlh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# A-level answers\n",
        "\n",
        "q2_A_answer_1 = 'Machine learning is extremely important tool for businesses. It can be used in a variety of ways, but most importantly, it can be used to identify patterns within their data that might not otherwise be identified by human beings. For example, it can understand customer behaviors, optimize logistics, and expand efficiencies throughout the business. Machine learning does not get tired, meaning it can work as long as you want it to. It can sift through massive amounts of data, that no human being can look through in an efficient manner. Machine learning can be used as a tool to identify anomalies when something needs to be checked to save or gain money. The predictions that companies gain from machine learning are cheap, accurate, and automate. These machine learning algorithms can be brought to larger scales to encompass the whole business and its operations. It is important to note, Machine learning is just predictions. Predictions to understand important patterns that could make or break a company since they understand the patterns of their business more. It is an amazing tool, but should be used wisely and carefully because if not, it can expensive, useless, and straight up wrong.'\n",
        "q2_A_answer_2 = 'Machine learning is important for most of the sectors in business. Overall, it gives the company of an overview about what would be the trend for their business industry, and analyze the customer behavior to help business segment their customers groups. Today, many companies have a vast amount of information generated by behavior, computer, events, people, and devices. This massive amount of data is difficult for human to handle, and even if human manages it, it is not profitable as human labor is expensive. Thanks to machine learning, companies can utilize their in-house or even third-party data to make something useful for their business. In medical analysis, for example, with human, it takes a very long time to find patterns in thousands of MRI scans. On the other hand, machines can detect patterns in seconds by entering data as long as the information is correctly labeled or trained properly. Another example would be segmenting customer group. In marketing department, the business could use unsupervised machine learning to cluster their customer segments to generate personalized contents that are relevant for each of individuals.'\n",
        "\n",
        "# List creation\n",
        "\n",
        "q2_A_answers_list = [q2_A_answer_1, q2_A_answer_2]\n"
      ],
      "metadata": {
        "id": "yQT6aExSr1dP"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# B-level answers\n",
        "\n",
        "q2_B_answer_1 = 'Companies use ML models to improve different aspects of their business, like manufacturing, hiring, deployment, advertising, etc. The main goal is to improve productive and increase profitability of the company. The ML models are fed with company and externally available data to help the company optimize its departments and in turn become more financially successful/ productive. For example, using purchasing history, the company can predict who to advertise products to, to increase sales.'\n",
        "q2_B_answer_2 = 'Machine learning allows business to have automated decision, scale, predictive analysis and performance. Machine learning also helps a business have a data strategy. This is how a firm uses data, data infrastructure, governance, etc. to accomplish its strategic goals and maintain/grow their competitive advantage within their industry.'\n",
        "q2_B_answer_3 = 'The short answer is ML can help make decisions for businesses. To be clarified, ML does not make decisions for businesses. I mean it can, but people have not trusted ML enough yet and ML has not been that good to let it directly make business decisions. Business people only use ML to help themselves get intuitions of how decisions should be made and make predictions of results they might get based on their decisions. For example, if a business tries to launch a new product, it will use ML to test whether it will work or not on a small scale before it is introduced to a large scale. People called this step piloting. In this step, people collect data that is generated by using the pilot product and analyze their next move. They could tweak some features based on the feedback. If they think the data in their interest shows the product performs well, they might predict that this product will be successful when it is introduced on a large scale. Then, they will launch it.'\n",
        "# List creation\n",
        "\n",
        "q2_B_answers_list = [q2_B_answer_1, q2_B_answer_2, q2_B_answer_3]"
      ],
      "metadata": {
        "id": "KB1CmeRwtRvf"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# C-level answers\n",
        "\n",
        "q2_C_answer_1 = 'Machine learning powers many of the services we use today, such as the recommendation systems of Spotify and Netflix; search engines such as Google and Bing; social media such as TikTok and Instagram; voices such as Siri and Alexa, the list can go on. All these examples show that machine learning is already starting to play a pivotal role in today\"s data-rich world. Machines can help us sift through useful information that can lead to big breakthroughs, and we have seen the widespread use of this technology in various industries such as finance, healthcare, insurance, manufacturing, transformational change, etc.'\n",
        "q2_C_answer_2 = 'As technology advanced, there are tons of new data generated and stored. All industries experienced this surge in data, including business. There is a huge amount of business data stored and waited in the database of each firm and they need solutions to utilize these data. Machine learning is a very promising approach for firms to puts these data in and output a meaning pattern or result that could help the firms with their existing work. This could turn into a working product or provide insights that could enhance the efficiency of the company’s workflow. With machine learning, a firm could either enter a new market with the new product or save time and effort with the meaningful insights. Achieving these possibilities with the data they already owned is a low effort but high reward action. This is the reason machine learning is valued by many businesses recently.'\n",
        "\n",
        "# List creation\n",
        "\n",
        "q2_C_answers_list = [q2_C_answer_1, q2_C_answer_2]"
      ],
      "metadata": {
        "id": "3diAz43othjc"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "q2_Q_and_A = [\"Question:\", q2, \"A-level Answers\", q2_A_answers_list, \"B-level Answers\", q2_B_answers_list, \"C-level Answers\", q2_C_answers_list]"
      ],
      "metadata": {
        "id": "oAnrMSU6u9do"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "from google.colab import files\n",
        "\n",
        "def save_example_answers(examples, filename='wiki_ABC_Q2examples.json'):\n",
        "  with open(filename, 'w') as file:\n",
        "    json.dump(examples, file)\n",
        "  files.download(filename)\n",
        "\n",
        "save_example_answers(q2_Q_and_A)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "B16iYMEnri9s",
        "outputId": "3e565b3d-804c-4b5e-acc8-efeb955c6c14"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_8ec6f24c-9653-4335-8c1b-766926434399\", \"wiki_ABC_Q2examples.json\", 5931)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating a Vector Store\n",
        "Here we create a vector store based on the .json file (which you can find at this [link](https://drive.google.com/file/d/1nk6JhbqoUHFie-Ewb436pdV-onlObZUt/view?usp=sharing), but will need to unzip)."
      ],
      "metadata": {
        "id": "exXR-A2oxWeg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Upload .json or zip\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "metadata": {
        "id": "Wpt6qsmEw8WP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Unzip file if neccessary\n",
        "!unzip '/content/wiki_ABC_Q2examples (2).json.zip'"
      ],
      "metadata": {
        "id": "7T2LpkiZh9LT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create file path\n",
        "data = '/content/wiki_ABC_Q2examples (2).json'"
      ],
      "metadata": {
        "id": "CVY0CVvhxyCu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the .json\n",
        "data = json.loads(Path(file_path).read_text())\n",
        "data = str(data)"
      ],
      "metadata": {
        "id": "dHjK0nN6yYPH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create Vector Store\n",
        "\n",
        "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
        "texts = text_splitter.split_text(data)\n",
        "\n",
        "embeddings = OpenAIEmbeddings()\n",
        "\n",
        "db = Chroma.from_texts(texts, embeddings)\n",
        "\n",
        "qa = VectorDBQA.from_chain_type(llm=llm, chain_type=\"stuff\", vectorstore=db, k=1)"
      ],
      "metadata": {
        "id": "paibxyeuxnu1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating examples from student answers to test the Wiki\n",
        "\n",
        "example1 = 'The most powerful aspect of machine learning is its ability to automate processes. If the goal is well defined and repeatable, an algorithm can be trained to perform that task far faster than any human and often with more reliability. Because of this, businesses can implement machine learning algorithms to solve problems human labor was previously required for, whether that be mental or physical labor. Although implementing machine learning often has high initial costs, because computers do not require payment outside of maintaining their operation, in the long run companies can save money by either making their workers’ tasks more efficient or by entirely automating tasks. This increases the profit margins for those firms.'\n",
        "# B level\n",
        "example2 ='ML systems can help improve access to data while managing compliance. It helps businesspeople deal with data in a more efficient way, since there is numerous data generated per second in such industry. If they use some of the simple classification models with high accuracy, it will save them a lot of time on data cleaning and validation, which are kind of repetitive and time-consuming. For example, NLP provides an easy way for personnel to query business information, understand business processes, and discover new relationships between business data, ideas based on intuition and insight often emerge. Using models to do prediction helps people making wiser decision. Since models can handle way more data than human, newly collected data can feed in the model and get some predictive result as a reference to the decision makers. This is significant because in this industry, time is precious, and traders must decide quickly and precisely. A little negligence will lead to a big mistake, lose a lot of money, and even affect the company\"s reputation. Models can see patterns that are not easy for human to spot, which is also valuable for modify the way people doing analysis and interpret.'\n",
        "# C level\n",
        "example3 = 'The machine learning model (or one a broader view, artificial intelligence) is about prediction. According to the lecture, there are tree main innovations in it. Prediction is cheap, more accurate and automated. As a result, armed with machine learning, businesses could automatically gain much more accurate and powerful capabilities in forecasting, leading to a big savings both in time and money.'\n",
        "\n",
        "# Randomized list of answers\n",
        "training_answers = [example2, example1, example3]"
      ],
      "metadata": {
        "id": "-fyY-ftV-ZbG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Grading based on A, B, and C-level answers from previous students to Question 4 from the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions):\n",
        "\n",
        "**Question 4:** Describe an example where machine learning is not likely to work well. This should be an example where intelligent, rational people could think of using machine learning or even are already implementing it. Describe specifically why you think ML is unlikely to work well. This should not be an example where someone is implementing it badly (i.e. skipping testing). Rather, it should be an example of how ML simply will not work well, no matter how many smart people work carefully on the problem. You may use examples we’ve discussed in class; it does not need to be a novel application."
      ],
      "metadata": {
        "id": "sSnxqnGDmPzC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating .json file from case examples (Question 4)\n",
        "The purpose of this cell is to create a json file based on the previously submitted, graded work of students based on the case file provided by Dr. Blocher in the Wiki."
      ],
      "metadata": {
        "id": "RMkTwaEFmhWd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Context: question\n",
        "\n",
        "q4 = 'Question 4: Describe an example where machine learning is not likely to work well. This should be an example where intelligent, rational people could think of using machine learning or even are already implementing it. Describe specifically why you think ML is unlikely to work well. This should not be an example where someone is implementing it badly (i.e. skipping testing). Rather, it should be an example of how ML simply will not work well, no matter how many smart people work carefully on the problem. You may use examples we’ve discussed in class; it does not need to be a novel application.'"
      ],
      "metadata": {
        "id": "_UVEgvzKmltw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# A-level answers\n",
        "\n",
        "q4_A_answer_1 = 'This is one of the cases discussed in class: Machine learning on the management of at-risk children. Traditionally, at-risk children are mostly determined by social workers and they would report to their agencies if some children are at-risk. The at-risk children will subsequentially be taken into custody before their circumstance is evaluated and a decision about their future is reached. If a machine-learning algorithm were to replace the aforementioned social worker: the biggest concern would be when the algorithm outputs a false positive or false negative prediction. A false positive means that the children are said to be at risk while actually not and removed from their households. A false negative would be when some children are not removed from the dangerous household because the algorithm returned negative results. The costs of both incorrect decisions are too big for anyone to afford. Admittedly, data scientists might be able to develop a great algorithm with very low false positive and false negative rates, and that social workers can easily make the same amount of mistakes or more, but it is still too much for our humane society to take by putting such a life-changing or even life-or-death decision in a machine’s metaphorical hands.'\n",
        "q4_A_answer_2 = 'I believe machine learning will not work well in the court system when making sentences and decisions. I have heard in the news that machine learning was helping judges sentence defenders. Currently, I am unaware if this is still happening but this will not work because our court system has a racial basis. This racial bias stems back hundreds of years and a machine-learning model would exploit the discrimination already so present in our system. The dataset used to train the model has bias that would unfairly sentence a person because of their demographic. This overall is bad because there is no dataset you can use to train a model that already does not include some sort of inherited bias.'\n",
        "\n",
        "# List creation\n",
        "\n",
        "q4_A_answers_list = [q4_A_answer_1, q4_A_answer_2]\n"
      ],
      "metadata": {
        "id": "Irw0U9PBmupJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# B-level answers\n",
        "\n",
        "q4_B_answer_1 = 'I believe that machine learning won’t work well in the advent of self-driving cars. As discussed in class, machines are bad at “analogies, small data or rare events, and identifying causality” (Lecture 4: Data Strategy.) While most driving occurrences are habitual and can be predicted, especially day-to-day events like driving to work or the grocery store, so much driving is small or rare events that can’t necessarily be predicted. For example, you can’t predict that the person behind you is texting while driving and is going to rear end you, or that a snowstorm can make you start to hydroplane on the road. So much of the process of learning is understanding the exceptions, such as driving in the rain, at night, or almost avoiding an accident. For self-driving cars to be successful, so much data is required to reduce error, which I don’t think is possible to acquire. While most algorithms can have leeway in their accuracy, self-driving cars cannot have any errors — a difficult feat.'\n",
        "q4_B_answer_2 = 'As we discussed in class, in a hospital setting, ML cannot do the task of diagnosing medical conditions well. While ML have been used in some diagnostic tasks, such as detecting some certain types of cancer, it generally to be considered difficult and will not be used in a large scale because medical conditions can have many different conditions can have many different symptoms and can be difficult to diagnose even for experienced doctors. The training of a machine learning model for diagnosing medical conditions requires a large dataset of medical records, along with labels indicating each diagnosis. But there are several barriers. First, medical records are often private and sensitive, and sharing them may raise legal and ethical concerns. Second, medical conditions are complicated, it’s hard to ensure the labels are correct, some of which may takes years or decades to testify. Even if they can recognize some pattern for some disease, symptoms vary from person to person. At a first glance, even after running some tests, it’s hard for doctors to tell the case. It will be harder to use ML in diagnose in general.'\n",
        "\n",
        "q4_B_answers_list = [q4_B_answer_1, q4_B_answer_2]"
      ],
      "metadata": {
        "id": "haKh_FsenmT3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# C-level answers\n",
        "\n",
        "q4_C_answer_1 = 'One example would be unstable models. Some of the models frequently exhibit extreme instability and degrade over time. In such circumstances, the company can call for model monitoring and high-frequency model revision. Businesses may begin to revert to intuition-based approach when model creation lead times increase.'\n",
        "q4_C_answer_2 = 'The example I want to describe is about GPS. We have heard that some drivers who blindly followed their GPS into disaster. The news reported that Three tourists from Bellevue, Washington, got lost after midnight in 2011 when they were unable to find their way back to the hotel. They took what they believed to be a road that would take them to the freeway after requesting the GPS to reroute them. Instead, their SUV submerged into a large body of water. These people trust algorithms and data more than their own logic and judgement. People who drive into the water just because they believe too much in GPS. Algorithms will never think like a human being and cannot make moral judgement like a human being. Machine learning is unable to provide any guidance on the accepted norms. Machine minds & human minds could never be the same. Everyone’s answer is all different, We cannot collect everyone’s information around the world, that is so hard. As long as there is one person’s information not counted, then it will have difference from what you predict. The world always require human’s judgement and ethics, that is the point can not be changed.'\n",
        "q4_C_answer_3 = 'The organization and the abundance of your data is paramount in the applying machine learning. When your data are either incomplete, like not enough data, or unstructured, like messy data with a lot of missingness and inconsistencies, you are not likely to obtain a good result even from implementing the best model in the world. There are some techniques that you might be able to amplify your data or transfer other data into your own, such as transfer learning or domain adaptation. However, it is still strongly recommended that you should have plenty of data to perform machine learning on. Your data should also be able to well represent the question you are trying to answer. Low coverage of the true target population is likely to result in skewness of the result. Machines learning is not likely to work well in this scenario.'\n",
        "\n",
        "# List creation\n",
        "\n",
        "q4_C_answers_list = [q4_C_answer_1, q4_C_answer_2, q4_C_answer_3]"
      ],
      "metadata": {
        "id": "YNHULek7oCE_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "q4_Q_and_A = [\"Question:\", q4, \"A-level Answers\", q4_A_answers_list, \"B-level Answers\", q4_B_answers_list, \"C-level Answers\", q4_C_answers_list]"
      ],
      "metadata": {
        "id": "9bXSVmPnojIh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "from google.colab import files\n",
        "\n",
        "def save_example_answers(examples, filename='wiki_ABC_Q4examples.json'):\n",
        "  with open(filename, 'w') as file:\n",
        "    json.dump(examples, file)\n",
        "  files.download(filename)\n",
        "\n",
        "save_example_answers(q4_Q_and_A)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "3e565b3d-804c-4b5e-acc8-efeb955c6c14",
        "id": "Qqwk0mvroqWz"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_8ec6f24c-9653-4335-8c1b-766926434399\", \"wiki_ABC_Q2examples.json\", 5931)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Grading based on A, B, and C-level answers from previous students to Question 5 from the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions):\n",
        "\n",
        "**Question 5:** Describe a new example where you think machine learning could work. This cannot be examples we’ve given in class like customer churn, fraud detection, image or object recognition, disease diagnosis, stock prediction, self-driving cars, etc. t does not need to be completely novel, e.g., you can look for news stories and find something new. I want something that is different from the commonly used examples. You may use an example that you or another student brought up in class if it meets these criteria. Be sure to explain why you think it is an example where ML could work. Note: You do not need to be proven right that ML works. I am mostly interested in your logic and arguments of why you think it is a good idea, drawing on what we’ve discussed in class."
      ],
      "metadata": {
        "id": "RfofiUreov7C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating .json file from case examples (Question 5)\n",
        "The purpose of this cell is to create a json file based on the previously submitted, graded work of students based on the case file provided by Dr. Blocher in the Wiki."
      ],
      "metadata": {
        "id": "P2h8XtXQo6eO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Context: question\n",
        "\n",
        "q5 = 'Question 5: Describe a new example where you think machine learning could work. This cannot be examples we’ve given in class like customer churn, fraud detection, image or object recognition, disease diagnosis, stock prediction, self-driving cars, etc. t does not need to be completely novel, e.g., you can look for news stories and find something new. I want something that is different from the commonly used examples. You may use an example that you or another student brought up in class if it meets these criteria. Be sure to explain why you think it is an example where ML could work. Note: You do not need to be proven right that ML works. I am mostly interested in your logic and arguments of why you think it is a good idea, drawing on what we’ve discussed in class.'"
      ],
      "metadata": {
        "id": "evpoXMUSpCi4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# A-level answers\n",
        "\n",
        "q5_A_answer_1 = 'As I recall, our group discussed that using ML to do intrusion detection for computers. Nowadays, people’s information is digitalized and stored in databases. Since databases are also servers (i.e., computers), there must be some vulnerability. Software is more common to do detection for now, but it can just recognize the intrusion pattern that already known or preset by the programmer. However, this can be accomplished by employing algorithms that can analyze large amounts of data and detect anomalies that may indicate an attempted intrusion. We can feed ML model with network traffic, labeled or not labeled, system logs to let it learn and identify unusual patterns or behavior. With well-learned models, we can use them to find out signs of unauthorized access or even malware. One of the greatest things is that ML model may be able to find some hidden pattern that human has not and predict how and when the intrusion will take place, or even figure out how the malware will function to attach. This trait is one of the most important traits that distinguish ML model from regular software.'\n",
        "q5_A_answer_2 = 'Machine learning can be used in NLP (natural language processing). It is very useful to deal with massive unstructured text data. For example, we want to build a machine learning model to reflect the personality of Einstein according to the letters he wrote in the past decades to his family and friends. We can use machine learning models for NLP to transfer the unstructured text data to usable data first. Then we will use these data to train the model. After training the model by the letters Einstein wrote in the past decades, the model will have the same or similar personality as Einstein. And now we input the test letters into the model again, we expect the model will give us the sentiment scores of each letter (for example: humor-0.732, joyful-0.193, fear-0.015, sad-0.002). After we got the scores for each letter, we use statistical methods to analyze the scores and we will finally get an aggregate personality of Einstein.'\n",
        "q5_A_answer_3 = 'The sports world is a place where I generally think machine learning can be well-utilized, but not in all cases. Something I’m thinking about is fantasy football, in which football fans can draft a team and decide who to “start” on their team for a given week and who to “sit.” NFL pundits and commentators can absolutely give ideas on who would be optimal, and do a great job at it, but when it comes to actual predictions for how many points a particular player might score in a given week, based on opponent average stats, expected workload, and other factors, while also measuring that against every other potential player, having an algorithm to back this up is extremely helpful. Sports betting can also benefit from these types of algorithms, and being able to accurately predict something as volatile as sports could have major business (and fandom) implications.'\n",
        "\n",
        "# List creation\n",
        "\n",
        "q5_A_answers_list = [q5_A_answer_1, q5_A_answer_2, q5_A_answer_3]\n"
      ],
      "metadata": {
        "id": "B9abYhEupKA_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# B-level answers\n",
        "\n",
        "q5_B_answer_1 = 'Machine learning can work well when it comes to finance organization, planning, and tracking. Since machine learning works well with statistical inference, it’s useful in decision making. For things such as planning and organizing, one example that machine learning can work well in is identifying price values for toiletries, groceries, and necessity items across various store within a specific location and output the best location for the user. We know that certain retailers mark up prices for the items on their shelfs. Some items are more expensive in one location than others and often time, certain prices are not posted online. In addition to that, traveling to different stores and compare prices are very inefficient because it can waste gas and time. To solve the problem, the algorithm will examine the user’s location and the item(s) that the user wants and output the best stores for the items. How this item work is that they take in data from all the closest stores, rank each item based on value, distance, and availability. On top of that, if users compile a lists of necessity items and output them in the model, the model can use data and compare stores that have more than one item on the list plus the other features (value, distance, and availability), and recommend the place, or places if item are not available, that could cut down time and expense for the users.'\n",
        "q5_B_answer_2 = 'Natural gas consumption has rebounded in the last year, with economic recovery and increased extreme weather leading to increased demand for natural gas, resulting in a tight market and soaring prices. Gas supply agencies want to use ML to predict the deliverability of natural gas storage in depleted reservoirs. First, natural gas is most commonly stored in underground formations: 1) depleted reservoirs, 2) aquifers, and 3) salt caverns. A depleted reservoir must have some elements of a good oil and gas formation to be converted to a subsurface natural gas reservoir, such as good porosity and permeability, the presence of good seal rock, and the presence of cap rock. I think this thing can be achieved with ML because there is a huge amount of data collected from natural gas extraction sites around the world since 1915, and this data provides a good basis and realizability for our research. We can test and build models from this data so that we can use ML to make predictions.'\n",
        "\n",
        "# List creation\n",
        "\n",
        "q5_B_answers_list = [q5_B_answer_1, q5_B_answer_2]"
      ],
      "metadata": {
        "id": "r0_Tt0lspkQ-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# C-level answers\n",
        "\n",
        "q5_C_answer_1 = 'One example where ML would work in is speech recognition, converting speech into text.'\n",
        "q5_C_answer_2 = 'ML may work well with education industry. Teachers can utilize ML for lesson preparation, such as analyze academic data, recommendation of high-quality teaching resources, on-demand generation of teaching plans. Others might be like intelligent teaching assistants instead of an actual person TA, or dynamically adjust the teaching content based on comprehensive factors such as learning situation analysis and learner style, and student feedback.'\n",
        "\n",
        "# List creation\n",
        "\n",
        "q5_C_answers_list = [q5_C_answer_1, q5_C_answer_2]"
      ],
      "metadata": {
        "id": "oRlId1mtp46L"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "q5_Q_and_A = [\"Question:\", q5, \"A-level Answers\", q5_A_answers_list, \"B-level Answers\", q5_B_answers_list, \"C-level Answers\", q5_C_answers_list]"
      ],
      "metadata": {
        "id": "XJC2UNxSqHBp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "from google.colab import files\n",
        "\n",
        "def save_example_answers(examples, filename='wiki_ABC_Q5examples.json'):\n",
        "  with open(filename, 'w') as file:\n",
        "    json.dump(examples, file)\n",
        "  files.download(filename)\n",
        "\n",
        "save_example_answers(q5_Q_and_A)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "3e565b3d-804c-4b5e-acc8-efeb955c6c14",
        "id": "MRcW9_1MqPG-"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_8ec6f24c-9653-4335-8c1b-766926434399\", \"wiki_ABC_Q2examples.json\", 5931)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " # Exploring with Langchain Example Sets, FewShotPrompts and Example Selectors\n",
        " Here we're exploring another avenue for this goal to see if we can yield better results than the vector stores. Again, we are using question 2 from the Wiki to test."
      ],
      "metadata": {
        "id": "PpKRr5Vw4_0F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Context: question\n",
        "\n",
        "q2 = 'Question 2: Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses).'"
      ],
      "metadata": {
        "id": "-kmMFUaLs_Q1"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain.prompts.few_shot import FewShotPromptTemplate\n",
        "from langchain.prompts.prompt import PromptTemplate\n",
        "\n",
        "examples = [\n",
        "  {\n",
        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {q2_A_answer_1}.\"\"\",\n",
        "    \"answer\": \"This student should recieve an A-level grade.\"\n",
        "  },\n",
        "  {\n",
        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {q2_A_answer_2}.\"\"\",\n",
        "    \"answer\": \"The student should recieve an A-level grade.\"\n",
        "  },\n",
        "  {\n",
        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {q2_B_answer_1}.\"\"\",\n",
        "    \"answer\": \"This student should recieve a B-level grade.\"\n",
        "  },\n",
        "  {\n",
        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {q2_B_answer_2}.\"\"\",\n",
        "    \"answer\": \"This student should recieve a B-level grade.\"\n",
        "  },\n",
        "  {\n",
        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {q2_B_answer_3}.\"\"\",\n",
        "    \"answer\": \"This student should recieve a B-level grade.\"\n",
        "  },\n",
        "  {\n",
        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {q2_C_answer_1}.\"\"\",\n",
        "    \"answer\": \"This student should recieve a C-level grade.\"\n",
        "  },\n",
        "  {\n",
        "    \"question\": f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {q2_C_answer_2}.\"\"\",\n",
        "    \"answer\": \"This student should recieve a C-level grade.\"\n",
        "  }\n",
        "]"
      ],
      "metadata": {
        "id": "1idnQLYW-697"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Example selector\n",
        "from langchain.prompts.example_selector import SemanticSimilarityExampleSelector, MaxMarginalRelevanceExampleSelector, NGramOverlapExampleSelector\n",
        "from langchain.vectorstores import Chroma\n",
        "from langchain.embeddings import OpenAIEmbeddings\n",
        "\n",
        "\n",
        "example_selector_semantic = SemanticSimilarityExampleSelector.from_examples(\n",
        "    # This is the list of examples available to select from.\n",
        "    examples,\n",
        "    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.\n",
        "    OpenAIEmbeddings(),\n",
        "    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.\n",
        "    Chroma,\n",
        "    # This is the number of examples to produce.\n",
        "    k=1\n",
        ")\n",
        "\n",
        "example_selector_mmr = MaxMarginalRelevanceExampleSelector.from_examples(\n",
        "    # This is the list of examples available to select from.\n",
        "    examples,\n",
        "    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.\n",
        "    OpenAIEmbeddings(),\n",
        "    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.\n",
        "    Chroma,\n",
        "    # This is the number of examples to produce.\n",
        "    k=1,\n",
        ")\n",
        "\n",
        "example_selector_ngram = NGramOverlapExampleSelector(\n",
        "    # These are the examples it has available to choose from.\n",
        "    examples=examples,\n",
        "    # This is the PromptTemplate being used to format the examples.\n",
        "    example_prompt=example_prompt,\n",
        "    # This is the threshold, at which selector stops.\n",
        "    # It is set to -1.0 by default.\n",
        "    threshold=-1.0,\n",
        "    # For negative threshold:\n",
        "    # Selector sorts examples by ngram overlap score, and excludes none.\n",
        "    # For threshold greater than 1.0:\n",
        "    # Selector excludes all examples, and returns an empty list.\n",
        "    # For threshold equal to 0.0:\n",
        "    # Selector sorts examples by ngram overlap score,\n",
        "    # and excludes those with no ngram overlap with input.\n",
        ")\n",
        "\n"
      ],
      "metadata": {
        "id": "Y7vNiHKhuIjm"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "example_prompt = PromptTemplate(input_variables=[\"question\", \"answer\"], template=\"Question: {question}\\n{answer}\")\n",
        "\n",
        "print(example_prompt.format(**examples[0]))"
      ],
      "metadata": {
        "id": "x2dCmQdACwGi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prompt = FewShotPromptTemplate(\n",
        "    #examples=examples,\n",
        "    example_selector = example_selector_semantic,\n",
        "    example_prompt = example_prompt,\n",
        "    suffix=\"Question: {input}\",\n",
        "    input_variables=[\"input\"]\n",
        ")\n",
        "\n",
        "print(prompt.format(input=f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {example1}.\"\"\"))"
      ],
      "metadata": {
        "id": "Uc_BvY0rBZgx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prompt_semantic = FewShotPromptTemplate(\n",
        "    #examples=examples,\n",
        "    example_selector = example_selector_semantic,\n",
        "    example_prompt = example_prompt,\n",
        "    suffix=\"Question: {input}\",\n",
        "    input_variables=[\"input\"]\n",
        ")\n",
        "\n",
        "prompt_mmr = FewShotPromptTemplate(\n",
        "    #examples=examples,\n",
        "    example_selector = example_selector_mmr,\n",
        "    example_prompt = example_prompt,\n",
        "    suffix=\"Question: {input}\",\n",
        "    input_variables=[\"input\"]\n",
        ")\n",
        "\n",
        "prompt_ngram = FewShotPromptTemplate(\n",
        "    #examples=examples,\n",
        "    example_selector = example_selector_ngram,\n",
        "    example_prompt = example_prompt,\n",
        "    suffix=\"Question: {input}\",\n",
        "    input_variables=[\"input\"]\n",
        ")"
      ],
      "metadata": {
        "id": "k1laCoAPxhcH"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# First using the Example Set to test without an example selector, just the base OpenAI model.\n",
        "from langchain.llms import OpenAI\n",
        "base_llm = OpenAI()\n",
        "base_llm(prompt.format(input=f\"\"\" Please grade the following students' answers to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: student 1: {example1}, student 2: {example2}, and student 3: {example3}. Grade based on the following advice from the professor:\n",
        "     One aspect of it was being specific. The poor answers (C) have a lot of platitudes, the better answers give\n",
        "     specific examples (B and A). Secondly, they should discuss automation and/or prediction specifically. Those are\n",
        "     the things that ML does (A and B answers), it is not 'technology' broadly (what a C answer might say).\n",
        "     The difference between A and B answers should be in their complexity, specificity, thoughfulness\n",
        "     , and general insight on the subject. However, keep in mind that these are short answer questions.\n",
        "     The number of examples or the legnth of the response is not as important as the content itself. After assigning your grade, provide a brief\n",
        "     explanation on your reasoning for assigning the grade based on the professor's advice, and\n",
        "     what differentiated the answers from the higher grade level (if applicable). There should be three grades assigned in your response, and assign based on the order the\n",
        "     answers were passed in.\"\"\"))"
      ],
      "metadata": {
        "id": "rUdx2axQuec1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Example Selector 1:\n",
        "\n",
        "base_llm(prompt_mmr.format(input=f\"\"\" Please grade the following students' answers to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: student 1: {example1}, student 2: {example2}, and student 3: {example3}. Grade based on the following advice from the professor:\n",
        "     One aspect of it was being specific. The poor answers (C) have a lot of platitudes, the better answers give\n",
        "     specific examples (B and A). Secondly, they should discuss automation and/or prediction specifically. Those are\n",
        "     the things that ML does (A and B answers), it is not 'technology' broadly (what a C answer might say).\n",
        "     The difference between A and B answers should be in their complexity, specificity, thoughfulness\n",
        "     , and general insight on the subject. However, keep in mind that these are short answer questions.\n",
        "     The number of examples or the legnth of the response is not as important as the content itself. After assigning your grade, provide a brief\n",
        "     explanation on your reasoning for assigning the grade based on the professor's advice, and\n",
        "     what differentiated the answers from the higher grade level (if applicable). There should be three grades assigned in your response, and assign based on the order the\n",
        "     answers were passed in.\"\"\"))"
      ],
      "metadata": {
        "id": "fe24IOslyITA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Example selector 2:\n",
        "\n",
        "base_llm(prompt_ngram.format(input=f\"\"\" Please grade the following students' answers to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: student 1: {example1}, student 2: {example2}, and student 3: {example3}. Grade based on the following advice from the professor:\n",
        "     One aspect of it was being specific. The poor answers (C) have a lot of platitudes, the better answers give\n",
        "     specific examples (B and A). Secondly, they should discuss automation and/or prediction specifically. Those are\n",
        "     the things that ML does (A and B answers), it is not 'technology' broadly (what a C answer might say).\n",
        "     The difference between A and B answers should be in their complexity, specificity, thoughfulness\n",
        "     , and general insight on the subject. However, keep in mind that these are short answer questions.\n",
        "     The number of examples or the legnth of the response is not as important as the content itself. After assigning your grade, provide a brief\n",
        "     explanation on your reasoning for assigning the grade based on the professor's advice, and\n",
        "     what differentiated the answers from the higher grade level (if applicable). There should be three grades assigned in your response, and assign based on the order the\n",
        "     answers were passed in.\"\"\"))"
      ],
      "metadata": {
        "id": "dXwJa30wyL4S"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Results from the ngram example selector with the following prompt (yielded the best results (relatively):\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "input=f\"\"\" Please grade the following student's answer to the question ({q2})\n",
        "     on an A-, B-, C-level grading scale: {example3}. Grade based on the following advice from the professor:\n",
        "     One aspect of it was being specific. The poor answers have a lot of platitudes, the better answers give\n",
        "     specific examples. Secondly, they should discuss automation and/or prediction specifically. Those are\n",
        "     the things that ML does, it is not 'technology' broadly. The best answers\n",
        "     (A) should be more complex and specific in answering the question and meeting that criteria\n",
        "     than the better answers (B), though length may not be a determinant. After assigning your grade, provide a brief\n",
        "     explanation on your reasoning for assigning the grade based on the professor's advice.\"\"\"\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "1. Example 1 (A-level grade): \"This student should recieve a B-level grade. The student's answer includes specific examples of how machine learning can be used in business to increase efficiency and profit margins, as well as how machine learning can automate processes previously done by humans. The student also mentions the importance of prediction in machine learning, which is one of its core functions.\" (incorrect)\n",
        "2. Example 2 (B-level grade): \"This student should recieve a B-level grade. The student's answer provides specific examples of how ML can be used in business such as NLP and predictive analysis. The student also references automation and prediction which are key aspects of ML.\" (correct)\n",
        "3. Example 3 (C-level grade): \"This student should recieve a B-level grade. The student's answer provides a specific example about how machine learning can help businesses save time and money by providing more accurate predictions. It also mentions the three main innovations that machine learning brings to the table (prediction is cheap, more accurate, and automated).\" (incorrect)"
      ],
      "metadata": {
        "id": "bKqtzH8hcoHD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Exploring Chat Prompt Templates and System/Human Messages\n",
        "This section is unfinalized as we are not sure if it is neccessary for the goals of this project, and we ran into errors which we were unsuccessful in debugging. This approach may yield better results but would need to be further devloped and debugged."
      ],
      "metadata": {
        "id": "YGWq7cw697Wv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain import PromptTemplate, LLMChain\n",
        "from langchain.prompts.chat import (\n",
        "    ChatPromptTemplate,\n",
        "    SystemMessagePromptTemplate,\n",
        "    AIMessagePromptTemplate,\n",
        "    HumanMessagePromptTemplate,\n",
        ")\n",
        "from langchain.schema import AIMessage, HumanMessage, SystemMessage\n",
        "\n",
        "\n",
        "\n",
        "from langchain.prompts import (\n",
        "    ChatPromptTemplate,\n",
        "    PromptTemplate,\n",
        "    SystemMessagePromptTemplate,\n",
        "    AIMessagePromptTemplate,\n",
        "    HumanMessagePromptTemplate,\n",
        ")\n",
        "from langchain.schema import (\n",
        "    AIMessage,\n",
        "    HumanMessage,\n",
        "    SystemMessage\n",
        ")"
      ],
      "metadata": {
        "id": "NzK7Yo_OEqlG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "template=\"You are a helpful assistant that grades student responses ({answer}) to questions ({question}) about machine learning.\"\n",
        "system_message_prompt = SystemMessagePromptTemplate.from_template(template)\n",
        "human_template=\"{text}\"\n",
        "human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)"
      ],
      "metadata": {
        "id": "XEsYhpKMFcud"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "fprompt=PromptTemplate(\n",
        "    template=\"You are a helpful assistant that grades student responses ({answer}) to questions ({question}) about machine learning.\",\n",
        "    input_variables=[\"question\", \"answer\"],\n",
        ")\n",
        "system_message_prompt = SystemMessagePromptTemplate(prompt=prompt)"
      ],
      "metadata": {
        "id": "En5SM-ZHCT8l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])\n",
        "\n",
        "# get a chat completion from the formatted messages\n",
        "chat_prompt.format_prompt(input = [q2, example1], text= \"The student should recieve an A-level grade.\" ).to_messages()"
      ],
      "metadata": {
        "id": "W2zIxXxDFNIh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "example_human = SystemMessagePromptTemplate.from_template(\n",
        "    \"Machine learning is awesome\", additional_kwargs={\"name\": \"example_user\"}\n",
        ")\n",
        "example_ai = SystemMessagePromptTemplate.from_template(\n",
        "    \"This student should reieve a C-level grade.\", additional_kwargs={\"name\": \"example_assistant\"}\n",
        ")\n",
        "human_template = \"{text}\"\n",
        "human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)"
      ],
      "metadata": {
        "id": "b_r2Y7YXTa0R"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "chat = ChatOpenAI(temperature=0)"
      ],
      "metadata": {
        "id": "R2XA1z27cl6V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "chat_prompt = ChatPromptTemplate.from_messages(\n",
        "    [system_message_prompt, example_human, example_ai, human_message_prompt]\n",
        ")\n",
        "chain = LLMChain(llm=chat, prompt=chat_prompt)\n",
        "# get a chat completion from the formatted messages\n",
        "text_dict = {'text': 'Please grade the following students response to question 2 based on the A-, B-, C-level grading scale'}\n",
        "chain.run(text_dict)"
      ],
      "metadata": {
        "id": "3WNMa70jVuf5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "query = f\"\"\" Please grade the following students answer: {example1} to the question ({q2}).\n",
        "The uploaded json should serve as as examples of A, B, and C level answers. In the document, the\n",
        "original question is printed, as well as examples of previous student answers that have recieved\n",
        "A, B, and C grades (labeled accordingly)\"\"\"\n",
        "grades = qa.run(query)\n",
        "print(grades)"
      ],
      "metadata": {
        "id": "naKuTxKa2-U8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Conclusions based on Question 2\n",
        "\n",
        "Using different types of chains and example selectors made minor differences, but the results were still not entirely correct, and also inconsistent. Using a persona prompt as the system message was semi-useful, as was adding the grading criteria to the prompt. But it did not remedy our problem.\n",
        "\n",
        "Moving forward:\n",
        "\n",
        "1. Grading criteria helped, but for longer/more complex questions, more context may be needed (i.e., more specifics on what seperates an A from B, B from C, etc. The model did not perform as well in distinguishing between these levels/did not seem to understand the nuances of the grading criteria).\n",
        "2. The model would likely perform better with a better example set/more context to be trained on.\n",
        "\n"
      ],
      "metadata": {
        "id": "C7kDC2pe7bNd"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.10.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}