{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "view-in-github" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "S4MkldrwPA_S" }, "source": [ "# LLMs for Self-Study\n", "> A prompt and code template for better understanding texts\n", "\n", "This notebook provides a guide for using LLMs for self-study programmatically. A number of prompt templates are provided to assist with generating great assessments for self-study, and code is additionally provided for fast usage. This notebook is best leveraged for a set of documents (text or PDF preferred) **to be uploaded** for interaction with the model.\n", "\n", "This version of the notebook is best suited for those who prefer to use files from their local drive as context rather than copy and pasting directly into the notebook to be used as context for the model. If you prefer to copy and paste text, you should direct yourself to the [prompt_with_context](https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/prompt_with_context.ipynb) notebook." ] }, { "cell_type": "markdown", "metadata": { "id": "6Gf-_aZVPA_S" }, "source": [ "# Code Setup\n", "Run the following cells to setup the rest of the environment for prompting. In the following section, we set up the computational environment with imported code, setup your API key access to OpenAI, and loading access to your language model. Note that the following cells may take a long time to run." ] }, { "cell_type": "markdown", "metadata": { "id": "6040J5eXPA_T" }, "source": [ "## Library installation and loading\n", "The following `pip install` code should be run if you're using Google Colab, or otherwise do not have a computational environment (e.g., _venv_, _conda virtual environment_, _Docker, Singularity, or other container_) with these packages installed." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "7_XCtEMbPA_T", "outputId": "aa36bcf0-1d11-4947-f96b-d28c67d4e431" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "# run this code if you're using Google Colab or don't have these packages installed in your computing environment\n", "! pip install -q langchain openai gradio numpy tiktoken" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "6fdfMar8PA_U" }, "outputs": [], "source": [ "# import required libraries\n", "import numpy as np\n", "import getpass\n", "import os\n", "from langchain.chat_models import ChatOpenAI\n", "from langchain.chains import RetrievalQA\n", "from langchain.schema import SystemMessage, HumanMessage, AIMessage" ] }, { "cell_type": "markdown", "metadata": { "id": "5iI75yjaPA_V" }, "source": [ "## API and model setup\n", "\n", "Use these cells to load the API keys required for this notebook and create a basic OpenAI LLM model. The code below uses the variable you created above when you input your API Key." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "8IgmjGEFPA_V", "outputId": "5ac82647-2f93-4bd3-9284-e18dd04b1c81" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "··········\n" ] } ], "source": [ "# Set up OpenAI API Key\n", "openai_api_key = getpass.getpass()\n", "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n", "\n", "llm = ChatOpenAI(model='gpt-3.5-turbo-16k')\n", "messages = [\n", " SystemMessage(content=\"You are a world-class tutor helping students to perform better on oral and written exams though interactive experiences.\"),\n", " HumanMessage(content=\"\")\n", "]\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Zlx1_Y8rYGn3" }, "source": [ "# Add your context and assign the prefix to your query.\n", "The query assigned here serves as an example." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "g_CAvacjEuCd" }, "outputs": [], "source": [ "context = \"\"\" Two roads diverged in a yellow wood,\n", "And sorry I could not travel both\n", "And be one traveler, long I stood\n", "And looked down one as far as I could\n", "To where it bent in the undergrowth;\n", "Then took the other, as just as fair,\n", "And having perhaps the better claim,\n", "Because it was grassy and wanted wear;\n", "Though as for that the passing there\n", "Had worn them really about the same,\n", "And both that morning equally lay\n", "In leaves no step had trodden black.\n", "Oh, I kept the first for another day!\n", "Yet knowing how way leads on to way,\n", "I doubted if I should ever come back.\n", "I shall be telling this with a sigh\n", "Somewhere ages and ages hence:\n", "Two roads diverged in a wood, and I—\n", "I took the one less traveled by,\n", "And that has made all the difference.\n", "—-Robert Frost—-\n", "Education Place: http://www.eduplace.com \"\"\"\n", "\n", "# Query prefix\n", "query_prefix = \"The following text should be used as the basis for the instructions which follow: \" + context + '\\n'\n", "\n", "# Query\n", "query = \"\"\"Please design a 5 question quiz about Robert Frost's \"Road Not Taken\" which reflects the learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes. The questions should be multiple choice.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response.\n", "If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional\n", "chances to respond until I get the correct choice. Explain why the correct choice is right. \"\"\"" ] }, { "cell_type": "markdown", "metadata": { "id": "hLMJRXc8PA_W" }, "source": [ "# A guide to prompting for self-study\n", "In this section, we provide a number of different approaches for using AI to help you assess and explain the knowledge of your document. Start by interacting with the model and then try out the rest of the prompts!" ] }, { "cell_type": "markdown", "metadata": { "id": "jNBwByBFaJVP" }, "source": [ "## Interact with the model\n", "\n", "Now that your vector store is created, you can begin interacting with the model! Below, we have a comprehensive list of examples using different question types, but feel free to use this code block to experiment with the model.\n", "\n", "Input your prompt into the empty string in the code cell. See example below:\n", "\n", "\n", "\n", "```\n", "query = 'Your Prompt Here'\n", "```\n", "\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "3fPd2b3yUZic" }, "outputs": [], "source": [ "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cskXB5p5DupT" }, "outputs": [], "source": [ "query = 'Your Prompt Here'\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "FLy4qDuWclWF" }, "source": [ "### Our example using query from cell 103\n", "\n", "Run the following code to see a simple example using the prompt defined in an earlier cell (#103)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4QBBlJI-adPf", "outputId": "0d421ad5-74ca-40eb-bd68-afcb47768275" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: Who is the narrator of the poem \"Road Not Taken\"?\n", "\n", "a) Robert Frost\n", "b) A traveler\n", "c) The poet\\'s friend\n", "d) The reader\n" ] } ], "source": [ "# Experiment with interacting with the model by inputting your own prompts into the empty string below.\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "KYro6H82bENS" }, "source": [ "## Types of Questions and Prompts\n", "\n", "Below is a comprehensive list of question types and prompt templates designed by our team. There are also example code blocks, where you can see how the model performed with the example and try it for yourself using the prompt template." ] }, { "cell_type": "markdown", "metadata": { "id": "WAUSc7qJPA_X" }, "source": [ "### Multiple Choice\n", "\n", "Prompt: The following text should be used as the basis for the instructions which follow: {context}. Please design a {number of questions} question quiz about {name or reference to context} which reflects the learning objectives: {list of learning objectives}. The questions should be multiple choice. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "nFMRDL4cPA_X", "outputId": "b341fcb8-49df-43e6-9810-7ff3f232949e" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: Who is the narrator of the poem \"Road Not Taken\"?\n", "\n", "A) Robert Frost\n", "B) The reader\n", "C) The traveler\n", "D) The poet\n" ] } ], "source": [ "# Multiple choice code example\n", "query = \"\"\"Please design a 5 question quiz about Robert Frost's \"Road Not Taken\" which reflects the learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes. The questions should be multiple choice.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response\n", "If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional\n", "chances to respond until I get the correct choice. Explain why the correct choice is right. \"\"\"\n", "\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "WkSIU94GPA_Y" }, "source": [ "### Short Answer\n", "\n", "Prompt: Please design a {number of questions} question quiz about {context} which reflects the learning objectives: {list of learning objectives}. The questions should be short answer. Expect the correct answers to be {anticipated length} long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "enC8ydfEPA_Y", "outputId": "639efa60-3304-4b63-c0e8-a975d2d7dfaf" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: Who is the narrator of the poem \"Road Not Taken\"?\n" ] } ], "source": [ "# Short answer code example\n", "query = \"\"\" Please design a 5-question quiz about Robert Frost's\n", "\"Road Not Taken\" which reflects the learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "The questions should be short answer. Expect the correct answers to be\n", "1-2 sentences long.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response\n", "If I get any part of the answer wrong,\n", "provide me with an explanation of why it was incorrect,\n", "and then give me additional chances to respond until I get the correct choice. \"\"\"\n", "\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "Ta4A527kPA_Y" }, "source": [ "### Fill-in-the-blank\n", "\n", "Prompt: Create a {number of questions} question fill in the blank quiz refrencing {context}. The quiz should reflect the learning objectives: {learning objectives}. The \"blank\" part of the question should appear as \"________\". The answers should reflect what word(s) should go in the blank an accurate statement.\n", "\n", "An example is the follow: \"The author of the book is \"________.\"\n", "\n", "The question should be a statement. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "szMOxEoiPA_Z", "outputId": "0c4c823b-6f07-4ba0-cbf3-54b3e28d0d6c" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: In \"The Road Not Taken,\" the narrator comes to a point where two roads ____________ in a yellow wood.\n" ] } ], "source": [ "# Fill in the blank code example\n", "query = \"\"\" Create a 5 question fill in the blank quiz refrencing Robert Frost's \"The Road Not Taken.\"\n", "The \"blank\" part of the question should appear as \"________\". The answers should reflect what word(s) should go in the blank an accurate statement.\n", "An example is the follow: \"The author of the book is ______.\" The question should be a statement.\n", "The quiz should reflect the learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response\n", "If I answer incorrectly, please explain why my answer is incorrect. \"\"\"\n", "\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "yEwzAB28PA_Z" }, "source": [ "### Sequencing\n", "\n", "Prompt: Please develop a {number of questions} question questionnaire that will ask me to recall the steps involved in the following learning objectives in regard to {context}: {learning objectives}. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. After I respond, explain their sequence to me." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3YRyhhWtPA_Z", "outputId": "7e26ed3b-99c7-4577-c808-fb85c3e11b4c" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: Who is the narrator of the poem?\n" ] } ], "source": [ "# Sequence example\n", "query = \"\"\" Please develop a 5 question questionnaire that will ask me to recall the steps involved in the following learning objectives in regard to Robert Frost's \"The Road Not Taken\":\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response.\n", "After I respond, explain their sequence to me.\"\"\"\n", "\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "0DGBiJofPA_Z" }, "source": [ "### Relationships/drawing connections\n", "\n", "Prompt: Please design a {number of questions} question quiz that asks me to explain the relationships that exist within the following learning objectives, referencing {context}: {learning objectives}. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Mw3RBpsnPA_a", "outputId": "3f721340-7fef-4617-cc2c-2420779ecb2d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: Identify the key elements of the poem: narrator, setting, and underlying message.\n" ] } ], "source": [ "# Relationships example\n", "query = \"\"\" Please design a 5 question quiz that asks me to explain the relationships that exist within the following learning objectives, referencing Robert Frost's \"The Road Not Taken\":\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response.\"\"\"\n", "\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "4YO3wCTwPA_a" }, "source": [ "### Concepts and Definitions\n", "\n", "Prompt: Design a {number of questions} question quiz that asks me about definitions related to the following learning objectives: {learning objectives} - based on {context}\".\n", "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "TbvJPhkmPA_a", "outputId": "f174e720-a31a-4476-8701-36ed9f637770" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: Who is the narrator of the poem \"The Road Not Taken\"?\n", "\n", "Please provide your response.\n" ] } ], "source": [ "# Concepts and definitions example\n", "query = \"\"\" Design a 5 question quiz that asks me about definitions related to the following learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message, and\n", "2. Understand the literary devices used in poetry and their purposes - based on Robert Frost's \"The Road Not Taken\".\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response\n", "Once I write out my response, provide me with your own response, highlighting why my answer is correct or incorrect.\"\"\"\n", "\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "vc-llAgfPA_a" }, "source": [ "### Real Word Examples\n", "\n", "Prompt: Demonstrate how {context} can be applied to solve a real-world problem related to the following learning objectives: {learning objectives}. Ask me questions regarding this theory/concept.\n", "\n", "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "sSwRLkR8PA_a", "outputId": "4d91df15-b7b8-4bc9-d1b2-3af7e629be4f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: Who is the narrator of the poem?\n" ] } ], "source": [ "# Real word example\n", "query = \"\"\" Demonstrate how Robert Frost’s “The Road Not Taken” can be applied to solve a real-world problem. Ask me questions regarding\n", "this theory/concept and relate them to the following learning objectives:\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response\n", "\"\"\"\n", "\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "2y6gHhGKPA_b" }, "source": [ "### Randomized Question Types\n", "\n", "Prompt: Please generate a high-quality assessment consisting of {number of questions} varying questions, each of different types (open-ended, multiple choice, etc.), to determine if I achieved the following learning objectives in regards to {context}: {learning objectives}.\n", "\n", "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "xC4t4WMwPA_b", "outputId": "331b1ef7-ef34-4d21-ddbd-e2ebae1d2fd4" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: \n", "Identify the narrator of the poem \"The Road not Taken\" by Robert Frost.\n", "\n", "Question 2: \n", "What is the setting of the poem \"The Road not Taken\" by Robert Frost?\n", "\n", "Question 3: \n", "What is the underlying message or theme of the poem \"The Road not Taken\" by Robert Frost?\n", "\n", "Question 4 (Multiple Choice):\n", "Which literary device is NOT used in the poem \"The Road not Taken\" by Robert Frost?\n", "a) Metaphor\n", "b) Simile\n", "c) Alliteration\n", "d) Personification\n", "\n", "Question 5 (Open-ended):\n", "Provide an example of a literary device used in the poem \"The Road not Taken\" by Robert Frost and explain its purpose.\n" ] } ], "source": [ "# Randomized question types\n", "query = \"\"\" Please generate a high-quality assessment consisting of 5 varying questions,\n", "each of different types (open-ended, multiple choice, etc.),\n", "to determine if I achieved the following learning objectives in regards to Robert Frost’s “The Road not Taken\":\n", "1. Identify the key elements of the poem: narrator, setting, and underlying message.\n", "2. Understand the literary devices used in poetry and their purposes. If I answer incorrectly for any of the questions,\n", "please explain why my answer is incorrect.\n", "Provide one question at a time, and wait for my response before providing me with feedback.\n", "Again, while the quiz asks for 5 questions, you should\n", "only provide ONE question in you initial response. Do not include the answer in your response.\n", "\"\"\"\n", "\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "ndo_cdWYPA_b" }, "source": [ "### Quantiative evaluation the correctness of a student's answer\n", "\n", "Prompt: (A continuation of the previous chat) Please generate the main points of the student’s answer to the previous question, and evaluate on a scale of 1 to 5 how comprehensive the student’s answer was in relation to the learning objectives, and explain why he or she received this rating, including what was missed in his or her answer if the student’s answer wasn’t complete.\n" ] }, { "cell_type": "code", "execution_count": 118, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WhhNI0FZPA_b", "outputId": "fb037251-2448-44e7-acef-3e6a28da7cb3" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Question 1: Who is the narrator of the poem \"The Road not Taken\"?\n", "a) Robert Frost\n", "b) The traveler\n", "c) The reader\n", "d) The poet\\'s imagination\n" ] } ], "source": [ "# qualitative evaluation\n", "qualitative_query = \"\"\" Please generate the main points of the student’s answer to the previous question,\n", " and evaluate on a scale of 1 to 5 how comprehensive the student’s answer was in relation to the learning objectives,\n", " and explain why he or she received this rating, including what was missed in his or her answer if the student’s answer wasn’t complete.\"\"\"\n", "\n", "# Note that this uses the previous result and query in the context\n", "def prompt_define(query, prefix, context):\n", " prompt = (query + prefix)\n", " prompt = prompt + context\n", " return prompt\n", "\n", "prompt = prompt_define(query, query_prefix, context)\n", "\n", "def get_result(prompt):\n", " messages[1] = HumanMessage(content=prompt)\n", " result = llm(messages)\n", " str_result = str(result)\n", " import re\n", " str_result = str_result.replace(r'\\n', '\\n')\n", " str_result = str_result.replace(\"content='\", \"\")\n", " str_result = str_result.replace(\"' additional_kwargs={} example=False\", \"\")\n", " return str_result\n", "\n", "result = get_result(prompt)\n", "\n", "print(result)" ] } ], "metadata": { "colab": { "include_colab_link": true, "provenance": [] }, "kernelspec": { "display_name": "Python 3.11.5 64-bit", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" }, "vscode": { "interpreter": { "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e" } } }, "nbformat": 4, "nbformat_minor": 0 }