Spaces:
Configuration error
Configuration error
File size: 50,578 Bytes
28e2c84 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 |
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/124-implement-baseline-functionality-for-oral-exam-module/UI_design_oral_exam_chatbot.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PIbogPXyM0wr"
},
"source": [
"# Project IO Achievement - UI Design (Oral Exam)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "x_Vp8SiKM4p1"
},
"source": [
"## Problem Definition\n",
"\n",
"The v1 functionality for the Oral Exam module requires the following:\n",
"\n",
"1. Upload or generation of questions: either the user should upload a set of questions or we should allow the model to generate the questions. The user should pick or it should be inherent if there is no upload of questions. Note that we must also allow for context to be uploaded (vector store, vector store link, specific documents)\n",
"2. The model should prompt the user with a question and pause.\n",
"The user should respond by audio.\n",
"3. This should continue on until some final point where the exam is over.\n",
"\n",
"Then:\n",
"\n",
"1. We should use Whisper to do the transcription, and\n",
"2. Send the transcription, questions, and context for GPT4 for evaluation\n",
"Return the evaluation.\n",
"3. This will primarily be work on a user interface."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o_60X8H3NEne"
},
"source": [
"## Libraries\n",
"\n",
"This section will install and import some important libraries such as Langchain, openai, Gradio, and so on"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "pxcqXgg2aAN7"
},
"outputs": [],
"source": [
"# install libraries here\n",
"# -q flag for \"quiet\" install\n",
"%%capture\n",
"!pip install -q langchain\n",
"!pip install -q openai\n",
"!pip install -q gradio\n",
"# !pip install -q datasets\n",
"!pip install -q torchaudio\n",
"!pip install -q git+https://github.com/openai/whisper.git\n",
"!pip install -q docx\n",
"!pip install -q PyPDF2\n",
"!pip install -q python-docx"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "pEjM1tLsMZBq"
},
"outputs": [],
"source": [
"# import libraries here\n",
"from langchain.llms import OpenAI\n",
"from langchain.prompts import PromptTemplate\n",
"from langchain.document_loaders import TextLoader\n",
"from langchain.indexes import VectorstoreIndexCreator\n",
"from langchain import ConversationChain, LLMChain, PromptTemplate\n",
"from langchain.chat_models import ChatOpenAI\n",
"from langchain.memory import ConversationBufferWindowMemory\n",
"from langchain.prompts import ChatPromptTemplate\n",
"from langchain.text_splitter import CharacterTextSplitter\n",
"from langchain.embeddings import OpenAIEmbeddings\n",
"import openai\n",
"import os\n",
"from getpass import getpass\n",
"# from IPython.display import display, Javascript, HTML\n",
"# from google.colab.output import eval_js\n",
"# from base64 import b64decode\n",
"# import ipywidgets as widgets\n",
"# from IPython.display import clear_output\n",
"import time\n",
"import requests\n",
"# from datasets import load_dataset\n",
"# from torchaudio.transforms import Resample\n",
"import whisper\n",
"import numpy as np\n",
"import torch\n",
"import librosa\n",
"# from datasets import load_dataset\n",
"#from jiwer import wer\n",
"import pandas as pd\n",
"import gradio as gr\n",
"from docx import Document\n",
"import PyPDF2\n",
"from pydub import AudioSegment\n",
"import tempfile"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "03KLZGI_a5W5"
},
"source": [
"## API Keys\n",
"\n",
"Use these cells to load the API keys required for this notebook. The below code cell uses the `getpass` library."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5smcWj4DbFgy",
"outputId": "6bc91507-cd3c-4808-8976-811d7fc7cb29"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"··········\n"
]
}
],
"source": [
"openai_api_key = getpass()\n",
"os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
"openai.api_key = openai_api_key"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pMo9x8u4AEV1"
},
"source": [
"## Prompt Design"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "UgnCZRMhADvo",
"outputId": "462e62c7-a618-4549-e651-858514757235"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"ChatOpenAI(cache=None, verbose=False, callbacks=None, callback_manager=None, tags=None, metadata=None, client=<class 'openai.api_resources.chat_completion.ChatCompletion'>, model_name='gpt-4', temperature=0.0, model_kwargs={}, openai_api_key='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', openai_api_base='', openai_organization='', openai_proxy='', request_timeout=None, max_retries=6, streaming=False, n=1, max_tokens=None, tiktoken_model_name=None)"
]
},
"metadata": {},
"execution_count": 4
}
],
"source": [
"chat = ChatOpenAI(temperature=0.0, model_name='gpt-4')\n",
"chat"
]
},
{
"cell_type": "markdown",
"source": [
"### Chatbot Prompts"
],
"metadata": {
"id": "2tTNiyU-ZcDU"
}
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "r-VmK_7vHrmw",
"outputId": "8c314a8f-dad5-47b9-ddba-f6009a73b80d"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['history', 'input', 'instruction', 'questions']"
]
},
"metadata": {},
"execution_count": 15
}
],
"source": [
"template_string3 = \"\"\"\n",
"Please ask me the following questions in sequence, and after I provide the answer, \\\n",
"please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. \\\n",
"Just ask me the question, and please do not show any other text (no need for greetings for example) \\\n",
"Here are the questions that you can will me: {questions}. \\\n",
"Here are the chat history: {history}. \\\n",
"{input}\n",
"\n",
"Once all questions are answered, thank the user and give overall feedback for the question answering part.\n",
"\"\"\"\n",
"prompt_template3 = ChatPromptTemplate.from_template(template_string3)\n",
"prompt_template3.messages[0].prompt.input_variables"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 72
},
"id": "XaK7D5B4bMYv",
"outputId": "55f495b5-d8d0-4a80-b5c4-128dba34eebe"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'\\nPlease ask me the following questions in sequence, and after I provide the answer, please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. Just ask me the question, and please do not show any other text (no need for greetings for example) Here are the questions that you can will me: {questions}. Here are the chat history: {history}. {input}\\n\\nOnce all questions are answered, thank the user and give overall feedback for the question answering part.\\n'"
],
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
}
},
"metadata": {},
"execution_count": 16
}
],
"source": [
"prompt_template3.messages[0].prompt.template"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l4o8R5eUE1n8"
},
"source": [
"### Functions"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "ABN0X9xQHeii"
},
"outputs": [],
"source": [
"def embed_key(openai_api_key):\n",
" os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
"\n",
"def transcribe(audio_file_path):\n",
" try:\n",
" with open(audio_file_path, \"rb\") as audio_file:\n",
" # Call OpenAI's Whisper model for transcription\n",
" transcript = openai.Audio.transcribe(\"whisper-1\", audio_file)\n",
" transcribed_text = transcript[\"text\"]\n",
" return transcribed_text\n",
" except:\n",
" return \"Your answer will be transcribed here\"\n",
"\n",
"def process_file(files):\n",
" for file in files:\n",
" try:\n",
" extension = file.name.split('.')[-1].lower()\n",
" if extension == 'docx':\n",
" doc = Document(file.name)\n",
" full_text = []\n",
" for paragraph in doc.paragraphs:\n",
" full_text.append(paragraph.text)\n",
" return '\\n'.join(full_text)\n",
"\n",
" elif extension == 'pdf':\n",
" pdf_file = open(file.name, 'rb')\n",
" reader = PyPDF2.PdfReader(pdf_file)\n",
" num_pages = len(reader.pages)\n",
" full_text = []\n",
" for page in range(num_pages):\n",
" page_obj = reader.pages[page]\n",
" full_text.append(page_obj.extract_text())\n",
" pdf_file.close()\n",
" return '\\n'.join(full_text)\n",
"\n",
" elif extension == 'txt':\n",
" with open(file.name, 'r') as txt_file:\n",
" full_text = txt_file.read()\n",
" return full_text\n",
"\n",
" else:\n",
" return \"Unsupported file type\"\n",
" except FileNotFoundError:\n",
" return \"File not found\"\n",
" except PermissionError:\n",
" return \"Permission denied\"\n",
"\n",
"def generate_questions(text, prompt):\n",
" test_input1 = question_template.format_messages(\n",
" context = text,\n",
" pre_prompt = prompt)\n",
"\n",
" response = chat(test_input1)\n",
" return response.content\n",
"\n",
"\n",
"def ai_evaluate(context, audio_transcript, QA, instructions):\n",
" test_input1 = evaluate_template.format_messages(\n",
" context = context,\n",
" transcript = audio_transcript,\n",
" QA = QA,\n",
" instructions = instructions)\n",
"\n",
" response = chat(test_input1)\n",
" return response.content\n",
"\n",
"def upload_file(files):\n",
" file_paths = [file.name for file in files]\n",
" return file_paths\n",
"\n",
"def use_these_questions(input):\n",
" return input\n",
"\n",
"################################\n",
"\n",
"def add_text(history, text, prompt = template_string3):\n",
" new_history = [(prompt, None)] + history + [(text, None)]\n",
" return new_history, gr.update(value=\"\", interactive=False)\n",
"\n",
"# def add_file(history, file):\n",
"# history = history + [((file.name,), None)]\n",
"# return history\n",
"\n",
"\n",
"def bot_initialize(input, instruction_feedback, questions_used, history):\n",
"\n",
" template_string3 = \"\"\"\n",
" Please ask me the following questions in sequence, and after I provide the answer, \\\n",
" please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. \\\n",
" Here are the questions that you can ask me: {questions}. \\\n",
" Here are the chat history: {history}. \\\n",
" {input} \\\n",
"\n",
" *** Remember, just ask me the question, give feedbacks, and ask the next questions. Do not forget to ask the next question after feedbacks. \\\n",
" \"\"\"\n",
" prompt_template3 = ChatPromptTemplate.from_template(template_string3)\n",
"\n",
" test_input1 = prompt_template3.format_messages(\n",
" instruction = instruction_feedback,\n",
" history = history,\n",
" questions = questions_used,\n",
" input = input)\n",
"\n",
" response = chat(test_input1)\n",
" return response.content\n",
"\n",
"# def initialize(instruction_feedback, questions_used, chat_history, ready):\n",
"# test_input1 = prompt_template3.format_messages(\n",
"# instruction = instruction_feedback,\n",
"# chat_history = chat_history,\n",
"# questions = questions_used,\n",
"# ready = ready)\n",
"# response = chat(test_input1)\n",
"# return response.content\n",
"\n",
"# def bot(history):\n",
"# response = \"**That's cool!**\"\n",
"# history[-1][1] = \"\"\n",
"# for character in response:\n",
"# history[-1][1] += character\n",
"# time.sleep(0.05)\n",
"# yield history\n",
"\n",
"def message_and_history(input, instruction_feedback, questions_used, history):\n",
" history = history or []\n",
" s = list(sum(history, ()))\n",
" s.append(input)\n",
" inp = ' '.join(s)\n",
" output = bot_initialize(inp, instruction_feedback, questions_used, history)\n",
" history.append((input, output))\n",
" return history, history\n",
"\n",
"def prompt_select(selection, number, length):\n",
" if selection == \"Random\":\n",
" prompt = f\"Please design a {number} question quiz based on the context provided and the inputted learning objectives (if applicable).\"\n",
" elif selection == \"Fill in the Blank\":\n",
" prompt = f\"Create a {number} question fill in the blank quiz refrencing the context provided. The quiz should reflect the learning objectives (if inputted). The 'blank' part of the question should appear as '________'. The answers should reflect what word(s) should go in the blank an accurate statement. An example is the follow: 'The author of the article is ______.' The question should be a statement.\"\n",
" elif selection == \"Short Answer\":\n",
" prompt = f\"Please design a {number} question quiz about which reflects the learning objectives (if inputted). The questions should be short answer. Expect the correct answers to be {length} sentences long.\"\n",
" else:\n",
" prompt = f\"Please design a {number} question {selection.lower()} quiz based on the context provided and the inputted learning objectives (if applicable).\"\n",
" return prompt\n",
"\n",
"# def prompt_select(selection, number, length):\n",
"# if selection == \"Random\":\n",
"# prompt = f\"Please design a {number} question quiz based on the context provided and the inputted learning objectives (if applicable). The types of questions should be randomized (including multiple choice, short answer, true/false, short answer, etc.). Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
"# elif selection == \"Fill in the Blank\":\n",
"# prompt = f\"Create a {number} question fill in the blank quiz refrencing the context provided. The quiz should reflect the learning objectives (if inputted). The 'blank' part of the question should appear as '________'. The answers should reflect what word(s) should go in the blank an accurate statement. An example is the follow: 'The author of the article is ______.' The question should be a statement. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
"# elif selection == \"Short Answer\":\n",
"# prompt = f\"Please design a {number} question quiz about which reflects the learning objectives (if inputted). The questions should be short answer. Expect the correct answers to be {length} sentences long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct answer is right.\"\n",
"# else:\n",
"# prompt = f\"Please design a {number} question {selection.lower()} quiz based on the context provided and the inputted learning objectives (if applicable). Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n",
"# return prompt"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8PzIpcfg4-X0"
},
"source": [
"## Integrate Prompts from LO project"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E5vWxcm25EAC"
},
"source": [
"### Creating a Chain for Short Answer Generation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y95FExV-5IqI"
},
"source": [
"In this example, the context would include the poem \"The Road Not Taken\" by Robert Frost"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "j_qEXDWQ5RSW"
},
"outputs": [],
"source": [
"# This is what I used to test the function 'generate_questions_v2'\n",
"template_string = \"\"\"\n",
"You are a world-class tutor helping students to perform better on oral and written exams though interactive experiences.\"\n",
"\n",
"The following text should be used as the basis for the instructions which follow: {context} \\\n",
"\n",
"The following is the guideline for generating the questiion: {pre_prompt} \\\n",
"\n",
"The output should be formatted as following:\n",
"\n",
"Question 1: ...\n",
"Question 2: ...\n",
"Question 3: ...\n",
"...\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "yWOM1XdC5UhQ",
"outputId": "69d781f7-fb0c-4dde-9085-ddb9908b82af"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['context', 'pre_prompt']"
]
},
"metadata": {},
"execution_count": 9
}
],
"source": [
"question_template = ChatPromptTemplate.from_template(template_string)\n",
"question_template.messages[0].prompt.input_variables"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "4Mc1ZC3jaydQ"
},
"outputs": [],
"source": [
"# @title\n",
"con = \"\"\" Two roads diverged in a yellow wood,\n",
"And sorry I could not travel both\n",
"And be one traveler, long I stood\n",
"And looked down one as far as I could\n",
"To where it bent in the undergrowth;\n",
"Then took the other, as just as fair,\n",
"And having perhaps the better claim,\n",
"Because it was grassy and wanted wear;\n",
"Though as for that the passing there\n",
"Had worn them really about the same,\n",
"And both that morning equally lay\n",
"In leaves no step had trodden black.\n",
"Oh, I kept the first for another day!\n",
"Yet knowing how way leads on to way,\n",
"I doubted if I should ever come back.\n",
"I shall be telling this with a sigh\n",
"Somewhere ages and ages hence:\n",
"Two roads diverged in a wood, and I—\n",
"I took the one less traveled by,\n",
"And that has made all the difference.\n",
"—-Robert Frost—-\n",
"Education Place: http://www.eduplace.com \"\"\"\n",
"\n",
"pre = \"Please design a 3 question quiz about which reflects the learning objectives (if inputted). The questions should be short answer. Expect the correct answers to be sentences long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct answer is right.\"\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 36
},
"id": "KXssPFyEbG3f",
"outputId": "8a5389f7-3ec6-4332-d9ae-f45ff5781caf"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'Question 1: What is the main theme of Robert Frost\\'s poem \"The Road Not Taken\"?'"
],
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
}
},
"metadata": {},
"execution_count": 11
}
],
"source": [
"generate_questions(con,pre)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DMTybR3PVuoC"
},
"source": [
"### Creating a Chain for AI Evaluation"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "Wc-3XAFQVxO_"
},
"outputs": [],
"source": [
"template_evaluation = \"\"\"\n",
"Given\n",
"1. The follwing context of the oral exam/presentation: {context} \\\n",
"\n",
"2. The answer from the student: {transcript} \\\n",
"\n",
"3. The Questions asked to the student and student answers {QA} \\\n",
"\n",
"Please evaluate the students performance based on {instructions} \\\n",
"\n",
"If no instruction is provided, you can evaluate based on your judgement of the students performance.\n",
"\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "FZXeYNSVVy5g",
"outputId": "94a1a4be-03a6-4c20-97b8-4333910991fa"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['QA', 'context', 'instructions', 'transcript']"
]
},
"metadata": {},
"execution_count": 13
}
],
"source": [
"# @title\n",
"evaluate_template = ChatPromptTemplate.from_template(template_evaluation)\n",
"evaluate_template.messages[0].prompt.input_variables"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a3WUL_hFyMkr"
},
"source": [
"### Test process_file"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 157
},
"id": "LJX1AKTMyVm8",
"outputId": "24f8de7a-f456-47bc-b3b5-0284bbd7076f"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"\"\\ufeffHello,\\n\\n\\nWe are so excited for this semester’s partnership with Data Science Institute and Next Steps at Vanderbilt. Jonathan Wade will be interning Mondays and Wednesdays 2-5 pm and Fridays 12-4 pm starting Monday, January 31st. Jessica will be job coaching on Mondays and Wednesdays from 2-5 pm. It also used to be on Fridays from 2-4 pm but not anymore.\\n\\n\\nBelow is important information and reminders:\\n\\n\\nLocation: 1400 18th Ave S Building, Suite 2000, Nashville, TN 37212\\n\\n\\nDress: Business casual attire (This includes items such as dress pants, khakis, polos and dress shirts)\\n\\n\\nImportant Dates:\\n\\n\\nVanderbilt University is off for Spring Break March 5th - March 13th. No internships will take place these days.\\n\\n\\nAll internships end by Thursday, April 28th.\\n\\n\\nImportant COVID-19 Information: Attached is a document outlining the COVID-19 guidelines all Vanderbilt University students must follow, including Next Steps interns and job coaches while at internship sites. Please note these may change given the evolving nature of the pandemic and any changes will be communicated to internship sites, interns, and job coaches as needed.\\n\\n\\nCareer Development Resource Guide: I am also attaching the Next Steps Career Development Resource guide that outlines student expectations and provides helpful information and resources for site supervisors.\\n\\n\\nInternship Coordinator: Lynda Tricia is the coordinator for this internship and is the main point of contact for any questions or concerns that arise.\\n\\n\\nFinally, below you will find everyone's contact information for your convenience.\\n\\n\\nContacts:\\n\\n\\nIntern: Jonathan Wade, [email protected], 613-472-3867\\n\\n\\nSupervisor: Ruben Miller, [email protected], 216-574-3176\\n\\n\\nJob Coach: Jessica Cho, [email protected], 615-999-1134\\n\\n\\nInternship Coordinator: Lynda Tricia, [email protected], 606-415-9999\\n\\n\\nPlease let us know if there are any questions. Thank you!\\n\\n\\n\\n\\nNext Steps at Vanderbilt - Safety Guidelines for Internships\\nMore information on Vanderbilt’s Health and Safety Protocols can be found here: https://www.vanderbilt.edu/coronavirus/community/undergraduate-students/\\nMasks: All Next Steps interns and job coaches must wear masks indoors at internships even if the jobsite does not require masks.\\nInterns and job coaches should have a well-fitted mask that completely covers your nose and mouth, preferably a KN95, KF94 or FFP2 version.\\nLunch Breaks: If an intern or job coach needs a lunch break, they can remove their mask just when they eat or drink. They must be physically distanced from other co-workers when eating. \\nSymptom Monitoring: All interns and job coaches must be free of ANY symptoms related to COVID-19 to go to the internship. If an intern or job coach has symptoms, they should stay home, notify the Next Steps staff and internship supervisor, and get tested at Vanderbilt Student Health or with their medical provider.\\nAccording to the CDC, symptoms may appear 2 to 14 days after exposure to the virus. These include:\\n* Fever or chills\\n* Cough\\n* Shortness of breath or difficulty breathing\\n* Fatigue\\n* Muscle or body aches\\n* Headache\\n* New loss of taste or smell\\n* Sore throat\\n* Congestion or runny nose\\n* Nausea or vomiting\\n* Diarrhea\\n\\n\\nIf an intern or job coach tests positive: If an intern or job coach receives a COVID-19 positive test result, regardless of vaccination status, they should complete the following webform. The webform goes directly to the Vanderbilt Command Center.\\nThe intern or job coach will receive direct communication from the Command Center about their isolation (if they tested positive) or quarantine period (if considered a close contact) and will be instructed to contact Student Health or Occupational Health if they develop symptoms.\\nClose Contact/Quarantine: Interns or job coaches who are a close contact of someone who tests positive should complete the following webform. The webform goes directly to the Command Center.\\n* Close contacts who are unvaccinated will quarantine for 10 days based on CDC guidance.\\no Additional requirements are in place for days 10 to 14 following exposure including:\\n* For days 10 to 14 after last exposure, unvaccinated Vanderbilt community members identified as close contacts must not unmask at any time in public.\\n* Individuals should eat alone or complete any activities alone that require removing a mask in a private space during those four days between day 10-14.\\n* Close contacts who are vaccinated and asymptomatic will not have to quarantine but are recommended to monitor their symptoms and to get a COVID-19 test 5-7 days after last exposure. If asymptomatic, testing can be done at the VU testing center. If individuals develop symptoms, they should test at Student Health, Occupational Health or VUMC or other testing location in the community.\\n* Close contacts who are vaccinated and symptomatic may have to quarantine based on severity of symptoms and specific living situations. This determination will be made by their medical provider in consultation with the Command Center.\\nJob Coaching Supports: If a job coach tests positive and is unable to provide on-site supports, Next Steps staff will follow the procedures outlined below.\\n1. Identify another job coach or Next Steps staff member who can provide job coaching on-site during some of the student’s internship hours.\\n2. Identify another job coach or staff member who can check-in virtually with the student and supervisor during their shift. \\n3. Or, work with the student and supervisor to ensure natural supports are in place if the student must work without support of a job coach during that time period.\""
]
},
"execution_count": 69,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Might need some way to make pdf file to load more readable\n",
"# process_file('/content/instrutor_note.docx')\n",
"# process_file('/content/Big Data & Economics.pdf')\n",
"# process_file('/content/Big Data & Economics.pdf')\n",
"process_file('/content/Anonymized Job Coach QA Test Doc (1).txt')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M6IzVTjz5cex"
},
"source": [
"## UI Design\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "u2SY4Akt_t8h"
},
"source": [
"### Chatbot V2"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 853
},
"id": "6ENnsKlD_uOC",
"outputId": "7b694aa8-e314-4502-a590-3245ebc3e1e0"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Colab notebook detected. This cell will run indefinitely so that you can see errors and logs. To turn off, set debug=False in launch().\n",
"Note: opening Chrome Inspector may crash demo inside Colab notebooks.\n",
"\n",
"To create a public link, set `share=True` in `launch()`.\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.Javascript object>"
],
"application/javascript": [
"(async (port, path, width, height, cache, element) => {\n",
" if (!google.colab.kernel.accessAllowed && !cache) {\n",
" return;\n",
" }\n",
" element.appendChild(document.createTextNode(''));\n",
" const url = await google.colab.kernel.proxyPort(port, {cache});\n",
"\n",
" const external_link = document.createElement('div');\n",
" external_link.innerHTML = `\n",
" <div style=\"font-family: monospace; margin-bottom: 0.5rem\">\n",
" Running on <a href=${new URL(path, url).toString()} target=\"_blank\">\n",
" https://localhost:${port}${path}\n",
" </a>\n",
" </div>\n",
" `;\n",
" element.appendChild(external_link);\n",
"\n",
" const iframe = document.createElement('iframe');\n",
" iframe.src = new URL(path, url).toString();\n",
" iframe.height = height;\n",
" iframe.allow = \"autoplay; camera; microphone; clipboard-read; clipboard-write;\"\n",
" iframe.width = width;\n",
" iframe.style.border = 0;\n",
" element.appendChild(iframe);\n",
" })(7860, \"/\", \"100%\", 500, false, window.element)"
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stderr",
"text": [
"/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
" warnings.warn(warning.format(data.dtype))\n",
"/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
" warnings.warn(warning.format(data.dtype))\n",
"/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
" warnings.warn(warning.format(data.dtype))\n",
"/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
" warnings.warn(warning.format(data.dtype))\n",
"/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n",
" warnings.warn(warning.format(data.dtype))\n"
]
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Keyboard interruption in main thread... closing server.\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'\\nWhat Are the Different Types of Machine Learning?\\nHow Do You Handle Missing or Corrupted Data in a Dataset?\\nHow Can You Choose a Classifier Based on a Training Set Data Size?\\nExplain the Confusion Matrix with Respect to Machine Learning Algorithms.\\nWhat Are the Differences Between Machine Learning and Deep Learning\\n'"
],
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
}
},
"metadata": {},
"execution_count": 17
}
],
"source": [
"with gr.Blocks(theme=gr.themes.Monochrome()) as demo:\n",
" gr.Markdown(\"# Oral Exam App\")\n",
" gr.Markdown(\"## OpenAI API key\")\n",
" with gr.Box():\n",
" gr.HTML(\"\"\"Embed your OpenAI API key below; if you haven't created one already, visit\n",
" platform.openai.com/account/api-keys\n",
" to sign up for an account and get your personal API key\"\"\",\n",
" elem_classes=\"textbox_label\")\n",
" input = gr.Textbox(show_label=False, type=\"password\", container=False,\n",
" placeholder=\"●●●●●●●●●●●●●●●●●\")\n",
" input.change(fn=embed_key, inputs=input, outputs=None)\n",
"\n",
" with gr.Blocks():\n",
" #########################\n",
" #########Context#########\n",
" #########################\n",
" with gr.Accordion(\"Context section\"):\n",
" ### Should also allow vector stores\n",
" gr.Markdown(\"## Please upload the context document(s) for Oral exam\")\n",
" context_input = gr.File(label=\"Click to upload context file\",\n",
" file_count=\"multiple\",\n",
" file_types=[\".txt\", \".docx\", \".pdf\"])\n",
" outputs_context=gr.Textbox(label=\"Context\")\n",
" context_input.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n",
" # upload_button = gr.Button(value=\"Show context\")\n",
" # upload_button.click(process_file, context_input, outputs_context)\n",
"\n",
" with gr.Blocks():\n",
" gr.Markdown(\"\"\"\n",
" ## Generate a Premade Prompt\n",
" Select your type and number of desired questions. Click \"Generate Prompt\" to get your premade prompt,\n",
" and then \"Insert Prompt into Chat\" to copy the text into the chat interface below. \\\n",
" You can also copy the prompt using the icon in the upper right corner and paste directly into the input box when interacting with the model.\n",
" \"\"\")\n",
" with gr.Row():\n",
" with gr.Column():\n",
" question_type = gr.Dropdown([\"Multiple Choice\", \"True or False\", \"Short Answer\", \"Fill in the Blank\", \"Random\"], label=\"Question Type\")\n",
" number_of_questions = gr.Textbox(label=\"Enter desired number of questions\")\n",
" sa_desired_length = gr.Dropdown([\"1-2\", \"3-4\", \"5-6\", \"6 or more\"], label = \"For short answer questions only, choose the desired sentence length for answers. The default value is 1-2 sentences.\")\n",
" with gr.Column():\n",
" prompt_button = gr.Button(\"Generate Prompt\")\n",
" premade_prompt_output = gr.Textbox(label=\"Generated prompt (save or copy)\", show_copy_button=True)\n",
" prompt_button.click(prompt_select,\n",
" inputs=[question_type, number_of_questions, sa_desired_length],\n",
" outputs=premade_prompt_output)\n",
"\n",
" #########################\n",
" #######Main Audio########\n",
" #########################\n",
" with gr.Accordion(\"Main audio section\"):\n",
" gr.Markdown(\"## Upload your audio file or start recording\")\n",
"\n",
" with gr.Column():\n",
" with gr.Row():\n",
" file_input = gr.Audio(label=\"Upload Audio\", source=\"upload\", type=\"filepath\")\n",
" record_inputs = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n",
"\n",
" gr.Markdown(\"## Transcribe the audio uploaded or recorded\")\n",
" outputs_transcribe=gr.Textbox(label=\"Transcription\")\n",
"\n",
" file_input.change(fn=transcribe, inputs=file_input, outputs=outputs_transcribe)\n",
" record_inputs.change(fn=transcribe, inputs=record_inputs, outputs=outputs_transcribe)\n",
"\n",
" # #########################\n",
" # ###Question Generation###\n",
" # #########################\n",
" # with gr.Accordion(\"Question section\"):\n",
" # gr.Markdown(\"## Questions\")\n",
" # with gr.Column():\n",
" # outputs_qa=gr.Textbox(label=\"Generate questions or Use your own questions\")\n",
" # btn3 = gr.Button(value=\"Generate questions\")\n",
" # btn3.click(generate_questions, inputs=context_input, outputs=outputs_qa)\n",
"\n",
"\n",
" ########################\n",
" ##Question Generation###\n",
" ########################\n",
" with gr.Accordion(\"Question section\"):\n",
" gr.Markdown(\"## Questions\")\n",
" with gr.Row():\n",
" with gr.Column():\n",
" outputs_qa=gr.Textbox(label=\"Generate questions or Use your own questions\")\n",
" btn1 = gr.Button(value=\"Generate questions\")\n",
" btn1.click(generate_questions, inputs=[outputs_context, premade_prompt_output], outputs=outputs_qa)\n",
"\n",
" # with gr.Column():\n",
" # submit_question=gr.Textbox(label=\"Use existing questions\")\n",
" # btn4 = gr.Button(value=\"Use these questions\")\n",
" # btn4.click(use_this_question, inputs=outputs_transcribe, outputs=None)\n",
"\n",
"\n",
" #########################\n",
" #######Instruction#######\n",
" #########################\n",
" instruction_qa_input = gr.File(label=\"Click to upload instruction file\",\n",
" file_count=\"multiple\",\n",
" file_types=[\".txt\", \".docx\", \".pdf\"])\n",
" instruction_qa=gr.Textbox(label=\"Or please enter the instruction for question/answering section\")\n",
" instruction_qa.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n",
"\n",
"\n",
" #########################\n",
" #########Audio QA########\n",
" #########################\n",
" with gr.Accordion(\"Audio QA section\"):\n",
" gr.Markdown(\"## Question answering\")\n",
" gr.Markdown(\"### When you are ready to answer questions, press the 'I am ready' button\")\n",
" ##### This may be iterative\n",
" chatbot = gr.Chatbot([],\n",
" elem_id=\"chatbot\",\n",
" height=300)\n",
" state = gr.State()\n",
" message = gr.Textbox(show_label=False,\n",
" placeholder=\"Your answer will be transcribed here\",\n",
" container=False)\n",
" ready_button = gr.Button(value=\"I am ready\")\n",
" ready_button.click(message_and_history, inputs=[message, instruction_qa, outputs_qa, state], outputs=[chatbot, state])\n",
"\n",
" hidden = gr.Textbox(visible = False)\n",
" btn_record = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n",
" btn_record.change(fn=transcribe, inputs=btn_record, outputs=message)\n",
" btn_record.clear(use_these_questions, inputs = hidden, outputs = message)\n",
"\n",
" submit = gr.Button(\"Submit\")\n",
" submit.click(message_and_history,\n",
" inputs=[message, instruction_qa, outputs_qa, state],\n",
" outputs=[chatbot, state])\n",
"\n",
" message_records = gr.Textbox(show_label=False,\n",
" container=False)\n",
" show_records = gr.Button(\"Show QA history\")\n",
" show_records.click(use_these_questions,\n",
" inputs=state,\n",
" outputs=message_records)\n",
"\n",
" #########################\n",
" #######Evaluation########\n",
" #########################\n",
" with gr.Accordion(\"Evaluation section\"):\n",
" gr.Markdown(\"## Evaluation\")\n",
" with gr.Tab(\"General evalution\"):\n",
" evalution=gr.Textbox(label=\"AI Evaluation\")\n",
" btn5 = gr.Button(value=\"Evaluate\")\n",
" btn5.click(ai_evaluate, inputs=[outputs_context, outputs_transcribe, message_records, instruction_qa], outputs=evalution)\n",
" with gr.Tab(\"Quantitative evalution\"):\n",
" table_output = gr.Dataframe(label = \"Some kind of evaluation metrics?\")\n",
" btn6 = gr.Button(value=\"Evaluate\")\n",
" # btn6.click(ai_evaluate, inputs=[outputs_context, message_records, outputs_qa], outputs=table_output)\n",
"\n",
" # demo.launch()\n",
" # demo.launch(share=True)\n",
" demo.launch(debug=True)\n",
"\n",
"'''\n",
"What Are the Different Types of Machine Learning?\n",
"How Do You Handle Missing or Corrupted Data in a Dataset?\n",
"How Can You Choose a Classifier Based on a Training Set Data Size?\n",
"Explain the Confusion Matrix with Respect to Machine Learning Algorithms.\n",
"What Are the Differences Between Machine Learning and Deep Learning\n",
"'''"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "g2EVIogW69Fd"
},
"source": [
"## What's left\n",
"- vector store (link) upload\n",
"- how to not show the warning when transcribing\n",
"- better prompt for evaluation\n",
"- try ChatInterface of Gradio"
]
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "-YwOAtNANrx_"
},
"execution_count": null,
"outputs": []
}
],
"metadata": {
"colab": {
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
} |