{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Install any required libraries" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "!pip install -U -q langchain langchain-openai langchain_core langchain-community langchainhub openai" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "!pip install -qU faiss_cpu pymupdf pandas" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "!pip install -qU ragas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set environment variables" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import os\n", "import openai\n", "from getpass import getpass\n", "\n", "openai.api_key = getpass(\"Please provide your OpenAI Key: \")\n", "os.environ[\"OPENAI_API_KEY\"] = openai.api_key" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Load Data" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from langchain_community.document_loaders import PyMuPDFLoader\n", "\n", "loader = PyMuPDFLoader(\n", " \"input_data/nvidia_10k.pdf\",\n", ")\n", "\n", "documents = loader.load()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'source': 'input_data/nvidia_10k.pdf',\n", " 'file_path': 'input_data/nvidia_10k.pdf',\n", " 'page': 0,\n", " 'total_pages': 96,\n", " 'format': 'PDF 1.4',\n", " 'title': '0001045810-24-000029',\n", " 'author': 'EDGAR® Online LLC, a subsidiary of OTC Markets Group',\n", " 'subject': 'Form 10-K filed on 2024-02-21 for the period ending 2024-01-28',\n", " 'keywords': '0001045810-24-000029; ; 10-K',\n", " 'creator': 'EDGAR Filing HTML Converter',\n", " 'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0',\n", " 'creationDate': \"D:20240221173732-05'00'\",\n", " 'modDate': \"D:20240221173744-05'00'\",\n", " 'trapped': '',\n", " 'encryption': 'Standard V2 R3 128-bit RC4'}" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "documents[0].metadata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Transform - Split documents into chunks" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", "\n", "text_splitter = RecursiveCharacterTextSplitter(\n", " chunk_size = 700,\n", " chunk_overlap = 50\n", ")\n", "\n", "documents = text_splitter.split_documents(documents)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "624" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(documents)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Load OpenAI Embeddings model" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from langchain_openai import OpenAIEmbeddings\n", "\n", "embeddings = OpenAIEmbeddings(\n", " model=\"text-embedding-3-small\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create a FAISS Vector Store" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "from langchain_community.vectorstores import FAISS\n", "\n", "vector_store = FAISS.from_documents(documents, embeddings)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create a Retriever" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "retriever = vector_store.as_retriever()\n", "retrieved_documents = retriever.invoke(\"Who is the E-VP, Operations - and how old are they?\")" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "page_content=\"Debora Shoquist\\n69\\nExecutive Vice President, Operations\\nTimothy S. Teter\\n57\\nExecutive Vice President and General Counsel\\nJen-Hsun Huang co-founded NVIDIA in 1993 and has served as our President, Chief Executive Officer, and a member of the Board of Directors since our\\ninception. From 1985 to 1993, Mr. Huang was employed at LSI Logic Corporation, a computer chip manufacturer, where he held a variety of positions including\\nas Director of Coreware, the business unit responsible for LSI's SOC. From 1983 to 1985, Mr. Huang was a microprocessor designer for AMD, a semiconductor\\ncompany. Mr. Huang holds a B.S.E.E. degree from Oregon State University and an M.S.E.E. degree from Stanford University.\" metadata={'source': 'input_data/nvidia_10k.pdf', 'file_path': 'input_data/nvidia_10k.pdf', 'page': 11, 'total_pages': 96, 'format': 'PDF 1.4', 'title': '0001045810-24-000029', 'author': 'EDGAR® Online LLC, a subsidiary of OTC Markets Group', 'subject': 'Form 10-K filed on 2024-02-21 for the period ending 2024-01-28', 'keywords': '0001045810-24-000029; ; 10-K', 'creator': 'EDGAR Filing HTML Converter', 'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creationDate': \"D:20240221173732-05'00'\", 'modDate': \"D:20240221173744-05'00'\", 'trapped': '', 'encryption': 'Standard V2 R3 128-bit RC4'}\n", "page_content='supports diverse hiring, retention, and employee engagement, which we believe makes NVIDIA a great place to work.\\nDuring fiscal year 2025, we will continue to have a flexible work environment and maintain our company wide 2-days off a quarter for employees to rest and\\nrecharge.\\nInformation About Our Executive Officers\\nThe following sets forth certain information regarding our executive officers, their ages, and positions as of February\\xa016, 2024:\\nName\\nAge\\nPosition\\nJen-Hsun Huang\\n60\\nPresident and Chief Executive Officer\\nColette M. Kress\\n56\\nExecutive Vice President and Chief Financial Officer\\nAjay K. Puri\\n69\\nExecutive Vice President, Worldwide Field Operations\\nDebora Shoquist\\n69' metadata={'source': 'input_data/nvidia_10k.pdf', 'file_path': 'input_data/nvidia_10k.pdf', 'page': 11, 'total_pages': 96, 'format': 'PDF 1.4', 'title': '0001045810-24-000029', 'author': 'EDGAR® Online LLC, a subsidiary of OTC Markets Group', 'subject': 'Form 10-K filed on 2024-02-21 for the period ending 2024-01-28', 'keywords': '0001045810-24-000029; ; 10-K', 'creator': 'EDGAR Filing HTML Converter', 'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creationDate': \"D:20240221173732-05'00'\", 'modDate': \"D:20240221173744-05'00'\", 'trapped': '', 'encryption': 'Standard V2 R3 128-bit RC4'}\n", "page_content='Puri previously held marketing, management consulting, and product development positions at Hewlett-Packard, an information technology company, Booz Allen\\nHamilton Inc., a management and technology consulting company, and Texas Instruments Incorporated. Mr. Puri holds a B.S.E.E. degree from the University of\\nMinnesota, an M.S.E.E. degree from the California Institute of Technology and an M.B.A. degree from Harvard Business School.\\nDebora Shoquist joined NVIDIA in 2007 as Senior Vice President of Operations and in 2009 became Executive Vice President of Operations. Prior to NVIDIA,' metadata={'source': 'input_data/nvidia_10k.pdf', 'file_path': 'input_data/nvidia_10k.pdf', 'page': 12, 'total_pages': 96, 'format': 'PDF 1.4', 'title': '0001045810-24-000029', 'author': 'EDGAR® Online LLC, a subsidiary of OTC Markets Group', 'subject': 'Form 10-K filed on 2024-02-21 for the period ending 2024-01-28', 'keywords': '0001045810-24-000029; ; 10-K', 'creator': 'EDGAR Filing HTML Converter', 'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creationDate': \"D:20240221173732-05'00'\", 'modDate': \"D:20240221173744-05'00'\", 'trapped': '', 'encryption': 'Standard V2 R3 128-bit RC4'}\n", "page_content='Table of Contents\\nstrategy, planning, reporting and business development for the division. Prior to joining Microsoft, Ms. Kress spent eight years at Texas Instruments Incorporated,\\na semiconductor company, where she held a variety of finance positions. Ms. Kress holds a B.S. degree in Finance from University of Arizona and an M.B.A.\\ndegree from Southern Methodist University.\\nAjay K. Puri joined NVIDIA in 2005 as Senior Vice President, Worldwide Sales and became Executive Vice President, Worldwide Field Operations in 2009. Prior\\nto NVIDIA, he held positions in sales, marketing, and general management over a 22-year career at Sun Microsystems, Inc., a computing systems company. Mr.' metadata={'source': 'input_data/nvidia_10k.pdf', 'file_path': 'input_data/nvidia_10k.pdf', 'page': 12, 'total_pages': 96, 'format': 'PDF 1.4', 'title': '0001045810-24-000029', 'author': 'EDGAR® Online LLC, a subsidiary of OTC Markets Group', 'subject': 'Form 10-K filed on 2024-02-21 for the period ending 2024-01-28', 'keywords': '0001045810-24-000029; ; 10-K', 'creator': 'EDGAR Filing HTML Converter', 'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creationDate': \"D:20240221173732-05'00'\", 'modDate': \"D:20240221173744-05'00'\", 'trapped': '', 'encryption': 'Standard V2 R3 128-bit RC4'}\n" ] } ], "source": [ "for doc in retrieved_documents:\n", " print(doc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create a RAG Chain" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "from langchain.prompts import ChatPromptTemplate\n", "\n", "template = \"\"\"Answer the question based only on the following context. If you cannot answer the question with the context, please respond with 'I don't know':\n", "\n", "Context:\n", "{context}\n", "\n", "Question:\n", "{question}\n", "\"\"\"\n", "\n", "prompt = ChatPromptTemplate.from_template(template)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "from operator import itemgetter\n", "\n", "from langchain_openai import ChatOpenAI\n", "from langchain_core.output_parsers import StrOutputParser\n", "from langchain_core.runnables import RunnablePassthrough\n", "\n", "primary_qa_llm = ChatOpenAI(model_name=\"gpt-3.5-turbo\", temperature=0)\n", "\n", "retrieval_augmented_qa_chain = (\n", " # INVOKE CHAIN WITH: {\"question\" : \"<>\"}\n", " # \"question\" : populated by getting the value of the \"question\" key\n", " # \"context\" : populated by getting the value of the \"question\" key and chaining it into the base_retriever\n", " {\"context\": itemgetter(\"question\") | retriever, \"question\": itemgetter(\"question\")}\n", " # \"context\" : is assigned to a RunnablePassthrough object (will not be called or considered in the next step)\n", " # by getting the value of the \"context\" key from the previous step\n", " | RunnablePassthrough.assign(context=itemgetter(\"context\"))\n", " # \"response\" : the \"context\" and \"question\" values are used to format our prompt object and then piped\n", " # into the LLM and stored in a key called \"response\"\n", " # \"context\" : populated by getting the value of the \"context\" key from the previous step\n", " | {\"response\": prompt | primary_qa_llm, \"context\": itemgetter(\"context\")}\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Mid-Term Challenge Tests" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Who is the E-VP, Operations - and how old are they?" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Debora Shoquist, 69 years old.\n" ] } ], "source": [ "question = \"Who is the E-VP, Operations - and how old are they?\"\n", "\n", "result = retrieval_augmented_qa_chain.invoke({\"question\" : question})\n", "\n", "print(result[\"response\"].content)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### What is the gross carrying amount of Total Amortizable Intangible Assets for Jan 29, 2023?" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "$3,539 million\n" ] } ], "source": [ "question = \"What is the gross carrying amount of Total Amortizable Intangible Assets for Jan 29, 2023?\"\n", "\n", "result = retrieval_augmented_qa_chain.invoke({\"question\" : question})\n", "\n", "print(result[\"response\"].content)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Evaluation using RAGAS" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "eval_documents = documents\n", "\n", "text_splitter = RecursiveCharacterTextSplitter(\n", " chunk_size = 1500,\n", " chunk_overlap = 400\n", ")\n", "\n", "eval_documents = text_splitter.split_documents(eval_documents)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "624" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(documents)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/var/folders/_1/hpp7wgls4pb3zx13lgn3cp8w0000gn/T/ipykernel_93059/735169035.py:4: DeprecationWarning: The function with_openai was deprecated in 0.1.4, and will be removed in the 0.2.0 release. Use from_langchain instead.\n", " generator = TestsetGenerator.with_openai()\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d98b77660cf94470b304eb2909723ffb", "version_major": 2, "version_minor": 0 }, "text/plain": [ "embedding nodes: 0%| | 0/1248 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
questioncontextsground_truthevolution_typemetadataepisode_done
0What steps does the vendor risk assessment pro...[mitigate and recover from identified and sign...The vendor risk assessment process involves th...simple[{'source': 'input_data/nvidia_10k.pdf', 'file...True
1How has NVIDIA been successful in attracting g...[relating to human capital management.\\nTo be ...NVIDIA has been successful in attracting globa...simple[{'source': 'input_data/nvidia_10k.pdf', 'file...True
2How is revenue designated based on billing loc...[Table of Contents\\nAll Other operating loss -...Revenue by geographic region is designated bas...reasoning[{'source': 'input_data/nvidia_10k.pdf', 'file...True
3How did the U.S. federal research tax credit a...[The effective tax rate increased due to a dec...The U.S. federal research tax credit, along wi...reasoning[{'source': 'input_data/nvidia_10k.pdf', 'file...True
4What is the purpose of a company's internal co...[procedures as we considered necessary in the ...A company's internal control over financial re...multi_context[{'source': 'input_data/nvidia_10k.pdf', 'file...True
\n", "" ], "text/plain": [ " question \\\n", "0 What steps does the vendor risk assessment pro... \n", "1 How has NVIDIA been successful in attracting g... \n", "2 How is revenue designated based on billing loc... \n", "3 How did the U.S. federal research tax credit a... \n", "4 What is the purpose of a company's internal co... \n", "\n", " contexts \\\n", "0 [mitigate and recover from identified and sign... \n", "1 [relating to human capital management.\\nTo be ... \n", "2 [Table of Contents\\nAll Other operating loss -... \n", "3 [The effective tax rate increased due to a dec... \n", "4 [procedures as we considered necessary in the ... \n", "\n", " ground_truth evolution_type \\\n", "0 The vendor risk assessment process involves th... simple \n", "1 NVIDIA has been successful in attracting globa... simple \n", "2 Revenue by geographic region is designated bas... reasoning \n", "3 The U.S. federal research tax credit, along wi... reasoning \n", "4 A company's internal control over financial re... multi_context \n", "\n", " metadata episode_done \n", "0 [{'source': 'input_data/nvidia_10k.pdf', 'file... True \n", "1 [{'source': 'input_data/nvidia_10k.pdf', 'file... True \n", "2 [{'source': 'input_data/nvidia_10k.pdf', 'file... True \n", "3 [{'source': 'input_data/nvidia_10k.pdf', 'file... True \n", "4 [{'source': 'input_data/nvidia_10k.pdf', 'file... True " ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_df = testset.to_pandas()\n", "test_df.head()" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "test_questions = test_df[\"question\"].values.tolist()\n", "test_groundtruths = test_df[\"ground_truth\"].values.tolist()" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "answers = []\n", "contexts = []\n", "\n", "for question in test_questions:\n", " response = retrieval_augmented_qa_chain.invoke({\"question\" : question})\n", " answers.append(response[\"response\"].content)\n", " contexts.append([context.page_content for context in response[\"context\"]])" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "from datasets import Dataset\n", "\n", "response_dataset = Dataset.from_dict({\n", " \"question\" : test_questions,\n", " \"answer\" : answers,\n", " \"contexts\" : contexts,\n", " \"ground_truth\" : test_groundtruths\n", "})" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'question': 'What steps does the vendor risk assessment process involve to mitigate cybersecurity threats?',\n", " 'answer': 'The vendor risk assessment process involves the distribution and review of supplier questionnaires designed to evaluate cybersecurity risks associated with third parties.',\n", " 'contexts': ['mitigate and recover from identified and significant cybersecurity threats.\\nWe also have a vendor risk assessment process consisting of the distribution and review of supplier questionnaires designed to help us evaluate cybersecurity\\nrisks that we may encounter when working with third parties that have access to confidential and other sensitive company information. We take steps designed\\nto ensure that such vendors have implemented data privacy and security controls that help mitigate the cybersecurity risks associated with these vendors. We',\n", " 'routinely assess our high-risk suppliers’ conformance to industry standards (e.g., ISO 27001, ISO 28001, and C-TPAT), and we evaluate them for additional\\ninformation, product, and physical security requirements.\\nRefer to “Item 1A. Risk factors” in this annual report on Form 10-K for additional information about cybersecurity-related risks.\\nGovernance\\nInformation security matters, including managing and assessing risks from cybersecurity threats, remain under the oversight of the Company’s Board of\\nDirectors, or the Board. The Audit Committee of the Board, or the Audit Committee, also reviews the adequacy and effectiveness of the Company’s information',\n", " 'threats. Our information security management program generally follows processes outlined in frameworks such as the ISO 27001 international standard for\\nInformation Security and we evaluate and evolve our security measures as appropriate. We consult with external parties, such as cybersecurity firms and risk\\nmanagement and governance experts, on risk management and strategy.\\nIdentifying, assessing, and managing cybersecurity risk is integrated into our overall risk management systems and processes, and we have in place\\ncybersecurity and data privacy training and policies designed to (a) respond to new requirements in global privacy laws and (b) prevent, detect, respond to,',\n", " 'material risks from cybersecurity threats. Our Chief Security Officer’s cybersecurity expertise includes over 17 years of combined government and private sector\\nassignments.\\n31'],\n", " 'ground_truth': 'The vendor risk assessment process involves the distribution and review of supplier questionnaires designed to evaluate cybersecurity risks. The goal is to ensure that vendors have implemented data privacy and security controls to mitigate cybersecurity risks.'}" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response_dataset[0]" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [], "source": [ "from ragas import evaluate\n", "from ragas.metrics import (\n", " faithfulness,\n", " answer_relevancy,\n", " answer_correctness,\n", " context_recall,\n", " context_precision,\n", ")\n", "\n", "metrics = [\n", " faithfulness,\n", " answer_relevancy,\n", " context_recall,\n", " context_precision,\n", " answer_correctness,\n", "]" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d9f291cd7c7a457a8873d8650383df87", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Evaluating: 0%| | 0/50 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
questionanswercontextsground_truthfaithfulnessanswer_relevancycontext_recallcontext_precisionanswer_correctness
0What steps does the vendor risk assessment pro...The vendor risk assessment process involves th...[mitigate and recover from identified and sign...The vendor risk assessment process involves th...1.00.9410411.00.8333330.620211
1How has NVIDIA been successful in attracting g...NVIDIA has been successful in attracting globa...[relating to human capital management.\\nTo be ...NVIDIA has been successful in attracting globa...1.00.9999991.00.8333331.000000
2How is revenue designated based on billing loc...Revenue is designated based on the billing loc...[10 \\n(2)\\n— \\nTotal\\n$\\n(4,890)\\n$\\n(5,411)\\n...Revenue by geographic region is designated bas...1.00.9000001.01.0000000.743623
3How did the U.S. federal research tax credit a...The U.S. federal research tax credit, along wi...[The effective tax rate increased due to a dec...The U.S. federal research tax credit, along wi...0.50.8545371.00.9166670.745303
4What is the purpose of a company's internal co...The purpose of a company's internal control ov...[Management’s Annual Report on Internal Contro...A company's internal control over financial re...1.00.9501081.01.0000000.743341
5How does NVIDIA align their principles with su...NVIDIA aligns their principles with sustainabi...[Sustainability and Governance\\nNVIDIA invents...NVIDIA integrates sound environmental, social,...1.00.9656901.01.0000000.838499
6What technologies does NVIDIA RTX use to enhan...NVIDIA RTX uses ray tracing technology and dee...[television graphics.\\nThe NVIDIA RTX platform...NVIDIA RTX uses ray tracing technology and dee...1.00.9763491.00.6388890.999318
7What types of stock-based awards can be grante...The types of stock-based awards that can be gr...[The 2007 Plan authorizes the issuance of ince...The types of stock-based awards that can be gr...1.00.9558351.00.8055560.617505
8What is the purpose of rebates for resellers a...The purpose of rebates for resellers is to ser...[For products sold with a right of return, we ...The purpose of rebates for resellers is to ser...1.00.9445861.00.0000000.611390
9How has NVIDIA been successful in attracting g...NVIDIA has been successful in attracting globa...[relating to human capital management.\\nTo be ...NVIDIA has been successful in attracting globa...1.01.0000001.00.8333331.000000
\n", "" ], "text/plain": [ " question \\\n", "0 What steps does the vendor risk assessment pro... \n", "1 How has NVIDIA been successful in attracting g... \n", "2 How is revenue designated based on billing loc... \n", "3 How did the U.S. federal research tax credit a... \n", "4 What is the purpose of a company's internal co... \n", "5 How does NVIDIA align their principles with su... \n", "6 What technologies does NVIDIA RTX use to enhan... \n", "7 What types of stock-based awards can be grante... \n", "8 What is the purpose of rebates for resellers a... \n", "9 How has NVIDIA been successful in attracting g... \n", "\n", " answer \\\n", "0 The vendor risk assessment process involves th... \n", "1 NVIDIA has been successful in attracting globa... \n", "2 Revenue is designated based on the billing loc... \n", "3 The U.S. federal research tax credit, along wi... \n", "4 The purpose of a company's internal control ov... \n", "5 NVIDIA aligns their principles with sustainabi... \n", "6 NVIDIA RTX uses ray tracing technology and dee... \n", "7 The types of stock-based awards that can be gr... \n", "8 The purpose of rebates for resellers is to ser... \n", "9 NVIDIA has been successful in attracting globa... \n", "\n", " contexts \\\n", "0 [mitigate and recover from identified and sign... \n", "1 [relating to human capital management.\\nTo be ... \n", "2 [10 \\n(2)\\n— \\nTotal\\n$\\n(4,890)\\n$\\n(5,411)\\n... \n", "3 [The effective tax rate increased due to a dec... \n", "4 [Management’s Annual Report on Internal Contro... \n", "5 [Sustainability and Governance\\nNVIDIA invents... \n", "6 [television graphics.\\nThe NVIDIA RTX platform... \n", "7 [The 2007 Plan authorizes the issuance of ince... \n", "8 [For products sold with a right of return, we ... \n", "9 [relating to human capital management.\\nTo be ... \n", "\n", " ground_truth faithfulness \\\n", "0 The vendor risk assessment process involves th... 1.0 \n", "1 NVIDIA has been successful in attracting globa... 1.0 \n", "2 Revenue by geographic region is designated bas... 1.0 \n", "3 The U.S. federal research tax credit, along wi... 0.5 \n", "4 A company's internal control over financial re... 1.0 \n", "5 NVIDIA integrates sound environmental, social,... 1.0 \n", "6 NVIDIA RTX uses ray tracing technology and dee... 1.0 \n", "7 The types of stock-based awards that can be gr... 1.0 \n", "8 The purpose of rebates for resellers is to ser... 1.0 \n", "9 NVIDIA has been successful in attracting globa... 1.0 \n", "\n", " answer_relevancy context_recall context_precision answer_correctness \n", "0 0.941041 1.0 0.833333 0.620211 \n", "1 0.999999 1.0 0.833333 1.000000 \n", "2 0.900000 1.0 1.000000 0.743623 \n", "3 0.854537 1.0 0.916667 0.745303 \n", "4 0.950108 1.0 1.000000 0.743341 \n", "5 0.965690 1.0 1.000000 0.838499 \n", "6 0.976349 1.0 0.638889 0.999318 \n", "7 0.955835 1.0 0.805556 0.617505 \n", "8 0.944586 1.0 0.000000 0.611390 \n", "9 1.000000 1.0 0.833333 1.000000 " ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results_df = results.to_pandas()\n", "results_df" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "py_3_11_lamma2_run", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } }, "nbformat": 4, "nbformat_minor": 2 }