{ "cells": [ { "cell_type": "markdown", "id": "3eb9d2c1", "metadata": {}, "source": [ "## 7. Bulk prompts" ] }, { "cell_type": "markdown", "id": "960b1cbf", "metadata": {}, "source": [ "Our reusable prompting function is pretty cool. But requesting answers one by one across a big dataset could take forever. And with the Hugging Face free API, we’re likely to hit rate limits or timeouts if we send too many requests too quickly.\n", "\n", "One solution is to submit your requests in batches and then ask the LLM to return its responses in bulk.\n", "\n", "A common way to do that is to prompt the LLM to return its responses in JSON, a JavaScript data format that is easy to work with in Python.\n", "\n", "To try that, we start by adding the built-in json library to our imports." ] }, { "cell_type": "code", "execution_count": 49, "id": "ec94fe49", "metadata": {}, "outputs": [], "source": [ "import json # NEW\n", "from rich import print\n", "import requests\n", "from retry import retry\n", "import os\n", "from huggingface_hub import InferenceClient\n", "\n", "api_key = os.getenv(\"HF_TOKEN\")\n", "client = InferenceClient(\n", " token=api_key,\n", ")" ] }, { "cell_type": "markdown", "id": "eac8a34e", "metadata": {}, "source": [ "Next, we make a series of changes to our function to adapt it to work with a batch of inputs. Get ready. It’s a lot.\n", "- We tweak the name of the function.\n", "- We change our input argument to a list.\n", "- We expand our prompt to explain that we will provide a list of team names.\n", "- We ask the LLM to classify them individually, returning its answers in a JSON list.\n", "- We insist on getting one answer for each input.\n", "- We tweak our few-shot training to reflect this new approach.\n", "- We submit our input as a single string with new lines separating each team name.\n", "- We convert the LLM’s response from a string to a list using the `json.loads` function.\n", "- We check that the LLM’s answers are in our list of acceptable answers with a loop through the list.\n", "- We merge the team names and the LLM’s answers into a dictionary returned by the function." ] }, { "cell_type": "code", "execution_count": 26, "id": "70477229", "metadata": {}, "outputs": [], "source": [ "@retry(ValueError, tries=2, delay=2)\n", "def classify_teams(name_list): # NEW\n", " prompt = \"\"\"\n", "You are an AI model trained to classify text.\n", "\n", "I will provide list of professional sports team names separated by new lines\n", "\n", "You will reply with the sports league in which they compete.\n", "\n", "Your responses must come from the following list:\n", "- Major League Baseball (MLB)\n", "- National Football League (NFL)\n", "- National Basketball Association (NBA)\n", "\n", "If the team's league is not on the list, you should label them as \"Other\".\n", "\n", "Your answers should be returned as a flat JSON list.\n", "\n", "It is very important that the length of JSON list you return is exactly the same as the number of names you receive.\n", "\n", "If I were to submit:\n", "\n", "\"Los Angeles Rams\\nLos Angeles Dodgers\\nLos Angeles Lakers\\nLos Angeles Kings\"\n", "\n", "You should return the following:\n", "\n", "[\"National Football League (NFL)\", \"Major League Baseball (MLB)\", \"National Basketball Association (NBA)\", \"Other\"]\n", "\"\"\"\n", "\n", " response = client.chat.completions.create(\n", " messages=[\n", " {\n", " \"role\": \"system\",\n", " \"content\": prompt,\n", " },\n", " ### <-- NEW \n", " {\n", " \"role\": \"user\",\n", " \"content\": \"Chicago Bears\\nChicago Cubs\\nChicago Bulls\\nChicago Blackhawks\",\n", " },\n", " {\n", " \"role\": \"assistant\",\n", " \"content\": '[\"National Football League (NFL)\", \"Major League Baseball (MLB)\", \"National Basketball Association (NBA)\", \"Other\"]',\n", " },\n", " {\n", " \"role\": \"user\",\n", " \"content\": \"\\n\".join(name_list),\n", " }\n", " ### --> \n", " ],\n", " model=\"meta-llama/Llama-3.3-70B-Instruct\",\n", " temperature=0,\n", " )\n", "\n", " answer_str = response.choices[0].message.content # NEW\n", " answer_list = json.loads(answer_str) # NEW\n", "\n", " acceptable_answers = [\n", " \"Major League Baseball (MLB)\",\n", " \"National Football League (NFL)\",\n", " \"National Basketball Association (NBA)\",\n", " \"Other\",\n", " ]\n", " ### <-- NEW\n", " for answer in answer_list:\n", " if answer not in acceptable_answers:\n", " raise ValueError(f\"{answer} not in list of acceptable answers\")\n", " return dict(zip(name_list, answer_list))\n", " ### -->" ] }, { "cell_type": "markdown", "id": "4a0909ed", "metadata": {}, "source": [ "Try that with our team list. And you’ll see that it works with only a single API call. The same technique will work for a batch of any size." ] }, { "cell_type": "code", "execution_count": 28, "id": "2bb71639", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'Minnesota Twins': 'Major League Baseball (MLB)',\n", " 'Minnesota Vikings': 'National Football League (NFL)',\n", " 'Minnesota Timberwolves': 'National Basketball Association (NBA)'}" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "team_list = [\"Minnesota Twins\", \"Minnesota Vikings\", \"Minnesota Timberwolves\"]\n", "classify_teams(team_list)" ] }, { "cell_type": "markdown", "id": "b6815a5c", "metadata": {}, "source": [ "Though, as you batches get bigger, one common problem is that the number of outputs from the LLM can fail to match the number of inputs you provide. This problem may lessen as LLMs improve, but for now it’s a good idea to limit to batches to a few dozen inputs and to verify that you’re getting the right number back." ] }, { "cell_type": "code", "execution_count": 29, "id": "8295afc9", "metadata": {}, "outputs": [], "source": [ "@retry(ValueError, tries=2, delay=2)\n", "def classify_teams(name_list):\n", " prompt = \"\"\"\n", "You are an AI model trained to classify text.\n", "\n", "I will provide list of professional sports team names separated by new lines\n", "\n", "You will reply with the sports league in which they compete.\n", "\n", "Your responses must come from the following list:\n", "- Major League Baseball (MLB)\n", "- National Football League (NFL)\n", "- National Basketball Association (NBA)\n", "\n", "If the team's league is not on the list, you should label them as \"Other\".\n", "\n", "Your answers should be returned as a flat JSON list.\n", "\n", "It is very important that the length of JSON list you return is exactly the same as the number of names you receive.\n", "\n", "If I were to submit:\n", "\n", "\"Los Angeles Rams\\nLos Angeles Dodgers\\nLos Angeles Lakers\\nLos Angeles Kings\"\n", "\n", "You should return the following:\n", "\n", "[\"National Football League (NFL)\", \"Major League Baseball (MLB)\", \"National Basketball Association (NBA)\", \"Other\"]\n", "\"\"\"\n", "\n", " response = client.chat.completions.create(\n", " messages=[\n", " {\n", " \"role\": \"system\",\n", " \"content\": prompt,\n", " },\n", " {\n", " \"role\": \"user\",\n", " \"content\": \"Chicago Bears,Chicago Cubs,Chicago Bulls,Chicago Blackhawks\",\n", " },\n", " {\n", " \"role\": \"assistant\",\n", " \"content\": '[\"National Football League (NFL)\", \"Major League Baseball (MLB)\", \"National Basketball Association (NBA)\", \"Other\"]',\n", " },\n", " {\n", " \"role\": \"user\",\n", " \"content\": \"\\n\".join(name_list),\n", " }\n", " ],\n", " model=\"meta-llama/Llama-3.3-70B-Instruct\",\n", " temperature=0,\n", " )\n", "\n", " answer_str = response.choices[0].message.content\n", " answer_list = json.loads(answer_str)\n", "\n", " acceptable_answers = [\n", " \"Major League Baseball (MLB)\",\n", " \"National Football League (NFL)\",\n", " \"National Basketball Association (NBA)\",\n", " \"Other\",\n", " ]\n", " for answer in answer_list:\n", " if answer not in acceptable_answers:\n", " raise ValueError(f\"{answer} not in list of acceptable answers\")\n", "\n", " ### <-- NEW\n", " try:\n", " assert len(name_list) == len(answer_list)\n", " except AssertionError:\n", " raise ValueError(f\"Number of outputs ({len(name_list)}) does not equal the number of inputs ({len(answer_list)})\")\n", " ### -->\n", " return dict(zip(name_list, answer_list))" ] }, { "cell_type": "markdown", "id": "587604c0", "metadata": {}, "source": [ "Okay. Naming sports teams is a cute trick, but what about something hard? And whatever happened to that George Santos idea?\n", "\n", "We’ll tackle that by pulling in our example dataset using `pandas`, a popular data manipulation library in Python.\n", "\n", "First, we need to install it. Back to our installation cell." ] }, { "cell_type": "code", "execution_count": null, "id": "9ff5e26c", "metadata": { "scrolled": true }, "outputs": [], "source": [ "%pip install huggingface_hub rich ipywidgets retry pandas" ] }, { "cell_type": "markdown", "id": "295346d4", "metadata": {}, "source": [ "Then import it." ] }, { "cell_type": "code", "execution_count": 32, "id": "c6d289d7", "metadata": {}, "outputs": [], "source": [ "import json\n", "from rich import print\n", "import requests\n", "from retry import retry\n", "import pandas as pd # NEW" ] }, { "cell_type": "markdown", "id": "1ee5cf9f", "metadata": {}, "source": [ "Now we’re ready to load the California expenditures data prepared for the class. It contains the distinct list of all vendors listed as payees in itemized receipts attached to disclosure filings." ] }, { "cell_type": "code", "execution_count": 33, "id": "3ae2b2fc", "metadata": {}, "outputs": [], "source": [ "df = pd.read_csv(\"https://raw.githubusercontent.com/palewire/first-llm-classifier/refs/heads/main/_notebooks/Form460ScheduleESubItem.csv\")" ] }, { "cell_type": "markdown", "id": "bca7002b", "metadata": {}, "source": [ "Have a look at a random sample to get a taste of what’s in there." ] }, { "cell_type": "code", "execution_count": 34, "id": "0aa44f42", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
payee
5822GRAND HYATT SAN FRANCISCO
8765LIZ FIGUEROA FOR LT. GOVERNOR
2027CA STATE UNIVERSITY NORTHRIDGE YOUNG DEMOCRATS
1371BEN FRANKLIN CRAFTS
9033LYNWOOD FOR BETTER HEALTHCARE, SPONSORED BY SE...
11720QUINTESSA
7983KOST FM
4324DUNKIN DONUTS
2434CARDENAS MARKETS, INC.
5738GOLDEN PALACE
\n", "
" ], "text/plain": [ " payee\n", "5822 GRAND HYATT SAN FRANCISCO\n", "8765 LIZ FIGUEROA FOR LT. GOVERNOR\n", "2027 CA STATE UNIVERSITY NORTHRIDGE YOUNG DEMOCRATS\n", "1371 BEN FRANKLIN CRAFTS\n", "9033 LYNWOOD FOR BETTER HEALTHCARE, SPONSORED BY SE...\n", "11720 QUINTESSA\n", "7983 KOST FM\n", "4324 DUNKIN DONUTS\n", "2434 CARDENAS MARKETS, INC.\n", "5738 GOLDEN PALACE" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.sample(10)" ] }, { "cell_type": "markdown", "id": "9fe1e695", "metadata": {}, "source": [ "Now let’s adapt what we have to fit. Instead of asking for a sports league back, we will ask the LLM to classify each payee as a restaurant, bar, hotel or other establishment." ] }, { "cell_type": "code", "execution_count": 35, "id": "970c5161", "metadata": {}, "outputs": [], "source": [ "@retry(ValueError, tries=2, delay=2)\n", "### <-- NEW\n", "def classify_payees(name_list):\n", " prompt = \"\"\"You are an AI model trained to categorize businesses based on their names.\n", "\n", "You will be given a list of business names, each separated by a new line.\n", "\n", "Your task is to analyze each name and classify it into one of the following categories: Restaurant, Bar, Hotel, or Other.\n", "\n", "It is extremely critical that there is a corresponding category output for each business name provided as an input.\n", "\n", "If a business does not clearly fall into Restaurant, Bar, or Hotel categories, you should classify it as \"Other\".\n", "\n", "Even if the type of business is not immediately clear from the name, it is essential that you provide your best guess based on the information available to you. If you can't make a good guess, classify it as Other.\n", "\n", "For example, if given the following input:\n", "\n", "\"Intercontinental Hotel\\nPizza Hut\\nCheers\\nWelsh's Family Restaurant\\nKTLA\\nDirect Mailing\"\n", "\n", "Your output should be a JSON list in the following format:\n", "\n", "[\"Hotel\", \"Restaurant\", \"Bar\", \"Restaurant\", \"Other\", \"Other\"]\n", "\n", "This means that you have classified \"Intercontinental Hotel\" as a Hotel, \"Pizza Hut\" as a Restaurant, \"Cheers\" as a Bar, \"Welsh's Family Restaurant\" as a Restaurant, and both \"KTLA\" and \"Direct Mailing\" as Other.\n", "\n", "Ensure that the number of classifications in your output matches the number of business names in the input. It is very important that the length of JSON list you return is exactly the same as the number of business names youyou receive.\n", "\"\"\"\n", "### -->\n", " response = client.chat.completions.create(\n", " messages=[\n", " {\n", " \"role\": \"system\",\n", " \"content\": prompt,\n", " },\n", " ### <-- NEW\n", " {\n", " \"role\": \"user\",\n", " \"content\": \"Intercontinental Hotel\\nPizza Hut\\nCheers\\nWelsh's Family Restaurant\\nKTLA\\nDirect Mailing\",\n", " },\n", " {\n", " \"role\": \"assistant\",\n", " \"content\": '[\"Hotel\", \"Restaurant\", \"Bar\", \"Restaurant\", \"Other\", \"Other\"]',\n", " },\n", " {\n", " \"role\": \"user\",\n", " \"content\": \"Subway Sandwiches\\nRuth Chris Steakhouse\\nPolitical Consulting Co\\nThe Lamb's Club\",\n", " },\n", " {\n", " \"role\": \"assistant\",\n", " \"content\": '[\"Restaurant\", \"Restaurant\", \"Other\", \"Bar\"]',\n", " },\n", " ### -->\n", " {\n", " \"role\": \"user\",\n", " \"content\": \"\\n\".join(name_list),\n", " }\n", " ],\n", " model=\"meta-llama/Llama-3.3-70B-Instruct\",\n", " temperature=0,\n", " )\n", "\n", " answer_str = response.choices[0].message.content\n", " answer_list = json.loads(answer_str)\n", "\n", " ### <-- NEW \n", " acceptable_answers = [\n", " \"Restaurant\",\n", " \"Bar\",\n", " \"Hotel\",\n", " \"Other\",\n", " ] \n", " ### -->\n", " \n", " for answer in answer_list:\n", " if answer not in acceptable_answers:\n", " raise ValueError(f\"{answer} not in list of acceptable answers\")\n", "\n", " try:\n", " assert len(name_list) == len(answer_list)\n", " except AssertionError:\n", " raise ValueError(f\"Number of outputs ({len(name_list)}) does not equal the number of inputs ({len(answer_list)})\")\n", "\n", " return dict(zip(name_list, answer_list))" ] }, { "cell_type": "markdown", "id": "bf9b69a0", "metadata": {}, "source": [ "Now pull out a random sample of payees as a list." ] }, { "cell_type": "code", "execution_count": 36, "id": "fe74a8ed", "metadata": {}, "outputs": [], "source": [ "sample_list = list(df.sample(10).payee)" ] }, { "cell_type": "markdown", "id": "688192ae", "metadata": {}, "source": [ "And see how it does." ] }, { "cell_type": "code", "execution_count": 38, "id": "a84d364a", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'CALIFORNIA NOW ORGANIZATION': 'Other',\n", " 'ALOHA SIGNS': 'Other',\n", " \"SABELLA'S ITALIAN MARKET\": 'Restaurant',\n", " 'ELIZABETH ESPARZA': 'Other',\n", " 'DATA-SCRIBE': 'Other',\n", " \"LISA HEMENWAY'S BISTRO\": 'Restaurant',\n", " 'NEW EDGE MULTIMEDIA': 'Other',\n", " 'FUSILLI': 'Restaurant',\n", " 'FRIENDS OF DR IRENE PINKARD FOR CITY COUNCIL': 'Other',\n", " 'ZEN SUSHI SACRAMENTO': 'Restaurant'}" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "classify_payees(sample_list)" ] }, { "cell_type": "markdown", "id": "197856d9", "metadata": {}, "source": [ "That’s nice for a sample. But how do you loop through the entire dataset and code them.\n", "\n", "One way to start is to write a function that will split up a list into batches of a certain size." ] }, { "cell_type": "code", "execution_count": 39, "id": "b784940f", "metadata": {}, "outputs": [], "source": [ "def get_batch_list(li, n=10):\n", " \"\"\"Split the provided list into batches of size `n`.\"\"\"\n", " batch_list = []\n", " for i in range(0, len(li), n):\n", " batch_list.append(li[i : i + n])\n", " return batch_list" ] }, { "cell_type": "markdown", "id": "9e35948e", "metadata": {}, "source": [ "Before we loop through our payees, let’s add a couple libraries that will let us avoid hammering HF and keep tabs on our progress." ] }, { "cell_type": "code", "execution_count": 40, "id": "f1593a7d", "metadata": {}, "outputs": [], "source": [ "import time # NEW\n", "import json\n", "from rich import print\n", "from rich.progress import track # NEW\n", "import requests\n", "from retry import retry\n", "import pandas as pd" ] }, { "cell_type": "markdown", "id": "da68c37f", "metadata": {}, "source": [ "That batching trick can then be fit into a new function that will accept a big list of payees and classify them batch by batch." ] }, { "cell_type": "code", "execution_count": 41, "id": "6e6965f9", "metadata": {}, "outputs": [], "source": [ "def classify_batches(name_list, batch_size=10, wait=2):\n", " \"\"\"Split the provided list of names into batches and classify with our LLM them one by one.\"\"\"\n", " # Create a place to store the results\n", " all_results = {}\n", "\n", " # Batch up the list\n", " batch_list = get_batch_list(name_list, n=batch_size)\n", "\n", " # Loop through the list in batches\n", " for batch in track(batch_list):\n", " # Classify it with the LLM\n", " batch_results = classify_payees(batch)\n", "\n", " # Add what we get back to the results\n", " all_results.update(batch_results)\n", "\n", " # Tap the brakes to avoid overloading groq's API\n", " time.sleep(wait)\n", "\n", " # Return the results\n", " return all_results" ] }, { "cell_type": "markdown", "id": "222a3846", "metadata": {}, "source": [ "Now, let’s take out a bigger sample." ] }, { "cell_type": "code", "execution_count": 42, "id": "39778766", "metadata": {}, "outputs": [], "source": [ "bigger_sample = list(df.sample(100).payee)" ] }, { "cell_type": "markdown", "id": "722de39a", "metadata": {}, "source": [ "And let it rip." ] }, { "cell_type": "code", "execution_count": null, "id": "7f676e52", "metadata": { "scrolled": true }, "outputs": [], "source": [ "classify_batches(bigger_sample)" ] }, { "cell_type": "markdown", "id": "fc2c1f94", "metadata": {}, "source": [ "Printing out to the console is interesting, but eventually you’ll want to be able to work with the results in a more structured way. So let’s convert the results into a `pandas` DataFrame by modifying our function." ] }, { "cell_type": "code", "execution_count": 44, "id": "c41b736f", "metadata": {}, "outputs": [], "source": [ "def classify_batches(name_list, batch_size=10, wait=2):\n", " # Store the results\n", " all_results = {}\n", "\n", " # Batch up the list\n", " batch_list = get_batch_list(name_list, n=batch_size)\n", "\n", " # Loop through the list in batches\n", " for batch in track(batch_list):\n", " # Classify it\n", " batch_results = classify_payees(batch)\n", "\n", " # Add it to the results\n", " all_results.update(batch_results)\n", "\n", " # Tap the brakes\n", " time.sleep(wait)\n", "\n", " # Return the results (NEW)\n", " return pd.DataFrame(\n", " all_results.items(),\n", " columns=[\"payee\", \"category\"]\n", " )" ] }, { "cell_type": "markdown", "id": "353adf11", "metadata": {}, "source": [ "Results can now be stored as a DataFrame." ] }, { "cell_type": "code", "execution_count": 45, "id": "51aa0550", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a4b561bb64554c70a7bfd309c43b60da", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results_df = classify_batches(bigger_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96bd75bc",
   "metadata": {},
   "source": [
    "And inspected using the standard `pandas` tools. Let's take a peek at the first records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "514971f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "
\n",
       "\n",
       "\n",
       "  \n",
       "    \n",
       "      \n",
       "      \n",
       "      \n",
       "    \n",
       "  \n",
       "  \n",
       "    \n",
       "      \n",
       "      \n",
       "      \n",
       "    \n",
       "    \n",
       "      \n",
       "      \n",
       "      \n",
       "    \n",
       "    \n",
       "      \n",
       "      \n",
       "      \n",
       "    \n",
       "    \n",
       "      \n",
       "      \n",
       "      \n",
       "    \n",
       "    \n",
       "      \n",
       "      \n",
       "      \n",
       "    \n",
       "  \n",
       "
payeecategory
0TAXIPASSOther
1THE JEFFERSON HOTELHotel
2NORMS RESTAURANTRestaurant
3JENNY OROPEZA FOR STATE SENATEOther
4BIG MAMA'S & PAPA'S PIZZERIARestaurant
\n",
       "
"
      ],
      "text/plain": [
       "                            payee    category\n",
       "0                        TAXIPASS       Other\n",
       "1             THE JEFFERSON HOTEL       Hotel\n",
       "2                NORMS RESTAURANT  Restaurant\n",
       "3  JENNY OROPEZA FOR STATE SENATE       Other\n",
       "4    BIG MAMA'S & PAPA'S PIZZERIA  Restaurant"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fef7be4e",
   "metadata": {},
   "source": [
    "Or a sum of all the categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "6911dc37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "category\n",
       "Other         67\n",
       "Restaurant    20\n",
       "Hotel         12\n",
       "Bar            1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df.category.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8705a48-49e6-4ec8-8f3f-126bcf011f0f",
   "metadata": {},
   "source": [
    "**[8. Evaluating prompts →](ch8-evaluating-prompts.ipynb)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d96f1b46-7e66-4e5a-8d17-84a75b70404e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}