{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "view-in-github" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "x_Vp8SiKM4p1" }, "source": [ "# Exploring Alternative Media Document Sources\n", "Test how one could get YouTube videos or websites as sources for documents in a vector store.\n", "\n", "- YouTube: https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/youtube_audio\n", "- Websites:\n", " - https://js.langchain.com/docs/modules/indexes/document_loaders/examples/web_loaders/\n", " - https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_base\n", " - Extracting relevant information from website: https://www.oncrawl.com/technical-seo/extract-relevant-text-content-from-html-page/\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "o_60X8H3NEne" }, "source": [ "## Libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "pxcqXgg2aAN7", "outputId": "0bb1c0aa-99f7-4d8d-a66f-992ea54eb5ff" }, "outputs": [], "source": [ "# install libraries here\n", "# -q flag for \"quiet\" install\n", "!pip install -q langchain\n", "!pip install -q openai\n", "!pip install -q unstructured\n", "!pip install -q tiktoken\n", "!pip install typing_extensions==4.5.0" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 784 }, "id": "mwpl3jYJoGo7", "outputId": "cbae7a1a-b4a4-4d32-f837-dcafb3d01746" }, "outputs": [], "source": [ "%pip install -q trafilatura\n", "%pip install -q justext" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "NU-7ynWHvwfM", "outputId": "cdd9a4db-1bd0-471c-d31e-e0c16aff0c98" }, "outputs": [], "source": [ "%pip install yt_dlp\n", "%pip install pydub" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "pEjM1tLsMZBq" }, "outputs": [], "source": [ "# import libraries here\n", "import os\n", "import time\n", "import pprint\n", "from getpass import getpass\n", "\n", "from langchain.docstore.document import Document\n", "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", "from langchain.embeddings import OpenAIEmbeddings\n", "\n", "from langchain.document_loaders.unstructured import UnstructuredFileLoader" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "0U6N_9xFsOcw" }, "outputs": [], "source": [ "from langchain.document_loaders.generic import GenericLoader\n", "from langchain.document_loaders.parsers import OpenAIWhisperParser\n", "from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "id": "JRw367IwryWd" }, "outputs": [], "source": [ "from langchain.document_loaders import WebBaseLoader\n", "import trafilatura\n", "import requests\n", "import justext" ] }, { "cell_type": "markdown", "metadata": { "id": "n0BTyPI_srMg" }, "source": [] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "NOX639OA2pOh" }, "outputs": [], "source": [ "# Export requirements.txt (if needed)\n", "%pip freeze > requirements.txt" ] }, { "cell_type": "markdown", "metadata": { "id": "03KLZGI_a5W5" }, "source": [ "## API Keys\n", "\n", "Use these cells to load the API keys required for this notebook. The below code cell uses the `getpass` library." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5smcWj4DbFgy", "outputId": "969c13c3-2b77-4d7b-aa69-e24629894e6a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "··········\n" ] } ], "source": [ "openai_api_key = getpass()\n", "os.environ[\"OPENAI_API_KEY\"] = openai_api_key" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "id": "Jgh9igPesX3F" }, "outputs": [], "source": [ "def splitter(text):\n", " # Split input text\n", " text_splitter = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=150)\n", " splits = text_splitter.split_text(text)\n", " return splits" ] }, { "cell_type": "markdown", "metadata": { "id": "F2W_fMfRUJj2" }, "source": [ "## YouTube" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "id": "xm2aHrWdvztG" }, "outputs": [], "source": [ "def youtube_transcript(urls, save_dir = \"content\"):\n", " # Transcribe the videos to text\n", " # save_dir: directory to save audio files\n", " youtube_loader = GenericLoader(YoutubeAudioLoader(urls, save_dir), OpenAIWhisperParser())\n", " youtube_docs = youtube_loader.load()\n", " # Combine doc\n", " combined_docs = [doc.page_content for doc in youtube_docs]\n", " text = \" \".join(combined_docs)\n", " return text" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "u3pRTWrBv_oJ" }, "outputs": [], "source": [ "# Two Karpathy lecture videos\n", "urls = [\"https://youtu.be/kCc8FmEb1nY\", \"https://youtu.be/VMj-3S1tku0\"]\n", "youtube_text = youtube_transcript(urls)\n", "youtube_text" ] }, { "cell_type": "markdown", "metadata": { "id": "wsMTgKjnUmql" }, "source": [ "## Websites" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "id": "aaIuM970pupK" }, "outputs": [], "source": [ "url = \"https://www.espn.com/\"" ] }, { "cell_type": "markdown", "metadata": { "id": "B2CW1oIgp5w3" }, "source": [ "### WebBaseLoader" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "id": "MYw1qpovlnxe" }, "outputs": [], "source": [ "def website_webbase(url):\n", " website_loader = WebBaseLoader(url)\n", " website_data = website_loader.load()\n", " # Combine doc\n", " combined_docs = [doc.page_content for doc in website_data]\n", " text = \" \".join(combined_docs)\n", " return text" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 139 }, "id": "M2A-lEpasbVo", "outputId": "0936e800-b254-47a8-b865-ed10bec191b1" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "\"\\n\\n\\n\\n\\n\\n\\n\\n\\nESPN - Serving Sports Fans. Anytime. Anywhere.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n Skip to main content\\n \\n\\n Skip to navigation\\n \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n<\\n\\n>\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nMenuESPN\\n\\n\\nSearch\\n\\n\\n\\nscores\\n\\n\\n\\nNFLNBANHLMLBSoccerTennis…NCAAFNCAAMNCAAWSports BettingBoxingCFLNCAACricketF1GolfHorseMMANASCARNBA G LeagueOlympic SportsPLLRacingRN BBRN FBRugbyWNBAWWEX GamesXFLMore ESPNFantasyListenWatchESPN+\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n \\n\\nSUBSCRIBE NOW\\n\\n\\n\\n\\n\\nThe Ultimate Fighter: Season 31\\n\\n\\n\\n\\n\\n\\n\\nWimbledon: Select Courts\\n\\n\\n\\n\\n\\n\\n\\nNBA Summer League: Select Games\\n\\n\\n\\n\\n\\n\\n\\nProjecting Messi's Performance In MLS\\n\\n\\nQuick Links\\n\\n\\n\\n\\nNBA Summer League\\n\\n\\n\\n\\n\\n\\n\\nNBA Free Agency Buzz\\n\\n\\n\\n\\n\\n\\n\\nNBA Trade Machine\\n\\n\\n\\n\\n\\n\\n\\n2023 MLB Draft\\n\\n\\n\\n\\n\\n\\n\\n2023 MLB All-Star Weekend\\n\\n\\n\\n\\n\\n\\n\\nNHL Free Agency\\n\\n\\n\\n\\n\\n\\n\\nWomen's World Cup\\n\\n\\n\\n\\n\\n\\n\\nHow To Watch PGA TOUR\\n\\n\\n\\n\\n\\n\\nFavorites\\n\\n\\n\\n\\n\\n\\n Manage Favorites\\n \\n\\n\\n\\nCustomize ESPNSign UpLog InESPN Sites\\n\\n\\n\\n\\nESPN Deportes\\n\\n\\n\\n\\n\\n\\n\\nAndscape\\n\\n\\n\\n\\n\\n\\n\\nespnW\\n\\n\\n\\n\\n\\n\\n\\nESPNFC\\n\\n\\n\\n\\n\\n\\n\\nX Games\\n\\n\\n\\n\\n\\n\\n\\nSEC Network\\n\\n\\nESPN Apps\\n\\n\\n\\n\\nESPN\\n\\n\\n\\n\\n\\n\\n\\nESPN Fantasy\\n\\n\\nFollow ESPN\\n\\n\\n\\n\\nFacebook\\n\\n\\n\\n\\n\\n\\n\\nTwitter\\n\\n\\n\\n\\n\\n\\n\\nInstagram\\n\\n\\n\\n\\n\\n\\n\\nSnapchat\\n\\n\\n\\n\\n\\n\\n\\nYouTube\\n\\n\\n\\n\\n\\n\\n\\nThe ESPN Daily Podcast\\n\\n\\nMeet all 23 USWNT players going to the World Cup: Fun facts, insightful stats and moreFrom Alex Morgan to Megan Rapinoe, get to know everyone on the Women's World Cup roster on and off the field.7hCaitlin MurrayIllustration by ESPNNotable absentees: From Becky Sauerbrunn to Beth MeadThere are over 60 players who won't play at the Women's World Cup because of injury.11hSophie LawsonHow Mia Hamm ended up playing as a goalie at a World CupWomen's World Cup 2023: Schedule, teams, venues, moreTOP HEADLINESKamara agrees to plea deal in Vegas assault caseMLBPA wants tweak to pitch timer before playoffsNorthwestern to keep assistant coaches for 2023Surfing star Jones dies after accident with boardMessi lands in U.S. ahead of Inter Miami unveilingDiet to discipline: Zion knows he can do moreDjokovic ties Federer with 46th Slam semifinalTour-PIF talks eyed Norman exit, Tiger LIV teamWho are the NFL's best cornerbacks?MLB All-Star GameMeet the All-Star Game first-timersFrom rookies to breakouts to one guy finally getting his due, here's what to expect from MLB's most notable new All-Stars.10hJesse RogersAP Photo/Ted WarrenAll-Star Game: Predictions and much moreBaseball's best are in Seattle. Here are the matchups we want to see and what our experts think will happen.3hESPNTHE TALK OF SUMMER LEAGUEChet Holmgren, Chris Paul and more buzz from VegasOur NBA insiders have the latest from Vegas, including Holmgren's return, the in-season tournament and Paul's fit on the Warriors.7hNBA insidersPhoto by Chris Gardner/Getty ImagesNOT BUYING INTO ZION'S WORDSZion's diet comments not well received by Kendrick Perkins and Richard Jefferson49m1:47WIMBLEDON SCOREBOARDTUESDAY'S MATCHESSee AllBIG UPSET AT WIMBLEDONElina Svitolina takes down No. 1 seed Iga Swiatek5h0:55Takeaways from Swiatek's surprising loss to SvitolinaIga Swiatek came into Wimbledon as a favorite to win -- but grass once again proved to be her downfall.3hAlyssa RoenigkSURVEYING EVERY POSITIONWho are the NFL's best cornerbacks? Execs, coaches and scouts help rank 2023's top 10Who are the best corners in the NFL? Execs, coaches and scouts from around the league ranked their top 10 in our annual summer series.10hJeremy FowlerPhoto by Ethan Miller/Getty ImagesRanking the best players at every position: Execs make their picksFor the fourth straight year, we asked execs, coaches, scouts and players to name their top 10 at all positions.10hJeremy FowlerFITZGERALD OUT AT NORTHWESTERNCOLLEGE FOOTBALLRece Davis 'shocked' by Northwestern hazing claims under Fitzgerald19h3:22BIG 12 MEDIA DAYSCOLLEGE FOOTBALLLast stand for Texas and Oklahoma, newcomers to watch and expansion talkThe Big 12 will start a busy month of media days on Wednesday, July 12. Here are the biggest questions facing the conference ahead of the 2023 season.1mBill Connelly and Dave WilsonRaymond Carlin/Icon Sportswire Top HeadlinesKamara agrees to plea deal in Vegas assault caseMLBPA wants tweak to pitch timer before playoffsNorthwestern to keep assistant coaches for 2023Surfing star Jones dies after accident with boardMessi lands in U.S. ahead of Inter Miami unveilingDiet to discipline: Zion knows he can do moreDjokovic ties Federer with 46th Slam semifinalTour-PIF talks eyed Norman exit, Tiger LIV teamWho are the NFL's best cornerbacks?Favorites FantasyManage FavoritesFantasy HomeCustomize ESPNSign UpLog InICYMI0:38J-Rod puts on a show with record 41 HRs in Round 1Mariners star Julio Rodriguez electrifies the Seattle crowd with a record 41 home runs in Round 1 of the Home Run Derby. Best of ESPN+Illustration by ESPNRanking the NFL's best players at every position for 2023: Execs, coaches, scouts pick their top 10 at every positionBest corners in the NFL? Edge rushers? Linebackers? For the fourth straight year, we asked execs, coaches, scouts and players to name their top 10.AP Photo/John LocherThree takeaways from Victor Wembanyama's second gameKevin Pelton looks at what worked for Wembanyama on Sunday compared to Friday and where the 19-year-old can continue to improve.Illustration by ESPN2024 NFL mock draft: Jordan Reid's early first-round predictionsThree QBs in the top 10? A run on offensive linemen? Impact defenders galore? Here are Jordan Reid's early projections for next year's 32 first-round picks. Trending NowBettmann/Getty ImagesInside the worst team in NBA history, the 1972-73 SixersThe Philadelphia 76ers started the 1972-73 season by losing 21 of 23 games. They'd finish with the worst record in NBA history, 9-73. This is the story of what the team learned about themselves through turmoil.Ron Chenoy-USA TODAY SportsCups of coffee: Seven former NFL players remember their one and only gameTheir NFL careers lasted a single game. Seven members of a unique professional club discuss their journeys.Illustration by MASAThe FC 100 for 2023: Haaland, Mbappe lead our list of best men's soccer playersAfter a brief hiatus thanks to the winter World Cup in Qatar, ESPN presents its seventh annual ranking of the best men's players and coaches in world soccer! Welcome to FC 100. How to Watch on ESPN+(AP Photo/Koji Sasahara, File)How to watch the PGA Tour, Masters, PGA Championship and FedEx Cup playoffs on ESPN, ESPN+Here's everything you need to know about how to watch the PGA Tour, Masters, PGA Championship and FedEx Cup playoffs on ESPN and ESPN+. Sign up for FREE!Create A LeagueJoin a Public LeagueReactivate a LeaguePractice with a Mock DraftSign up to play the #1 Fantasy game!Create A LeagueJoin Public LeagueReactivateMock Draft NowSign up for FREE!Create A LeagueJoin a Public LeagueReactivate a LeaguePractice With a Mock DraftSign up for FREE!Create A LeagueJoin a Public LeaguePractice With a Mock Draft\\n\\nESPN+\\n\\n\\n\\n\\nThe Ultimate Fighter: Season 31\\n\\n\\n\\n\\n\\n\\n\\nWimbledon: Select Courts\\n\\n\\n\\n\\n\\n\\n\\nNBA Summer League: Select Games\\n\\n\\n\\n\\n\\n\\n\\nProjecting Messi's Performance In MLS\\n\\n\\nQuick Links\\n\\n\\n\\n\\nNBA Summer League\\n\\n\\n\\n\\n\\n\\n\\nNBA Free Agency Buzz\\n\\n\\n\\n\\n\\n\\n\\nNBA Trade Machine\\n\\n\\n\\n\\n\\n\\n\\n2023 MLB Draft\\n\\n\\n\\n\\n\\n\\n\\n2023 MLB All-Star Weekend\\n\\n\\n\\n\\n\\n\\n\\nNHL Free Agency\\n\\n\\n\\n\\n\\n\\n\\nWomen's World Cup\\n\\n\\n\\n\\n\\n\\n\\nHow To Watch PGA TOUR\\n\\n\\nESPN Sites\\n\\n\\n\\n\\nESPN Deportes\\n\\n\\n\\n\\n\\n\\n\\nAndscape\\n\\n\\n\\n\\n\\n\\n\\nespnW\\n\\n\\n\\n\\n\\n\\n\\nESPNFC\\n\\n\\n\\n\\n\\n\\n\\nX Games\\n\\n\\n\\n\\n\\n\\n\\nSEC Network\\n\\n\\nESPN Apps\\n\\n\\n\\n\\nESPN\\n\\n\\n\\n\\n\\n\\n\\nESPN Fantasy\\n\\n\\nFollow ESPN\\n\\n\\n\\n\\nFacebook\\n\\n\\n\\n\\n\\n\\n\\nTwitter\\n\\n\\n\\n\\n\\n\\n\\nInstagram\\n\\n\\n\\n\\n\\n\\n\\nSnapchat\\n\\n\\n\\n\\n\\n\\n\\nYouTube\\n\\n\\n\\n\\n\\n\\n\\nThe ESPN Daily Podcast\\n\\n\\nTerms of UsePrivacy PolicyYour US State Privacy RightsChildren's Online Privacy PolicyInterest-Based AdsAbout Nielsen MeasurementDo Not Sell or Share My Personal InformationContact UsDisney Ad Sales SiteWork for ESPNCopyright: © ESPN Enterprises, Inc. All rights reserved.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\"" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "webbase_text = website_webbase(url)\n", "webbase_text" ] }, { "cell_type": "markdown", "metadata": { "id": "fLFNd6S8oAlD" }, "source": [ "### Trafilatura Parsing\n", "\n", "[Tralifatura](https://trafilatura.readthedocs.io/en/latest/) is a Python and command-line utility which attempts to extracts the most relevant information from a given website. " ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "id": "H3gtJjSfoK5C" }, "outputs": [], "source": [ "def website_trafilatura(url):\n", " downloaded = trafilatura.fetch_url(url)\n", " return trafilatura.extract(downloaded)" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 52 }, "id": "ft8QDKTTsekG", "outputId": "7eefb402-470d-4ff6-df72-4834a610f17a" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'|Sports|\\n|scores||News|\\n|© 2023 ESPN Internet Ventures. Terms of Use and Privacy Policy and Safety Information / Your California Privacy Rights are applicable to you. All rights reserved.|\\n|\\nMore From ESPN:\\n|\\nESPN en Español | Andscape | FiveThirtyEight | ESPN FC | ESPNCricinfo'" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trafilatura_text = website_trafilatura(url)\n", "trafilatura_text" ] }, { "cell_type": "markdown", "metadata": { "id": "evXJEtZtobn0" }, "source": [ "### jusText\n", "\n", "[jusText](https://pypi.org/project/jusText/) is another Python library for extracting content from a website." ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "id": "AfahISIvph_Y" }, "outputs": [], "source": [ "def website_justext(url):\n", " response = requests.get(url)\n", " paragraphs = justext.justext(response.content, justext.get_stoplist(\"English\"))\n", " content = [paragraph.text for paragraph in paragraphs \\\n", " if not paragraph.is_boilerplate]\n", " text = \" \".join(content)\n", " return text" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 52 }, "id": "eB_BdMjXsg3T", "outputId": "b65bfc3f-2538-4b0a-99da-926855f9e21d" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "\"Trending Now The Philadelphia 76ers started the 1972-73 season by losing 21 of 23 games. They'd finish with the worst record in NBA history, 9-73. This is the story of what the team learned about themselves through turmoil.\"" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "justext_text = website_justext(url)\n", "justext_text" ] } ], "metadata": { "colab": { "include_colab_link": true, "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }