Spaces:
Sleeping
Sleeping
| import streamlit as st | |
| # Custom CSS for better styling | |
| st.markdown(""" | |
| <style> | |
| .main-title { | |
| font-size: 36px; | |
| color: #4A90E2; | |
| font-weight: bold; | |
| text-align: center; | |
| } | |
| .sub-title { | |
| font-size: 24px; | |
| color: #333333; | |
| margin-top: 20px; | |
| } | |
| .section { | |
| background-color: #f9f9f9; | |
| padding: 15px; | |
| border-radius: 10px; | |
| margin-top: 20px; | |
| } | |
| .section h2 { | |
| font-size: 22px; | |
| color: #4A90E2; | |
| } | |
| .section p, .section ul { | |
| color: #666666; | |
| } | |
| .link { | |
| color: #4A90E2; | |
| text-decoration: none; | |
| } | |
| </style> | |
| """, unsafe_allow_html=True) | |
| # Introduction | |
| st.markdown('<div class="main-title">Text Preprocessing with Spark NLP</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>Welcome to the Spark NLP Text Preprocessing Demo App! In the field of Natural Language Processing (NLP), preprocessing is a crucial step that ensures the text data is clean and suitable for modeling. Effective preprocessing can significantly enhance the performance of NLP models.</p> | |
| <p>Spark NLP stands out as a leading library for text preprocessing, offering a range of tools and models within an easy-to-use pipeline design compatible with Apache Spark. This demo showcases how you can leverage Spark NLP to preprocess your text data efficiently.</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| # About Text Preprocessing | |
| st.markdown('<div class="sub-title">About Text Preprocessing</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>Text preprocessing involves a series of steps to clean and normalize text data. Common tasks include tokenization, stopword removal, stemming, lemmatization, and more. These steps are essential for preparing raw text for downstream NLP tasks.</p> | |
| <p>In Spark NLP, text preprocessing is facilitated through various annotators that can be combined into a preprocessing pipeline. We'll demonstrate how to use these annotators in Python to preprocess text data effectively.</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.image('https://www.johnsnowlabs.com/wp-content/uploads/2023/05/img_blog_2-4.jpg', caption='Text preprocessing pipeline visual', use_column_width='auto') | |
| # How to Use the Preprocessing Tools | |
| st.markdown('<div class="sub-title">How to Use Spark NLP for Text Preprocessing</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>To preprocess text using Spark NLP, we need to create a pipeline that includes various preprocessing annotators. These annotators transform the input text through steps like tokenization, normalization, and stopword removal.</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.markdown('<div class="sub-title">Installation</div>', unsafe_allow_html=True) | |
| st.code('!pip install spark-nlp', language='python') | |
| # Import Libraries and Read Data | |
| st.markdown('<div class="sub-title">Importing Libraries and Reading Data</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>First, we'll import Spark NLP and necessary libraries, read the data from a local file, and convert it into a Spark DataFrame.</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| import sparknlp | |
| from sparknlp.base import * | |
| from sparknlp.annotator import * | |
| spark= sparknlp.start() | |
| df= spark.read\\ | |
| .option("header", True)\\ | |
| .csv("spam_text_messages.csv")\\ | |
| .toDF("category", "text") | |
| df.show(5, truncate=30) | |
| >>> | |
| +--------+------------------------------+ | |
| |category| text| | |
| +--------+------------------------------+ | |
| | ham|Go until jurong point, craz...| | |
| | ham| Ok lar... Joking wif u oni...| | |
| | spam|Free entry in 2 a wkly comp...| | |
| | ham|U dun say so early hor... U...| | |
| | ham|Nah I don't think he goes t...| | |
| +--------+------------------------------+ | |
| only showing top 5 rows | |
| """, language='python') | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The dataset contains two columns: <strong>category</strong> and <strong>text</strong>. The text column contains messages, and the category column indicates whether the message is spam or not (ham).</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| # Document Assembler | |
| st.markdown('<div class="sub-title">Document Assembler</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The DocumentAssembler is the beginning part of any Spark NLP project. It creates the first annotation of type Document, which may be used by annotators down the road. We use it as follows:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| documentAssembler = DocumentAssembler() \\ | |
| .setInputCol("text") \\ | |
| .setOutputCol("document") \\ | |
| .setCleanupMode("shrink") | |
| df_doc = documentAssembler.transform(df) | |
| df_doc.printSchema() | |
| """, language='python') | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The <code>shrink</code> cleanup mode removes new lines and tabs, merging multiple spaces and blank lines into a single space. The schema after transformation includes the new <strong>document</strong> column.</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| # Tokenizer | |
| st.markdown('<div class="sub-title">Tokenizer</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The Tokenizer identifies the tokens in Spark NLP:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| tokenizer = Tokenizer() \\ | |
| .setInputCols(["document"]) \\ | |
| .setOutputCol("token") | |
| """, language='python') | |
| # Sentence Detector | |
| st.markdown('<div class="sub-title">Sentence Detector</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The SentenceDetector finds sentence boundaries in raw text:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| sentenceDetector = SentenceDetector() \\ | |
| .setInputCols(["document"]) \\ | |
| .setOutputCol("sentence") | |
| """, language='python') | |
| # Normalizer | |
| st.markdown('<div class="sub-title">Normalizer</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The Normalizer cleans dirty characters after regex pattern and removes words based on a given dictionary:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| normalizer = Normalizer() \\ | |
| .setInputCols(["token"]) \\ | |
| .setOutputCol("normalized") \\ | |
| .setLowercase(True) \\ | |
| .setCleanupPatterns(["[^\w\d\s]"]) | |
| """, language='python') | |
| # Stopwords Cleaner | |
| st.markdown('<div class="sub-title">Stopwords Cleaner</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The StopWordsCleaner removes stopwords from the text:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| stopwordsCleaner = StopWordsCleaner() \\ | |
| .setInputCols(["token"]) \\ | |
| .setOutputCol("cleaned_tokens") \\ | |
| .setCaseSensitive(True) | |
| """, language='python') | |
| # Token Assembler | |
| st.markdown('<div class="sub-title">Token Assembler</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The TokenAssembler assembles cleaned tokens back together:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| tokenAssembler = TokenAssembler() \\ | |
| .setInputCols(["sentence", "cleaned_tokens"]) \\ | |
| .setOutputCol("assembled") | |
| """, language='python') | |
| # Stemmer | |
| st.markdown('<div class="sub-title">Stemmer</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The Stemmer reduces inflectional forms and sometimes derivationally related forms of a word to a common base form:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| stemmer = Stemmer() \\ | |
| .setInputCols(["token"]) \\ | |
| .setOutputCol("stem") | |
| """, language='python') | |
| # Lemmatizer | |
| st.markdown('<div class="sub-title">Lemmatizer</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>The Lemmatizer reduces words to their base or root form, referring to a dictionary to understand the word's meaning:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| lemmatizer = Lemmatizer() \\ | |
| .setInputCols(["token"]) \\ | |
| .setOutputCol("lemma") \\ | |
| .setDictionary("AntBNC_lemmas_ver_001.txt", value_delimiter="\\t", key_delimiter="->") | |
| """, language='python') | |
| # Pipeline | |
| st.markdown('<div class="sub-title">Putting All Processes into a Spark ML Pipeline</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>Now, we will put all the preprocessing stages into a Spark ML Pipeline and apply it to our dataset.</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| from pyspark.ml import Pipeline | |
| nlpPipeline = Pipeline(stages=[ | |
| documentAssembler, | |
| tokenizer, | |
| sentenceDetector, | |
| normalizer, | |
| stopwordsCleaner, | |
| tokenAssembler, | |
| stemmer, | |
| lemmatizer | |
| ]) | |
| empty_df = spark.createDataFrame([[""]]).toDF("text") | |
| model = nlpPipeline.fit(empty_df) | |
| result = model.transform(df) | |
| """, language='python') | |
| # Showcase/Example | |
| st.markdown('<div class="sub-title">Showcase/Example</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>Let's examine the results of our preprocessing pipeline, starting with tokens and normalized tokens:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| from pyspark.sql import functions as F | |
| result.select("token.result", "normalized.result").show(5, truncate=30) | |
| >>> | |
| +------------------------------+------------------------------+ | |
| | result| result| | |
| +------------------------------+------------------------------+ | |
| |[Go, until, jurong, point, ...|[go, until, jurong, point, ...| | |
| |[Ok, lar, ..., Joking, wif,...|[ok, lar, joking, wif, u, oni]| | |
| |[Free, entry, in, 2, a, wkl...|[free, entry, in, 2, a, wkl...| | |
| |[U, dun, say, so, early, ho...|[u, dun, say, so, early, ho...| | |
| |[Nah, I, don't, think, he, ...|[nah, ı, dont, think, he, g...| | |
| +------------------------------+------------------------------+ | |
| only showing top 5 rows | |
| """, language='python') | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>Next, we check the cleaned data from stopwords:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| result.select(F.explode(F.arrays_zip("token.result", "cleaned_tokens.result")).alias("col")) \\ | |
| .select(F.expr("col['0']").alias("token"), F.expr("col['1']").alias("cleaned_sw")).show(10) | |
| >>> | |
| +------------------------------+------------------------------+ | |
| | result| result| | |
| +------------------------------+------------------------------+ | |
| |[Go, until, jurong, point, ...|[go, until, jurong, point, ...| | |
| |[Ok, lar, ..., Joking, wif,...|[ok, lar, joking, wif, u, oni]| | |
| |[Free, entry, in, 2, a, wkl...|[free, entry, in, 2, a, wkl...| | |
| |[U, dun, say, so, early, ho...|[u, dun, say, so, early, ho...| | |
| |[Nah, I, don't, think, he, ...|[nah, ı, dont, think, he, g...| | |
| +------------------------------+------------------------------+ | |
| only showing top 5 rows | |
| """, language='python') | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>Finally, we compare the sentence detector result and token assembler result:</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.code(""" | |
| result.select(F.explode(F.arrays_zip("sentence.result", "assembled.result")).alias("col")) \\ | |
| .select(F.expr("col['0']").alias("sentence"), F.expr("col['1']").alias("assembled")).show(5, truncate=30) | |
| >>> | |
| +------------------------------+------------------------------+ | |
| | sentence| assembled| | |
| +------------------------------+------------------------------+ | |
| | Go until jurong point, crazy.| Go jurong point, crazy| | |
| | .| | | |
| |Available only in bugis n g...|Available bugis n great wor...| | |
| | Cine there got amore wat.| Cine got amore wat| | |
| | .| | | |
| +------------------------------+------------------------------+ | |
| only showing top 5 rows | |
| result.withColumn("tmp", F.explode("assembled")) \\ | |
| .select("tmp.*").select("begin", "end", "result", "metadata.sentence").show(5, truncate=30) | |
| >>> | |
| +-----+---+------------------------------+--------+ | |
| |begin|end| result|sentence| | |
| +-----+---+------------------------------+--------+ | |
| | 0| 21| Go jurong point, crazy| 0| | |
| | 29| 28| | 1| | |
| | 31| 74|Available bugis n great wor...| 2| | |
| | 84|101| Cine got amore wat| 3| | |
| | 109|108| | 4| | |
| +-----+---+------------------------------+--------+ | |
| only showing top 5 rows | |
| """, language='python') | |
| st.markdown(""" | |
| <div class="section"> | |
| <p>In this example, we have successfully cleaned and preprocessed text data using various annotators and transformers in Spark NLP. This preprocessing pipeline is essential for preparing the data for further NLP tasks, ensuring that the text is clean and normalized.</p> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.markdown('<div class="sub-title">Additional Resources and References</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <ul> | |
| <li><a class="link" href="https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/1hr_workshop/SparkNLP_openSource_workshop_1hr.ipynb" target="_blank">Intro to Spark NLP workflow</a></li> | |
| <li><a class="link" href="https://sparknlp.org/docs/en/quickstart" target="_blank">Getting Started with Spark NLP</a></li> | |
| <li><a class="link" href="https://nlp.johnsnowlabs.com/models" target="_blank">Pretrained Models</a></li> | |
| <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp/tree/master/examples/python/annotation/text/english" target="_blank">Example Notebooks</a></li> | |
| <li><a class="link" href="https://sparknlp.org/docs/en/install" target="_blank">Installation Guide</a></li> | |
| </ul> | |
| </div> | |
| """, unsafe_allow_html=True) | |
| st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True) | |
| st.markdown(""" | |
| <div class="section"> | |
| <ul> | |
| <li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li> | |
| <li><a class="link" href="https://join.slack.com/t/spark-nlp/shared_invite/zt-198dipu77-L3UWNe_AJ8xqDk0ivmih5Q" target="_blank">Slack</a>: Live discussion with the community and team</li> | |
| <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub</a>: Bug reports, feature requests, and contributions</li> | |
| <li><a class="link" href="https://medium.com/spark-nlp" target="_blank">Medium</a>: Spark NLP articles</li> | |
| <li><a class="link" href="https://www.youtube.com/channel/UCmFOjlpYEhxf_wJUDuz6xxQ/videos" target="_blank">YouTube</a>: Video tutorials</li> | |
| </ul> | |
| </div> | |
| """, unsafe_allow_html=True) |