Update pages/Workflow & Model Overview.py

pages/Workflow & Model Overview.py

@@ -1,224 +1,238 @@
-import streamlit as st
-
-# Custom CSS for better styling
-st.markdown("""
-    <style>
-        .main-title {
-            font-size: 36px;
-            color: #4A90E2;
-            font-weight: bold;
-            text-align: center;
-        }
-        .sub-title {
-            font-size: 24px;
-            color: #4A90E2;
-            margin-top: 20px;
-        }
-        .section {
-            background-color: #f9f9f9;
-            padding: 15px;
-            border-radius: 10px;
-            margin-top: 20px;
-        }
-        .section h2 {
-            font-size: 22px;
-            color: #4A90E2;
-        }
-        .section p, .section ul {
-            color: #666666;
-        }
-        .link {
-            color: #4A90E2;
-            text-decoration: none;
-        }
-    </style>
-""", unsafe_allow_html=True)
-
-# Introduction
-st.markdown('<div class="main-title">Sentiment Analysis with Spark NLP</div>', unsafe_allow_html=True)
-
-st.markdown("""
-<div class="section">
-    <p>Welcome to the Spark NLP Sentiment Analysis Demo App! Sentiment analysis is an automated process capable of understanding the feelings or opinions that underlie a text. This process is considered a text classification and is one of the most interesting subfields of NLP. Using Spark NLP, it is possible to analyze the sentiment in a text with high accuracy.</p>
-    <p>This app demonstrates how to use Spark NLP's SentimentDetector to perform sentiment analysis using a rule-based approach.</p>
-</div>
-""", unsafe_allow_html=True)
-
-st.image('images/Sentiment-Analysis.jpg',caption="Difference between rule-based and machine learning based sentiment analysis applications", use_column_width='auto')
-
-# About Sentiment Analysis
-st.markdown('<div class="sub-title">About Sentiment Analysis</div>', unsafe_allow_html=True)
-st.markdown("""
-<div class="section">
-    <p>Sentiment analysis studies the subjective information in an expression, such as opinions, appraisals, emotions, or attitudes towards a topic, person, or entity. Expressions can be classified as positive, negative, or neutral — in some cases, even more detailed.</p>
-    <p>Some popular sentiment analysis applications include social media monitoring, customer support management, and analyzing customer feedback.</p>
-</div>
-""", unsafe_allow_html=True)
-
-# Using SentimentDetector in Spark NLP
-st.markdown('<div class="sub-title">Using SentimentDetector in Spark NLP</div>', unsafe_allow_html=True)
-st.markdown("""
-<div class="section">
-    <p>The SentimentDetector annotator in Spark NLP uses a rule-based approach to analyze the sentiment in text data. This method involves using a set of predefined rules or patterns to classify text as positive, negative, or neutral.</p>
-    <p>Spark NLP also provides Machine Learning (ML) and Deep Learning (DL) solutions for sentiment analysis. If you are interested in those approaches, please check the <a class="link" href="https://nlp.johnsnowlabs.com/docs/en/annotators#viveknsentiment" target="_blank" rel="noopener">ViveknSentiment </a> and <a class="link" href="https://nlp.johnsnowlabs.com/docs/en/annotators#sentimentdl" target="_blank" rel="noopener">SentimentDL</a> annotators of Spark NLP.</p>
-</div>
-""", unsafe_allow_html=True)
-
-st.markdown('<h2 class="sub-title">Example Usage in Python</h2>', unsafe_allow_html=True)
-st.markdown('<p>Here’s how you can implement sentiment analysis using the SentimentDetector annotator in Spark NLP:</p>', unsafe_allow_html=True)
-
-# Setup Instructions
-st.markdown('<div class="sub-title">Setup</div>', unsafe_allow_html=True)
-st.markdown('<p>To install Spark NLP in Python, use your favorite package manager (conda, pip, etc.). For example:</p>', unsafe_allow_html=True)
-st.code("""
-pip install spark-nlp
-pip install pyspark
-""", language="bash")
-
-st.markdown("<p>Then, import Spark NLP and start a Spark session:</p>", unsafe_allow_html=True)
-st.code("""
-import sparknlp
-
-# Start Spark Session
-spark = sparknlp.start()
-""", language='python')
-
-# load data
-st.markdown('<div class="sub-title">Start by loading the Dataset, Lemmas and the Sentiment Dictionary.</div>', unsafe_allow_html=True)
-st.code("""
-!wget -N https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/lemma-corpus-small/lemmas_small.txt -P /tmp
-!wget -N https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/sentiment-corpus/default-sentiment-dict.txt -P /tmp
-""", language="bash")
-
-st.image('images/dataset.png', caption="First few lines of the lemmas and sentiment dictionary", use_column_width='auto')
-
-# Sentiment Analysis Example
-st.markdown('<div class="sub-title">Example Usage: Sentiment Analysis with SentimentDetector</div>', unsafe_allow_html=True)
-st.code('''
-from sparknlp.base import DocumentAssembler, Pipeline, Finisher
-from sparknlp.annotator import (
-    SentenceDetector,
-    Tokenizer,
-    Lemmatizer,
-    SentimentDetector
-)
-import pyspark.sql.functions as F
-
-# Step 1: Transforms raw texts to document annotation
-document_assembler = (
-    DocumentAssembler()
-    .setInputCol("text")
-    .setOutputCol("document")
-)
-
-# Step 2: Sentence Detection
-sentence_detector = SentenceDetector().setInputCols(["document"]).setOutputCol("sentence")
-
-# Step 3: Tokenization
-tokenizer = Tokenizer().setInputCols(["sentence"]).setOutputCol("token")
-
-# Step 4: Lemmatization
-lemmatizer = (
-    Lemmatizer()
-    .setInputCols("token")
-    .setOutputCol("lemma")
-    .setDictionary("/tmp/lemmas_small.txt", key_delimiter="->", value_delimiter="\\t")
-)
-
-# Step 5: Sentiment Detection
-sentiment_detector = (
-    SentimentDetector()
-    .setInputCols(["lemma", "sentence"])
-    .setOutputCol("sentiment_score")
-    .setDictionary("/tmp/default-sentiment-dict.txt", ",")
-)
-
-# Step 6: Finisher
-finisher = (
-    Finisher()
-    .setInputCols(["sentiment_score"])
-    .setOutputCols(["sentiment"])
-)
-
-# Define the pipeline
-pipeline = Pipeline(
-    stages=[
-        document_assembler,
-        sentence_detector,
-        tokenizer,
-        lemmatizer,
-        sentiment_detector,
-        finisher,
-    ]
-)
-
-# Create a spark Data Frame with an example sentence
-data = spark.createDataFrame(
-    [
-        ["The restaurant staff is really nice"]
-    ]
-).toDF("text")  # use the column name `text` defined in the pipeline as input
-
-# Fit-transform to get predictions
-result = pipeline.fit(data).transform(data).show(truncate=50)
-''', language='python')
-
-st.text("""
-+-----------------------------------+----------+
-|                               text| sentiment|
-+-----------------------------------+----------+
-|The restaurant staff is really nice|[positive]|
-+-----------------------------------+----------+
-""")
-
-st.markdown("""
-<p>The code snippet demonstrates how to set up a pipeline in Spark NLP to perform sentiment analysis on text data using the SentimentDetector annotator. The resulting DataFrame contains the sentiment predictions.</p>
-""", unsafe_allow_html=True)
-
-# One-liner Alternative
-st.markdown('<div class="sub-title">One-liner Alternative</div>', unsafe_allow_html=True)
-st.markdown("""
-<div class="section">
-    <p>In October 2022, John Snow Labs released the open-source <code>johnsnowlabs</code> library that contains all the company products, open-source and licensed, under one common library. This simplified the workflow, especially for users working with more than one of the libraries (e.g., Spark NLP + Healthcare NLP). This new library is a wrapper on all of John Snow Lab’s libraries and can be installed with pip:</p>
-    <p><code>pip install johnsnowlabs</code></p>
-</div>
-""", unsafe_allow_html=True)
-
-st.markdown('<p>To run sentiment analysis with one line of code, we can simply:</p>', unsafe_allow_html=True)
-st.code("""
-# Import the NLP module which contains Spark NLP and NLU libraries
-from johnsnowlabs import nlp
-
-sample_text = "The restaurant staff is really nice"
-
-# Returns a pandas DataFrame, we select the desired columns
-nlp.load('en.sentiment').predict(sample_text, output_level='sentence')
-""", language='python')
-
-st.image('images/johnsnowlabs-sentiment-output.png', use_column_width='auto')
-
-st.markdown("""
-<p>This approach demonstrates how to use the <code>johnsnowlabs</code> library to perform sentiment analysis with a single line of code. The resulting DataFrame contains the sentiment predictions.</p>
-""", unsafe_allow_html=True)
-
-# Conclusion
-st.markdown("""
-<div class="section">
-    <h2>Conclusion</h2>
-    <p>In this app, we demonstrated how to use Spark NLP's SentimentDetector annotator to perform sentiment analysis on text data. These powerful tools enable users to efficiently process large datasets and identify sentiment, providing deeper insights for various applications. By integrating these annotators into your NLP pipelines, you can enhance text understanding, information extraction, and customer sentiment analysis.</p>
-</div>
-""", unsafe_allow_html=True)
-
-# References and Additional Information
-st.markdown('<div class="sub-title">For additional information, please check the following references.</div>', unsafe_allow_html=True)
-
-st.markdown("""
-<div class="section">
-        <ul>
-            <li>Documentation :&nbsp;<a href="https://nlp.johnsnowlabs.com/docs/en/transformers#sentiment" target="_blank" rel="noopener">SentimentDetector</a></li>
-            <li>Python Docs :&nbsp;<a href="https://nlp.johnsnowlabs.com/api/python/reference/autosummary/sparknlp/annotator/sentiment/index.html#sparknlp.annotator.sentiment.SentimentDetector" target="_blank" rel="noopener">SentimentDetector</a></li>
-            <li>Scala Docs :&nbsp;<a href="https://nlp.johnsnowlabs.com/api/com/johnsnowlabs/nlp/annotators/sentiment/SentimentDetector.html" target="_blank" rel="noopener">SentimentDetector</a></li>
-            <li>Example Notebook :&nbsp;<a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/training/english/classification/Sentiment%20Analysis.ipynb" target="_blank" rel="noopener">Sentiment Analysis</a></li>
-        </ul>
-    </div>
-""", unsafe_allow_html=True)
+import streamlit as st
+
+# Custom CSS for better styling
+st.markdown("""
+    <style>
+        .main-title {
+            font-size: 36px;
+            color: #4A90E2;
+            font-weight: bold;
+            text-align: center;
+        }
+        .sub-title {
+            font-size: 24px;
+            color: #4A90E2;
+            margin-top: 20px;
+        }
+        .section {
+            background-color: #f9f9f9;
+            padding: 15px;
+            border-radius: 10px;
+            margin-top: 20px;
+        }
+        .section h2 {
+            font-size: 22px;
+            color: #4A90E2;
+        }
+        .section p, .section ul {
+            color: #666666;
+        }
+        .link {
+            color: #4A90E2;
+            text-decoration: none;
+        }
+    </style>
+""", unsafe_allow_html=True)
+
+# Introduction
+st.markdown('<div class="main-title">Sentiment Analysis with Spark NLP</div>', unsafe_allow_html=True)
+
+st.markdown("""
+<div class="section">
+    <p>Welcome to the Spark NLP Sentiment Analysis Demo App! Sentiment analysis is an automated process capable of understanding the feelings or opinions that underlie a text. This process is considered a text classification and is one of the most interesting subfields of NLP. Using Spark NLP, it is possible to analyze the sentiment in a text with high accuracy.</p>
+    <p>This app demonstrates how to use Spark NLP's SentimentDetector to perform sentiment analysis using a rule-based approach.</p>
+</div>
+""", unsafe_allow_html=True)
+
+st.image('images/Sentiment-Analysis.jpg',caption="Difference between rule-based and machine learning based sentiment analysis applications", use_column_width='auto')
+
+# About Sentiment Analysis
+st.markdown('<div class="sub-title">About Sentiment Analysis</div>', unsafe_allow_html=True)
+st.markdown("""
+<div class="section">
+    <p>Sentiment analysis studies the subjective information in an expression, such as opinions, appraisals, emotions, or attitudes towards a topic, person, or entity. Expressions can be classified as positive, negative, or neutral — in some cases, even more detailed.</p>
+    <p>Some popular sentiment analysis applications include social media monitoring, customer support management, and analyzing customer feedback.</p>
+</div>
+""", unsafe_allow_html=True)
+
+# Using SentimentDetector in Spark NLP
+st.markdown('<div class="sub-title">Using SentimentDetector in Spark NLP</div>', unsafe_allow_html=True)
+st.markdown("""
+<div class="section">
+    <p>The SentimentDetector annotator in Spark NLP uses a rule-based approach to analyze the sentiment in text data. This method involves using a set of predefined rules or patterns to classify text as positive, negative, or neutral.</p>
+    <p>Spark NLP also provides Machine Learning (ML) and Deep Learning (DL) solutions for sentiment analysis. If you are interested in those approaches, please check the <a class="link" href="https://nlp.johnsnowlabs.com/docs/en/annotators#viveknsentiment" target="_blank" rel="noopener">ViveknSentiment </a> and <a class="link" href="https://nlp.johnsnowlabs.com/docs/en/annotators#sentimentdl" target="_blank" rel="noopener">SentimentDL</a> annotators of Spark NLP.</p>
+</div>
+""", unsafe_allow_html=True)
+
+st.markdown('<h2 class="sub-title">Example Usage in Python</h2>', unsafe_allow_html=True)
+st.markdown('<p>Here’s how you can implement sentiment analysis using the SentimentDetector annotator in Spark NLP:</p>', unsafe_allow_html=True)
+
+# Setup Instructions
+st.markdown('<div class="sub-title">Setup</div>', unsafe_allow_html=True)
+st.markdown('<p>To install Spark NLP in Python, use your favorite package manager (conda, pip, etc.). For example:</p>', unsafe_allow_html=True)
+st.code("""
+pip install spark-nlp
+pip install pyspark
+""", language="bash")
+
+st.markdown("<p>Then, import Spark NLP and start a Spark session:</p>", unsafe_allow_html=True)
+st.code("""
+import sparknlp
+
+# Start Spark Session
+spark = sparknlp.start()
+""", language='python')
+
+# load data
+st.markdown('<div class="sub-title">Start by loading the Dataset, Lemmas and the Sentiment Dictionary.</div>', unsafe_allow_html=True)
+st.code("""
+!wget -N https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/lemma-corpus-small/lemmas_small.txt -P /tmp
+!wget -N https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/sentiment-corpus/default-sentiment-dict.txt -P /tmp
+""", language="bash")
+
+st.image('images/dataset.png', caption="First few lines of the lemmas and sentiment dictionary", use_column_width='auto')
+
+# Sentiment Analysis Example
+st.markdown('<div class="sub-title">Example Usage: Sentiment Analysis with SentimentDetector</div>', unsafe_allow_html=True)
+st.code('''
+from sparknlp.base import DocumentAssembler, Pipeline, Finisher
+from sparknlp.annotator import (
+    SentenceDetector,
+    Tokenizer,
+    Lemmatizer,
+    SentimentDetector
+)
+import pyspark.sql.functions as F
+
+# Step 1: Transforms raw texts to document annotation
+document_assembler = (
+    DocumentAssembler()
+    .setInputCol("text")
+    .setOutputCol("document")
+)
+
+# Step 2: Sentence Detection
+sentence_detector = SentenceDetector().setInputCols(["document"]).setOutputCol("sentence")
+
+# Step 3: Tokenization
+tokenizer = Tokenizer().setInputCols(["sentence"]).setOutputCol("token")
+
+# Step 4: Lemmatization
+lemmatizer = (
+    Lemmatizer()
+    .setInputCols("token")
+    .setOutputCol("lemma")
+    .setDictionary("/tmp/lemmas_small.txt", key_delimiter="->", value_delimiter="\\t")
+)
+
+# Step 5: Sentiment Detection
+sentiment_detector = (
+    SentimentDetector()
+    .setInputCols(["lemma", "sentence"])
+    .setOutputCol("sentiment_score")
+    .setDictionary("/tmp/default-sentiment-dict.txt", ",")
+)
+
+# Step 6: Finisher
+finisher = (
+    Finisher()
+    .setInputCols(["sentiment_score"])
+    .setOutputCols(["sentiment"])
+)
+
+# Define the pipeline
+pipeline = Pipeline(
+    stages=[
+        document_assembler,
+        sentence_detector,
+        tokenizer,
+        lemmatizer,
+        sentiment_detector,
+        finisher,
+    ]
+)
+
+# Create a spark Data Frame with an example sentence
+data = spark.createDataFrame(
+    [
+        ["The restaurant staff is really nice"]
+    ]
+).toDF("text")  # use the column name `text` defined in the pipeline as input
+
+# Fit-transform to get predictions
+result = pipeline.fit(data).transform(data).show(truncate=50)
+''', language='python')
+
+st.text("""
++-----------------------------------+----------+
+|                               text| sentiment|
++-----------------------------------+----------+
+|The restaurant staff is really nice|[positive]|
++-----------------------------------+----------+
+""")
+
+st.markdown("""
+<p>The code snippet demonstrates how to set up a pipeline in Spark NLP to perform sentiment analysis on text data using the SentimentDetector annotator. The resulting DataFrame contains the sentiment predictions.</p>
+""", unsafe_allow_html=True)
+
+# One-liner Alternative
+st.markdown('<div class="sub-title">One-liner Alternative</div>', unsafe_allow_html=True)
+st.markdown("""
+<div class="section">
+    <p>In October 2022, John Snow Labs released the open-source <code>johnsnowlabs</code> library that contains all the company products, open-source and licensed, under one common library. This simplified the workflow, especially for users working with more than one of the libraries (e.g., Spark NLP + Healthcare NLP). This new library is a wrapper on all of John Snow Lab’s libraries and can be installed with pip:</p>
+    <p><code>pip install johnsnowlabs</code></p>
+</div>
+""", unsafe_allow_html=True)
+
+st.markdown('<p>To run sentiment analysis with one line of code, we can simply:</p>', unsafe_allow_html=True)
+st.code("""
+# Import the NLP module which contains Spark NLP and NLU libraries
+from johnsnowlabs import nlp
+
+sample_text = "The restaurant staff is really nice"
+
+# Returns a pandas DataFrame, we select the desired columns
+nlp.load('en.sentiment').predict(sample_text, output_level='sentence')
+""", language='python')
+
+st.image('images/johnsnowlabs-sentiment-output.png', use_column_width='auto')
+
+st.markdown("""
+<p>This approach demonstrates how to use the <code>johnsnowlabs</code> library to perform sentiment analysis with a single line of code. The resulting DataFrame contains the sentiment predictions.</p>
+""", unsafe_allow_html=True)
+
+# Conclusion
+st.markdown("""
+<div class="section">
+    <h2>Conclusion</h2>
+    <p>In this app, we demonstrated how to use Spark NLP's SentimentDetector annotator to perform sentiment analysis on text data. These powerful tools enable users to efficiently process large datasets and identify sentiment, providing deeper insights for various applications. By integrating these annotators into your NLP pipelines, you can enhance text understanding, information extraction, and customer sentiment analysis.</p>
+</div>
+""", unsafe_allow_html=True)
+
+# References and Additional Information
+st.markdown('<div class="sub-title">For additional information, please check the following references.</div>', unsafe_allow_html=True)
+
+st.markdown("""
+<div class="section">
+    <ul>
+        <li>Documentation: <a href="https://nlp.johnsnowlabs.com/docs/en/annotators#sentimentdetector" target="_blank" rel="noopener">SentimentDetector</a></li>
+        <li>Python Docs : <a href="https://nlp.johnsnowlabs.com/api/python/reference/autosummary/sparknlp/annotator/sentiment/sentiment_detector/index.html" target="_blank" rel="noopener">SentimentDetector</a></li>
+        <li>Scala Docs: <a href="https://nlp.johnsnowlabs.com/api/com/johnsnowlabs/nlp/annotators/sda/pragmatic/SentimentDetector.html" target="_blank" rel="noopener">SentimentDetector</a></li>
+        <li>Example Notebook :&nbsp;<a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/training/english/classification/Sentiment%20Analysis.ipynb" target="_blank" rel="noopener">Sentiment Analysis</a></li>
+        <li>For extended examples of usage, see the <a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/tree/master/tutorials/Certification_Trainings/Public" target="_blank" rel="noopener">Spark NLP Workshop repository</a>.</li>
+    </ul>
+</div>
+""", unsafe_allow_html=True)
+
+st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True)
+st.markdown("""
+<div class="section">
+    <ul>
+        <li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li>
+        <li><a class="link" href="https://join.slack.com/t/spark-nlp/shared_invite/zt-198dipu77-L3UWNe_AJ8xqDk0ivmih5Q" target="_blank">Slack</a>: Live discussion with the community and team</li>
+        <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub</a>: Bug reports, feature requests, and contributions</li>
+        <li><a class="link" href="https://medium.com/spark-nlp" target="_blank">Medium</a>: Spark NLP articles</li>
+        <li><a class="link" href="https://www.youtube.com/channel/UCmFOjlpYEhxf_wJUDuz6xxQ/videos" target="_blank">YouTube</a>: Video tutorials</li>
+    </ul>
+</div>
+""", unsafe_allow_html=True)