pages/Workflow & Model Overview.py

import streamlit as st

# Custom CSS for better styling
st.markdown("""

    <style>

        .main-title {

            font-size: 36px;

            color: #4A90E2;

            font-weight: bold;

            text-align: center;

        }

        .sub-title {

            font-size: 24px;

            color: #4A90E2;

            margin-top: 20px;

        }

        .section {

            background-color: #f9f9f9;

            padding: 15px;

            border-radius: 10px;

            margin-top: 20px;

        }

        .section h2 {

            font-size: 22px;

            color: #4A90E2;

        }

        .section p, .section ul {

            color: #666666;

        }

        .link {

            color: #4A90E2;

            text-decoration: none;

        }

    </style>

""", unsafe_allow_html=True)

# Introduction
st.markdown('<div class="main-title">Detecting Sarcasm with Spark NLP</div>', unsafe_allow_html=True)

st.markdown("""

<div class="section">

    <p>Welcome to the Spark NLP Sarcasm Detection Demo App! Detecting sarcasm in text is crucial for understanding sentiment and context. This app utilizes advanced natural language processing techniques to identify instances of sarcasm with high accuracy.</p>

    <p>This demo showcases the use of Spark NLP's ClassifierDLModel pretrained on Universal Sentence Encoder embeddings to classify text as sarcastic or normal.</p>

</div>

""", unsafe_allow_html=True)

st.image('images/sarcasm.jpg', use_column_width='auto')

# About Sarcasm Detection
st.markdown('<div class="sub-title">About Sarcasm Detection</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Sarcasm detection involves identifying language that is contrary to the literal meaning, often used to convey humor or irony. It plays a crucial role in sentiment analysis and understanding textual context.</p>

    <p>Effective sarcasm detection models improve the accuracy of sentiment analysis and help in better understanding user intent.</p>

</div>

""", unsafe_allow_html=True)

# Using ClassifierDLModel in Spark NLP
st.markdown('<div class="sub-title">Using ClassifierDLModel in Spark NLP</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>The ClassifierDLModel in Spark NLP utilizes deep learning techniques to classify text into predefined categories, in this case, detecting sarcasm or normal text. It is trained on Universal Sentence Encoder embeddings for robust performance.</p>

    <p>For more details, refer to the <a class="link" href="https://sparknlp.org/docs/en/annotators#classifierdl" target="_blank" rel="noopener">ClassifierDLModel documentation</a> on Spark NLP's official website.</p>

</div>

""", unsafe_allow_html=True)

st.markdown('<h2 class="sub-title">Example Usage in Python</h2>', unsafe_allow_html=True)
st.markdown('<p>Here’s how you can implement sarcasm detection using the ClassifierDLModel in Spark NLP:</p>', unsafe_allow_html=True)

# Setup Instructions
st.markdown('<div class="sub-title">Setup</div>', unsafe_allow_html=True)
st.markdown('<p>To use Spark NLP for sarcasm detection, follow these setup instructions:</p>', unsafe_allow_html=True)
st.code("""

pip install spark-nlp

pip install pyspark

""", language="bash")

st.markdown("<p>Then, import Spark NLP and start a Spark session:</p>", unsafe_allow_html=True)
st.code("""

import sparknlp



# Start Spark Session

spark = sparknlp.start()

""", language='python')

# Sentiment Analysis Example
st.markdown('<div class="sub-title">Example Usage: Sarcasm Detection with ClassifierDLModel</div>', unsafe_allow_html=True)
st.code('''

from sparknlp.base import DocumentAssembler

from sparknlp.annotator import UniversalSentenceEncoder, ClassifierDLModel

from pyspark.ml import Pipeline



# Step 1: DocumentAssembler

document_assembler = DocumentAssembler() \\

    .setInputCol("text") \\

    .setOutputCol("document")



# Step 2: UniversalSentenceEncoder

use = UniversalSentenceEncoder.pretrained() \\

    .setInputCols(["document"]) \\

    .setOutputCol("sentence_embeddings")



# Step 3: ClassifierDLModel for Sarcasm Detection

sentimentdl = ClassifierDLModel.pretrained('classifierdl_use_sarcasm') \\

    .setInputCols(["sentence_embeddings"]) \\

    .setOutputCol("sentiment")



# Define the NLP Pipeline

nlpPipeline = Pipeline(stages=[document_assembler, use, sentimentdl])



# Example Text

text = "Oh, great! Another meeting scheduled for Friday afternoon. That's just what I needed."



# Process the text through the pipeline

result = nlpPipeline.fit(spark.createDataFrame([[text]]).toDF("text")).transform(spark.createDataFrame([[text]]).toDF("text")).select('text', 'sentiment.result').show(truncate=False)

''', language='python')

st.text("""

+-------------------------------------------------------------------------------------+---------+

|text                                                                                 |result   |

+-------------------------------------------------------------------------------------+---------+

|Oh, great! Another meeting scheduled for Friday afternoon. That's just what I needed.|[sarcasm]|

+-------------------------------------------------------------------------------------+---------+

""")

st.markdown("""

<p>The above example demonstrates how to use Spark NLP's ClassifierDLModel to detect sarcasm in text using Universal Sentence Encoder embeddings.</p>

""", unsafe_allow_html=True)

# Benchmarking
st.markdown('<div class="sub-title">Benchmarking</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Performance metrics of the sarcasm detection model:</p>

    <pre>

    precision    recall  f1-score   support



    normal       0.98      0.89      0.93       495

    sarcasm      0.60      0.91      0.73        93



    accuracy                           0.89       588

    macro avg      0.79      0.90      0.83       588

    weighted avg   0.92      0.89      0.90       588

</div>

""", unsafe_allow_html=True)

# Conclusion
st.markdown("""

<div class="section">

    <h2>Conclusion</h2>

    <p>In this app, we explored how Spark NLP's ClassifierDLModel can be used to detect sarcasm in text. This capability enhances sentiment analysis and contextual understanding in various applications, improving the accuracy of natural language processing tasks.</p>

</div>

""", unsafe_allow_html=True)

# References and Additional Information
st.markdown('<div class="sub-title">For additional information, please check the following references.</div>', unsafe_allow_html=True)

st.markdown("""

<div class="section">

        <ul>

            <li>Documentation :&nbsp;<a class="link" href="https://nlp.johnsnowlabs.com/docs/en/annotators#classifierdlmodel" target="_blank" rel="noopener">ClassifierDLModel</a></li>

            <li>Python Docs :&nbsp;<a class="link" href="https://nlp.johnsnowlabs.com/api/python/reference/autosummary/sparknlp.annotator.ClassifierDLModel.html" target="_blank" rel="noopener">ClassifierDLModel</a></li>

            <li>Model Used :&nbsp;<a class="link" href="https://sparknlp.org/2021/01/09/classifierdl_use_sarcasm_en.html" target="_blank" rel="noopener">classifierdl_use_sarcasm</a></li>

        </ul>

    </div>

""", unsafe_allow_html=True)

st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li>

        <li><a class="link" href="https://join.slack.com/t/spark-nlp/shared_invite/zt-198dipu77-L3UWNe_AJ8xqDk0ivmih5Q" target="_blank">Slack</a>: Live discussion with the community and team</li>

        <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub</a>: Bug reports, feature requests, and contributions</li>

        <li><a class="link" href="https://medium.com/spark-nlp" target="_blank">Medium</a>: Spark NLP articles</li>

        <li><a class="link" href="https://www.youtube.com/channel/UCmFOjlpYEhxf_wJUDuz6xxQ/videos" target="_blank">YouTube</a>: Video tutorials</li>

    </ul>

</div>

""", unsafe_allow_html=True)