pages/Workflow & Model Overview.py

import streamlit as st

# Custom CSS for better styling
st.markdown("""
    <style>
        .main-title {
            font-size: 36px;
            color: #4A90E2;
            font-weight: bold;
            text-align: center;
        }
        .sub-title {
            font-size: 24px;
            color: #4A90E2;
            margin-top: 20px;
        }
        .section {
            background-color: #f9f9f9;
            padding: 15px;
            border-radius: 10px;
            margin-top: 20px;
        }
        .section h2 {
            font-size: 22px;
            color: #4A90E2;
        }
        .section p, .section ul {
            color: #666666;
        }
        .link {
            color: #4A90E2;
            text-decoration: none;
        }
    </style>
""", unsafe_allow_html=True)

# Main Title
st.markdown('<div class="main-title">The Ultimate Guide to Named Entity Recognition with Spark NLP</div>', unsafe_allow_html=True)

# Introduction
st.markdown("""
<div class="section">
    <p>Named Entity Recognition (NER) is the task of identifying important words in a text and associating them with a category. For example, we may be interested in finding all the personal names in documents, or company names in news articles. Other examples include domain-specific uses such as identifying all disease names in a clinical text, or company trading codes in financial ones.</p>
    <p>NER can be implemented with many approaches. In this post, we introduce two methods: using a manually crafted list of entities (gazetteer) or regular expressions, and using deep learning with the NerDL model. Both approaches leverage the scalability of Spark NLP with Python.</p>
</div>
""", unsafe_allow_html=True)

st.image("images/ner.png")

# Introduction to Spark NLP
st.markdown('<div class="sub-title">Introduction to Spark NLP</div>', unsafe_allow_html=True)
st.markdown("""
<div class="section">
    <p>Spark NLP is an open-source library maintained by John Snow Labs. It is built on top of Apache Spark and Spark ML and provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment.</p>
    <p>To install Spark NLP, you can simply use any package manager like conda or pip. For example, using pip you can simply run <code>pip install spark-nlp</code>. For different installation options, check the official <a href="https://nlp.johnsnowlabs.com/docs/en/install" target="_blank" class="link">documentation</a>.</p>
</div>
""", unsafe_allow_html=True)

# Using NerDL Model
st.markdown('<div class="sub-title">Using NerDL Model</div>', unsafe_allow_html=True)
st.markdown("""
<div class="section">
    <p>The NerDL model in Spark NLP is a deep learning-based approach for NER tasks. It uses a Char CNNs - BiLSTM - CRF architecture that achieves state-of-the-art results in most datasets. The training data should be a labeled Spark DataFrame in the format of CoNLL 2003 IOB with annotation type columns.</p>
</div>
""", unsafe_allow_html=True)

# Setup Instructions
st.markdown('<div class="sub-title">Setup</div>', unsafe_allow_html=True)
st.markdown('<p>To install Spark NLP in Python, use your favorite package manager (conda, pip, etc.). For example:</p>', unsafe_allow_html=True)
st.code("""
pip install spark-nlp
pip install pyspark
""", language="bash")

st.markdown("<p>Then, import Spark NLP and start a Spark session:</p>", unsafe_allow_html=True)
st.code("""
import sparknlp

# Start Spark Session
spark = sparknlp.start()
""", language='python')

# Example Usage with NerDL Model
st.markdown('<div class="sub-title">Example Usage with NerDL Model</div>', unsafe_allow_html=True)
st.markdown("""
<div class="section">
    <p>Below is an example of how to set up and use the NerDL model for named entity recognition:</p>
</div>
""", unsafe_allow_html=True)
st.code('''
from sparknlp.base import *
from sparknlp.annotator import *
from pyspark.ml import Pipeline

# Document Assembler
document_assembler = DocumentAssembler() \\
    .setInputCol("text") \\
    .setOutputCol("document")

# Sentence Detector
sentence_detector = SentenceDetector() \\
    .setInputCols(["document"]) \\
    .setOutputCol("sentence")

# Tokenizer
tokenizer = Tokenizer() \\
    .setInputCols(["sentence"]) \\
    .setOutputCol("token")

# Word Embeddings
embeddings = WordEmbeddingsModel.pretrained() \\
    .setInputCols(["sentence", "token"]) \\
    .setOutputCol("bert")

# NerDL Model
ner_tagger = NerDLModel.pretrained() \\
    .setInputCols(["sentence", "token", "bert"]) \\
    .setOutputCol("ner")

# Pipeline
pipeline = Pipeline().setStages([
    document_assembler,
    sentence_detector,
    tokenizer,
    embeddings,
    ner_tagger
])

# Example sentence
example = """
William Henry Gates III (born October 28, 1955) is an American business magnate, software developer, investor, and philanthropist. 
He is best known as the co-founder of Microsoft Corporation. Throughout his career at Microsoft, Gates held various positions, 
including chairman, chief executive officer (CEO), president, and chief software architect. He was also the largest individual 
shareholder until May 2014.

Gates is recognized as one of the foremost entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. 
Born and raised in Seattle, Washington, he co-founded Microsoft with childhood friend Paul Allen in 1975. Initially established 
in Albuquerque, New Mexico, Microsoft grew to become the world’s largest personal computer software company.

Gates led Microsoft as chairman and CEO until January 2000, when he stepped down as CEO but continued as chairman and chief 
software architect. During the late 1990s, Gates faced criticism for business practices considered anti-competitive, an opinion 
upheld by numerous court rulings.

In June 2006, Gates announced his transition to a part-time role at Microsoft while dedicating full time to the Bill & Melinda Gates 
Foundation, a private charitable organization he established with his wife, Melinda Gates, in 2000. Gates gradually transferred 
his responsibilities to Ray Ozzie and Craig Mundie and stepped down as chairman of Microsoft in February 2014. He then assumed 
the role of technology adviser to support the newly appointed CEO, Satya Nadella.
"""

data = spark.createDataFrame([[example]]).toDF("text")

# Transforming data
result = pipeline.fit(data).transform(data)
result.select("ner.result").show(truncate=False)
''', language="python")

st.text("""
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|result                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[O, B-PER, I-PER, I-PER, I-PER, O, O, O, O, O, O, O, O, O, B-MISC, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-ORG, I-ORG, O, O, O, O, O, B-ORG, O, B-PER, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-LOC, O, O, O, B-LOC, O, B-LOC, O, B-PER, O, B-ORG, O, O, O, B-PER, I-PER, O, O, O, O, B-LOC, O, B-LOC, I-LOC, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-PER, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-PER, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-PER, O, O, O, O, O, O, O, O, O, O, O, B-ORG, O, O, O, O, O, B-ORG, I-ORG, I-ORG, I-ORG, I-ORG, O, O, O, O, O, O, O, O, O, O, O, B-PER, I-PER, O, O, O, O, O, O, O, O, O, O, B-PER, I-PER, O, B-PER, I-PER, O, O, O, O, O, O, O, B-ORG, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-PER, I-PER, O]|
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
""")

# Using EntityRuler Annotator
st.markdown('<div class="sub-title">Using EntityRuler Annotator</div>', unsafe_allow_html=True)
st.markdown("""
<div class="section">
    <p>In addition to the deep learning-based approach, Spark NLP also supports a rule-based method for NER using the EntityRuler annotator. This method involves using a gazetteer or regular expressions to identify entities in the text.</p>
</div>
""", unsafe_allow_html=True)

# Example Usage with EntityRuler
st.markdown('<div class="sub-title">Example Usage with EntityRuler</div>', unsafe_allow_html=True)
st.markdown("""
<div class="section">
    <p>For the NER tasks based on gazetteer list, we will use the EntityRuler annotator, which has both Approach and Model versions.</p>
    <p>As this annotator consists in finding the entities based in a list of desired names, the EntityRulerApproach annotator will store the given list in the EntityRulerModel parameters. All we need is a JSON or CSV file with the list of names or regex rules. For example, we may use the following entities.json file:</p>
</div>
""", unsafe_allow_html=True)
st.code("""
[
    {
        "label": "PERSON",
        "patterns": [
           "John",
           "John Snow"
       ]
    },
    {
        "label": "PERSON",
        "patterns": [
            "Eddard",
            "Eddard Stark"
        ]
    },
    {
        "label": "LOCATION",
        "patterns": [
            "Winterfell"
        ]
    },
    {
        "label": "DATE",
        "patterns": [
            "[0-9]{4}-[0-9]{1,2}-[0-9]{1,2}"
        ],
        "regex": true
    }
]
""", language="json")

# Pipeline Setup
st.markdown('<div class="sub-title">Pipeline Setup</div>', unsafe_allow_html=True)
st.code("""
from sparknlp.base import DocumentAssembler, Pipeline
from sparknlp.annotator import EntityRulerApproach, Tokenizer

document_assembler = DocumentAssembler() \\
    .setInputCol("text") \\
    .setOutputCol("document")

tokenizer = Tokenizer() \\
    .setInputCols(["document"]) \\
    .setOutputCol("token")

entity_ruler = EntityRulerApproach() \\
    .setInputCols(["document", "token"]) \\
    .setOutputCol("entity") \\
    .setPatternsResource("entities.json")

pipeline = Pipeline(stages=[document_assembler, tokenizer, entity_ruler])
""", language="python")

# Example Sentences
st.markdown('<div class="sub-title">Example Sentences</div>', unsafe_allow_html=True)
st.code('''
example = """Game of Thrones was released in 2011-04-17.
Lord Eddard Stark was the head of House Stark.
John Snow lives in Winterfell."""

data = spark.createDataFrame([[example]]).toDF("text")
pipeline_model = pipeline.fit(data)
''', language="python")

# Save and Load Model
st.markdown('<div class="sub-title">We can Save and Load Model for future use (optional)</div>', unsafe_allow_html=True)
st.code("""
pipeline_model.stages[-1].write().overwrite().save('my_entityruler')

entity_ruler = EntityRulerModel.load("my_entityruler") \\
    .setInputCols(["document", "token"]) \\
    .setOutputCol("entity")
""", language="python")

# Result Visualization
st.markdown('<div class="sub-title">Result Visualization</div>', unsafe_allow_html=True)
st.code("""
result = pipeline_model.transform(data)
import pyspark.sql.functions as F

result.select(
    F.explode(F.col("entity")).alias("entity")
).select(
    F.col("entity.result").alias("keyword"),
    F.col("entity.metadata").alias("metadata")
).select(
    F.col("keyword"),
    F.expr("metadata['entity']").alias("label")
).show()
""", language="python")

st.text("""
+------------+--------+
|     keyword|   label|
+------------+--------+
|  2011-04-17|    DATE|
|Eddard Stark|  PERSON|
|   John Snow|  PERSON|
|  Winterfell|LOCATION|
+------------+--------+
""")

# Non-English Languages
st.markdown('<div class="sub-title">Non-English Languages</div>', unsafe_allow_html=True)

st.markdown("""
<p>The EntityRuler annotator utilizes the Aho-Corasick algorithm, which may not handle languages with unique characters or alphabets effectively. For example:</p>
<div class="section">
<ul>
    <li>Spanish includes the ñ character.</li>
    <li>Portuguese uses ç.</li>
    <li>Many languages have accented characters (á, ú, ê, etc.).</li>
</ul>
</div>
<p>To accommodate these characters, use the <code>.setAlphabetResource</code> parameter.</p>
<p>When a character is missing from the alphabet, you might encounter errors like this:</p>
<pre><code>Py4JJavaError: An error occurred while calling o69.fit.
: java.lang.UnsupportedOperationException: Char ú not found on alphabet. Please check alphabet</code></pre>
<p>To define a custom alphabet, create a text file (e.g., <code>custom_alphabet.txt</code>) with all required characters:</p>
""", unsafe_allow_html=True)
st.code("""
abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
áúéêçñ
ÁÚÉÊÇÑ
""")

st.markdown("""
<p>Alternatively, you can use predefined alphabets for common languages. For instance, for Spanish:</p>
""", unsafe_allow_html=True)
st.code("""
entity_ruler = (
    EntityRulerApproach()
    .setInputCols(["sentence"])
    .setOutputCol("entity")
    .setPatternsResource("locations.json")
    .setAlphabetResource("Spanish")
)
""")

# Summary
st.markdown('<div class="sub-title">Summary</div>', unsafe_allow_html=True)
st.markdown("""
<div class="section">
    <p>In this article, we talked about named entity recognition using both deep learning-based and rule-based methods. We introduced how to perform the task using the open-source Spark NLP library with Python, which can be used at scale in the Spark ecosystem. These methods can be used for natural language processing applications in various fields, including finance and healthcare.</p>
</div>
""", unsafe_allow_html=True)

# References
st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
st.markdown("""
<div class="section">
    <ul>
        <li><a class="link" href="https://nlp.johnsnowlabs.com/docs/en/annotators#entityruler" target="_blank" rel="noopener">EntityRuler</a> annotator documentation</li>
        <li>Python Docs: <a class="link" href="https://nlp.johnsnowlabs.com/api/python/reference/autosummary/sparknlp/annotator/er/entity_ruler/index.html#sparknlp.annotator.er.entity_ruler.EntityRulerApproach" target="_blank" rel="noopener">EntityRulerApproach</a>, <a class="link" href="https://nlp.johnsnowlabs.com/api/python/reference/autosummary/sparknlp/annotator/er/entity_ruler/index.html#sparknlp.annotator.er.entity_ruler.EntityRulerModel">EntityRulerModel</a></li>
        <li>Scala Docs: <a class="link" href="https://nlp.johnsnowlabs.com/api/com/johnsnowlabs/nlp/annotators/er/EntityRulerApproach.html" target="_blank" rel="noopener">EntityRulerApproach</a>, <a class="link" href="https://nlp.johnsnowlabs.com/api/com/johnsnowlabs/nlp/annotators/er/EntityRulerModel.html">EntityRulerModel</a></li>
        <li><a class="link" href="https://nlp.johnsnowlabs.com/recognize_entitie" target="_blank" rel="noopener">Visualization demos for NER in Spark NLP</a></li>
        <li><a class="link" href="https://www.johnsnowlabs.com/named-entity-recognition-ner-with-bert-in-spark-nlp/">Named Entity Recognition (NER) with BERT in Spark NLP</a></li>
    </ul>
</div>
""", unsafe_allow_html=True)

st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True)
st.markdown("""
<div class="section">
    <ul>
        <li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li>
        <li><a class="link" href="https://join.slack.com/t/spark-nlp/shared_invite/zt-198dipu77-L3UWNe_AJ8xqDk0ivmih5Q" target="_blank">Slack</a>: Live discussion with the community and team</li>
        <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub</a>: Bug reports, feature requests, and contributions</li>
        <li><a class="link" href="https://medium.com/spark-nlp" target="_blank">Medium</a>: Spark NLP articles</li>
        <li><a class="link" href="https://www.youtube.com/channel/UCmFOjlpYEhxf_wJUDuz6xxQ/videos" target="_blank">YouTube</a>: Video tutorials</li>
    </ul>
</div>
""", unsafe_allow_html=True)