pages/Workflow & Model Overview.py

import streamlit as st

# Custom CSS for better styling
st.markdown("""

    <style>

        .main-title {

            font-size: 36px;

            color: #4A90E2;

            font-weight: bold;

            text-align: center;

        }

        .sub-title {

            font-size: 24px;

            color: #4A90E2;

            margin-top: 20px;

        }

        .section {

            background-color: #f9f9f9;

            padding: 15px;

            border-radius: 10px;

            margin-top: 20px;

        }

        .section h2 {

            font-size: 22px;

            color: #4A90E2;

        }

        .section p, .section ul {

            color: #666666;

        }

        .link {

            color: #4A90E2;

            text-decoration: none;

        }

    </style>

""", unsafe_allow_html=True)

# Main Title
st.markdown('<div class="main-title">State-of-the-Art Named Entity Recognition with Spark NLP (Scandinavian Languages)</div>', unsafe_allow_html=True)

# Introduction
st.markdown("""

<div class="section">

    <p>Named Entity Recognition (NER) is the task of identifying important words in a text and associating them with a category. For example, we may be interested in finding all the personal names in documents, or company names in news articles. Other examples include domain-specific uses such as identifying all disease names in a clinical text, or company trading codes in financial ones.</p>

    <p>NER can be implemented with many approaches. In this post, we introduce a deep learning-based method using the BertForTokenClassification model. This approach leverages the scalability of Spark NLP with Python.</p>

</div>

""", unsafe_allow_html=True)

# Introduction to Spark NLP
st.markdown('<div class="sub-title">Introduction to Spark NLP</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Spark NLP is an open-source library maintained by John Snow Labs. It is built on top of Apache Spark and Spark ML and provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment.</p>

    <p>To install Spark NLP, you can simply use any package manager like conda or pip. For example, using pip you can simply run <code>pip install spark-nlp</code>. For different installation options, check the official <a href="https://nlp.johnsnowlabs.com/docs/en/install" target="_blank" class="link">documentation</a>.</p>

</div>

""", unsafe_allow_html=True)

# Using BertForTokenClassification Model
st.markdown('<div class="sub-title">Using BertForTokenClassification Model</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>The <code>BertForTokenClassification</code> model in Spark NLP is a deep learning-based approach for NER tasks. It uses BERT embeddings for token classification that achieve state-of-the-art results in most datasets. This model loads BERT models with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.</p>

</div>

""", unsafe_allow_html=True)

# Setup Instructions
st.markdown('<div class="sub-title">Setup</div>', unsafe_allow_html=True)
st.markdown('<p>To install Spark NLP in Python, use your favorite package manager (conda, pip, etc.). For example:</p>', unsafe_allow_html=True)
st.code("""

pip install spark-nlp

pip install pyspark

""", language="bash")

st.markdown("<p>Then, import Spark NLP and start a Spark session:</p>", unsafe_allow_html=True)
st.code("""

import sparknlp



# Start Spark Session

spark = sparknlp.start()

""", language='python')

# Example Usage with BertForTokenClassification Model in Scandinavian Languages
st.markdown('<div class="sub-title">Example Usage with BertForTokenClassification Model in Scandinavian Languages</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Below is an example of how to set up and use the <code>BertForTokenClassification</code> model for named entity recognition in Scandinavian languages:</p>

</div>

""", unsafe_allow_html=True)
st.code('''

from sparknlp.base import *

from sparknlp.annotator import *

from pyspark.ml import Pipeline



# Document Assembler

document_assembler = DocumentAssembler()\\

    .setInputCol("text")\\

    .setOutputCol("document")



# Sentence Detector

sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "xx")\\

    .setInputCols(["document"])\\

    .setOutputCol("sentence")



# Tokenizer

tokenizer = Tokenizer()\\

    .setInputCols(["sentence"])\\

    .setOutputCol("token")



# Token Classifier

tokenClassifier = BertForTokenClassification.pretrained("bert_token_classifier_scandi_ner", "xx")\\

    .setInputCols(["token", "document"])\\

    .setOutputCol("ner")



# NER Converter

ner_converter = NerConverter()\\

    .setInputCols(["sentence", "token", "ner"])\\

    .setOutputCol("ner_chunk")



# Pipeline

nlpPipeline = Pipeline(stages=[documentAssembler, sentenceDetector, tokenizer, tokenClassifier, ner_converter])



# Example sentence

text = combined_text = """

Danish:

Lars Andersen arbejder som softwareudvikler i København. Han bor sammen med sin familie i en charmerende lejlighed i Vesterbro. I weekenderne nyder Lars at tage på cykelture i Frederiksberg Have og besøge museer som Statens Museum for Kunst. Han er også en stor fan af fodbold og følger med i kampe fra FC København og Brøndby IF.



Bokmål (Norwegian):

Anna Johansen jobber som arkitekt i Oslo. Hun bor i en moderne leilighet i sentrum og elsker å tilbringe helgene med å utforske nye restauranter og kafeer. Anna har en stor interesse for kunst og besøker ofte Nasjonalmuseet for å se på utstillinger. Hun er også en ivrig leser og tilbringer tid på biblioteket.



Nynorsk (Norwegian):

Martin Berg er landskapsarkitekt i Bergen. Han bur i ein historisk bygning i sentrum og nyt å gå på turar i nærområdet i helgene. Martin er opptatt av bærekraft og brukar mykje tid på å arbeide med grøne prosjekt og parker. Han er også engasjert i lokalt frivillig arbeid og deltek i arrangement på det lokale kulturhuset.



Swedish:

Emma Svensson arbetar som lärare i Stockholm. Hon bor i en charmig lägenhet i Södermalm och tillbringar sina helger med att utforska olika kulturevenemang och utställningar. Emma är en stor beundrare av svensk litteratur och besöker ofta biblioteket för att läsa de senaste romanerna. Hon är också en aktiv medlem i en lokal bokklubb.



Icelandic:

Jón Guðmundsson vinnur sem læknir í Reykjavík. Hann býr í fallegri íbúð í miðborginni og nýtur þess að fara í gönguferðir í Esju og heimsækja listasafn Reykjavíkur um helgar. Jón er áhugasamur um íslenska sagnfræði og eyðir oft tíma í að lesa gamla bókmenntir. Hann tekur einnig þátt í samfélagsverkefnum sem stuðla að menningarlegri fræðslu.



Faroese:

Rúna Hansen arbeiðir sum lærari í Tórshavn. Hon býr í einum rúmligum íbúð í miðbýin og nýtur at ferðast runt í náttúruni í frítíðini. Rúna hevur stóran áhuga fyri føroyskum bókmentum og fer ofta til bókasavnið at lesa nýggjastu útgávurnar. Hon er eisini aktiv í einum lokala bókaklubi.

"""



# Transforming data

empty_data = spark.createDataFrame([[""]]).toDF("text")

model = nlpPipeline.fit(empty_data)

result = model.transform(spark.createDataFrame([[text]]).toDF("text"))



# Select the result, entity, and confidence columns

result.select(

    expr("explode(ner_chunk) as ner_chunk")

).select(

    col("ner_chunk.result").alias("chunk"),

    col("ner_chunk.metadata").getItem("entity").alias("ner_label")

).show(truncate=False)

''', language="python")

st.text("""

+------------------------+---------+

|chunk                   |ner_label|

+------------------------+---------+

|Danish                  |MISC     |

|Lars Andersen           |PER      |

|København               |LOC      |

|Vesterbro               |LOC      |

|Lars                    |PER      |

|Frederiksberg Have      |LOC      |

|Statens Museum for Kunst|ORG      |

|FC København            |ORG      |

|Brøndby IF              |ORG      |

|Bokmål                  |MISC     |

|Norwegian               |MISC     |

|Anna Johansen           |PER      |

|Oslo                    |LOC      |

|Anna                    |PER      |

|Nasjonalmuseet          |ORG      |

|Nynorsk                 |MISC     |

|Norwegian               |ORG      |

|Martin Berg             |PER      |

|Bergen                  |LOC      |

|Martin                  |PER      |

+------------------------+---------+

""")

# Available Models in Spark NLP
st.markdown('<div class="sub-title">Available Models in Spark NLP</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Spark NLP provides a variety of pre-trained models for different NLP tasks. These models are designed to be used out-of-the-box and cover a wide range of applications, including NER, sentiment analysis, text classification, and more. You can explore and choose from the available models at the <a href="https://sparknlp.org/models" target="_blank" class="link">Spark NLP Models Hub</a>.</p>

</div>

""", unsafe_allow_html=True)

# Benchmark Section
st.markdown('<div class="sub-title">Benchmark</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Evaluating the performance of NER models is crucial to understanding their effectiveness in real-world applications. Below are the benchmark results for the "bert_token_classifier_scandi_ner" model on 6 Scandinavian languages, focusing on various named entity categories. The metrics used include precision, recall, and F1-score, which are standard for evaluating classification models.</p>

</div>

""", unsafe_allow_html=True)
st.markdown("""

---

#### Classification Report



| Language   | F1 Score |

|------------|----------|

| Danish     | 0.8744   |

| Bokmål     | 0.9106   |

| Nynorsk    | 0.9042   |

| Swedish    | 0.8837   |

| Icelandic  | 0.8861   |

| Faroese    | 0.9022   |



---

""", unsafe_allow_html=True)

# Summary
st.markdown('<div class="sub-title">Summary</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>In this article, we discussed named entity recognition using a deep learning-based method with the "bert_token_classifier_scandi_ner" model for Scandinavian languages. We introduced how to perform the task using the open-source Spark NLP library with Python, which can be used at scale in the Spark ecosystem. These methods can be used for natural language processing applications in various fields, including finance and healthcare.</p>

</div>

""", unsafe_allow_html=True)

# References
st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/api/python/reference/autosummary/sparknlp/annotator/classifier_dl/bert_for_token_classification/index.html" target="_blank" rel="noopener">BertForTokenClassification </a> annotator documentation</li>

        <li>Model Used: <a class="link" href="https://sparknlp.org/2021/12/09/bert_token_classifier_scandi_ner_xx.html" target="_blank" rel="noopener">bert_token_classifier_scandi_ner</a></li>

        <li><a class="link" href="https://nlp.johnsnowlabs.com/recognize_entitie" target="_blank" rel="noopener">Visualization demos for NER in Spark NLP</a></li>

        <li><a class="link" href="https://www.johnsnowlabs.com/named-entity-recognition-ner-with-bert-in-spark-nlp/">Named Entity Recognition (NER) with BERT in Spark NLP</a></li>

    </ul>

</div>

""", unsafe_allow_html=True)

# Community & Support
st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li>

        <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub Repository</a>: Report issues or contribute</li>

        <li><a class="link" href="https://forum.johnsnowlabs.com/" target="_blank">Community Forum</a>: Ask questions, share ideas, and get support</li>

    </ul>

</div>

""", unsafe_allow_html=True)