pages/Workflow & Model Overview.py

import streamlit as st

# Custom CSS for better styling
st.markdown("""

    <style>

        .main-title {

            font-size: 36px;

            color: #4A90E2;

            font-weight: bold;

            text-align: center;

        }

        .sub-title {

            font-size: 24px;

            color: #4A90E2;

            margin-top: 20px;

        }

        .section {

            background-color: #f9f9f9;

            padding: 15px;

            border-radius: 10px;

            margin-top: 20px;

        }

        .section h2 {

            font-size: 22px;

            color: #4A90E2;

        }

        .section p, .section ul {

            color: #666666;

        }

        .link {

            color: #4A90E2;

            text-decoration: none;

        }

    </style>

""", unsafe_allow_html=True)

# Main Title
st.markdown('<div class="main-title">State-of-the-Art Named Entity Recognition with Spark NLP (Spanish)</div>', unsafe_allow_html=True)

# Introduction
st.markdown("""

<div class="section">

    <p>Named Entity Recognition (NER) is the task of identifying important words in a text and associating them with a category. For example, we may be interested in finding all the personal names in documents, or company names in news articles. Other examples include domain-specific uses such as identifying all disease names in a clinical text, or company trading codes in financial ones.</p>

    <p>NER can be implemented with many approaches. In this post, we introduce a deep learning-based method using the NerDL model. This approach leverages the scalability of Spark NLP with Python.</p>

</div>

""", unsafe_allow_html=True)

# Introduction to Spark NLP
st.markdown('<div class="sub-title">Introduction to Spark NLP</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Spark NLP is an open-source library maintained by John Snow Labs. It is built on top of Apache Spark and Spark ML and provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment.</p>

    <p>To install Spark NLP, you can simply use any package manager like conda or pip. For example, using pip you can simply run <code>pip install spark-nlp</code>. For different installation options, check the official <a href="https://nlp.johnsnowlabs.com/docs/en/install" target="_blank" class="link">documentation</a>.</p>

</div>

""", unsafe_allow_html=True)

# Using NerDL Model
st.markdown('<div class="sub-title">Using NerDL Model</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>The NerDL model in Spark NLP is a deep learning-based approach for NER tasks. It uses a Char CNNs - BiLSTM - CRF architecture that achieves state-of-the-art results in most datasets. The training data should be a labeled Spark DataFrame in the format of CoNLL 2003 IOB with annotation type columns.</p>

</div>

""", unsafe_allow_html=True)

# Setup Instructions
st.markdown('<div class="sub-title">Setup</div>', unsafe_allow_html=True)
st.markdown('<p>To install Spark NLP in Python, use your favorite package manager (conda, pip, etc.). For example:</p>', unsafe_allow_html=True)
st.code("""

pip install spark-nlp

pip install pyspark

""", language="bash")

st.markdown("<p>Then, import Spark NLP and start a Spark session:</p>", unsafe_allow_html=True)
st.code("""

import sparknlp



# Start Spark Session

spark = sparknlp.start()

""", language='python')

# Example Usage with NerDL Model in Italian
st.markdown('<div class="sub-title">Example Usage with NerDL Model in Italian</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Below is an example of how to set up and use the NerDL model for named entity recognition in Italian:</p>

</div>

""", unsafe_allow_html=True)
st.code('''

from sparknlp.base import *

from sparknlp.annotator import *

from pyspark.ml import Pipeline

from pyspark.sql.functions import col, expr, round, concat, lit



# Document Assembler

document_assembler = DocumentAssembler() \\

    .setInputCol("text") \\

    .setOutputCol("document")



# Tokenizer

tokenizer = Tokenizer() \\

    .setInputCols(["document"]) \\

    .setOutputCol("token")



# Word Embeddings

embeddings = WordEmbeddingsModel.pretrained('glove_840B_300', lang='xx') \\

    .setInputCols(["document", "token"]) \\

    .setOutputCol("embeddings")



# NerDL Model

ner_model = NerDLModel.pretrained('ner_wikiner_glove_840B_300', 'xx') \\

    .setInputCols(["document", "token", "embeddings"]) \\

    .setOutputCol("ner")



# NER Converter

ner_converter = NerConverter() \\

    .setInputCols(["document", "token", "ner"]) \\

    .setOutputCol("ner_chunk")



# Pipeline

pipeline = Pipeline(stages=[

    document_assembler,

    tokenizer,

    embeddings,

    ner_model,

    ner_converter

])



# Example sentence

example = """

En el corazón de Madrid, la Puerta del Sol es uno de los lugares más emblemáticos de España. Salvador Dalí, uno de los pintores más célebres del siglo XX, nació en Figueres en 1904. A lo largo de su carrera, Dalí creó obras como "La persistencia de la memoria" y "Cristo de San Juan de la Cruz", que se exhiben en museos de Barcelona y Nueva York.

El Museo del Prado, situado en el centro de Madrid, alberga obras maestras de artistas como Francisco de Goya y Diego Velázquez. El Rey Felipe VI, nacido Felipe Juan Pablo Alfonso de Todos los Santos de Borbón y de Grecia en Madrid en 1968, es el actual monarca de España.

La Sagrada Familia, diseñada por Antoni Gaudí, es un símbolo de la arquitectura modernista y se encuentra en Barcelona. Pablo Picasso, nacido en Málaga en 1881, es famoso por sus pinturas y esculturas como "Guernica" y "Las señoritas de Aviñón", que se exhiben en el Museo Reina Sofía de Madrid.

Sevilla es conocida por su catedral y el Alcázar. La Alhambra de Granada, un palacio y fortaleza de la época nazarí, es uno de los sitios más visitados del país. El Museo Guggenheim de Bilbao, diseñado por Frank Gehry, es un icono de la arquitectura contemporánea y alberga obras de artistas modernos.

Valencia acoge cada año Las Fallas, un festival en el que se queman grandes figuras de cartón. Zaragoza es famosa por la Basílica del Pilar y el río Ebro. San Sebastián, con su festival de cine, atrae a celebridades de todo el mundo.

En el ámbito literario, Miguel de Cervantes, autor de "Don Quijote de la Mancha", es una figura central de la literatura española. En la política, Pedro Sánchez, nacido en Madrid en 1972, es el actual presidente del Gobierno de España, mientras que el Partido Popular y el Partido Socialista Obrero Español son las principales fuerzas políticas del país.

España también es conocida por su gastronomía, con platos típicos como la paella de Valencia, el gazpacho andaluz y el jamón ibérico de Salamanca. La selección española de fútbol, campeona del mundo en 2010, es una de las más exitosas en la historia del deporte.

"""



data = spark.createDataFrame([[example]]).toDF("text")



# Transforming data

result = pipeline.fit(data).transform(data)



# Select the result, entity, and confidence columns

result.select(

    expr("explode(ner_chunk) as ner_chunk")

).select(

    col("ner_chunk.result").alias("result"),

    col("ner_chunk.metadata").getItem("entity").alias("entity"),

    concat(

        round((col("ner_chunk.metadata").getItem("confidence").cast("float") * 100), 2),

        lit("%")

    ).alias("confidence")

).show(truncate=False)

''', language="python")

st.text("""

+-------------------------------------------------------+------+----------+

|result                                                 |entity|confidence|

+-------------------------------------------------------+------+----------+

|Madrid                                                 |LOC   |82.31%    |

|Puerta del Sol                                         |LOC   |58.42%    |

|España                                                 |LOC   |88.22%    |

|Salvador Dalí                                          |PER   |68.36%    |

|Figueres                                               |LOC   |95.41%    |

|Dalí                                                   |PER   |99.26%    |

|San Juan de la Cruz                                    |LOC   |46.17%    |

|Barcelona                                              |LOC   |94.67%    |

|Nueva York                                             |LOC   |72.62%    |

|Museo del Prado                                        |LOC   |62.26%    |

|Madrid                                                 |LOC   |86.81%    |

|Francisco de Goya                                      |PER   |52.51%    |

|Diego Velázquez                                        |PER   |64.0%     |

|Rey Felipe VI                                          |PER   |62.7%     |

|Felipe Juan Pablo Alfonso de Todos los Santos de Borbón|PER   |58.58%    |

|Grecia                                                 |LOC   |73.8%     |

|Madrid                                                 |LOC   |93.21%    |

|España                                                 |LOC   |82.56%    |

|Sagrada Familia                                        |LOC   |44.24%    |

|Antoni Gaudí                                           |PER   |74.36%    |

+-------------------------------------------------------+------+----------+

""")

# Benchmark Section
st.markdown('<div class="sub-title">Benchmark</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>Evaluating the performance of NER models is crucial to understanding their effectiveness in real-world applications. Below are the benchmark results for the "ner_wikiner_glove_840B_300" model on <strong>Spanish</strong> text, focusing on various named entity categories. The metrics used include precision, recall, and F1-score, which are standard for evaluating classification models.</p>

</div>

""", unsafe_allow_html=True)
st.markdown("""

---

#### Classification Report



| Label | Precision | Recall | F1-Score | Support |

|-------|-----------|--------|----------|---------|

| B-LOC | 0.86      | 0.90   | 0.88     | 11963   |

| I-ORG | 0.82      | 0.78   | 0.80     | 1950    |

| I-LOC | 0.84      | 0.81   | 0.83     | 6162    |

| I-PER | 0.95      | 0.93   | 0.94     | 4678    |

| B-ORG | 0.83      | 0.77   | 0.80     | 2084    |

| B-PER | 0.93      | 0.94   | 0.94     | 7215    |



#### Averages



| Metric         | Precision | Recall | F1-Score | Support |

|----------------|-----------|--------|----------|---------|

| Micro Average  | 0.88      | 0.88   | 0.88     | 34052   |

| Macro Average  | 0.87      | 0.86   | 0.86     | 34052   |

| Weighted Avg   | 0.88      | 0.88   | 0.88     | 34052   |



#### Overall Metrics



- Processed 348,209 tokens with 24,505 phrases; found: 24,375 phrases; correct: 21,187.

- Accuracy (non-O): **85.54%**

- Overall Accuracy: **98.07%**

- Precision: **86.92%**

- Recall: **86.46%**

- F1 Score: **86.69**



#### Entity-Specific Metrics



| Entity | Precision | Recall | F1-Score | Instances |

|--------|-----------|--------|----------|-----------|

| LOC    | 85.32%    | 89.28% | 87.25    | 12518     |

| MISC   | 82.54%    | 67.78% | 74.43    | 2663      |

| ORG    | 81.85%    | 76.39% | 79.03    | 1945      |

| PER    | 92.66%    | 93.10% | 92.88    | 7249      |



---

""", unsafe_allow_html=True)

# Summary
st.markdown('<div class="sub-title">Summary</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>In this article, we discussed named entity recognition using a deep learning-based method with the "wikiner_840B_300" model for Italian. We introduced how to perform the task using the open-source Spark NLP library with Python, which can be used at scale in the Spark ecosystem. These methods can be used for natural language processing applications in various fields, including finance and healthcare.</p>

</div>

""", unsafe_allow_html=True)

# References
st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/api/python/reference/autosummary/sparknlp/annotator/ner/ner_dl/index.html" target="_blank" rel="noopener">NerDLModel</a> annotator documentation</li>

        <li>Model Used: <a class="link" href="https://sparknlp.org/2021/07/19/ner_wikiner_glove_840B_300_xx.html" target="_blank" rel="noopener">ner_wikiner_glove_840B_300</a></li>

        <li><a class="link" href="https://nlp.johnsnowlabs.com/recognize_entitie" target="_blank" rel="noopener">Visualization demos for NER in Spark NLP</a></li>

        <li><a class="link" href="https://www.johnsnowlabs.com/named-entity-recognition-ner-with-bert-in-spark-nlp/">Named Entity Recognition (NER) with BERT in Spark NLP</a></li>

    </ul>

</div>

""", unsafe_allow_html=True)

# Community & Support
st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li>

        <li><a class="link" href="https://join.slack.com/t/spark-nlp/shared_invite/zt-198dipu77-L3UWNe_AJf4Rqb3DaMb-7A" target="_blank">Slack Community</a>: Connect with other Spark NLP users</li>

        <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub Repository</a>: Source code and issue tracker</li>

    </ul>

</div>

""", unsafe_allow_html=True)