pages/Workflow & Model Overview.py

import streamlit as st

# Custom CSS for better styling
st.markdown("""

    <style>

        .main-title {

            font-size: 36px;

            color: #4A90E2;

            font-weight: bold;

            text-align: center;

        }

        .sub-title {

            font-size: 24px;

            color: #4A90E2;

            margin-top: 20px;

        }

        .section {

            background-color: #f9f9f9;

            padding: 15px;

            border-radius: 10px;

            margin-top: 20px;

        }

        .section h2 {

            font-size: 22px;

            color: #4A90E2;

        }

        .section p, .section ul {

            color: #666666;

        }

        .link {

            color: #4A90E2;

            text-decoration: none;

        }

        .benchmark-table {

            width: 100%;

            border-collapse: collapse;

            margin-top: 20px;

        }

        .benchmark-table th, .benchmark-table td {

            border: 1px solid #ddd;

            padding: 8px;

            text-align: left;

        }

        .benchmark-table th {

            background-color: #4A90E2;

            color: white;

        }

        .benchmark-table td {

            background-color: #f2f2f2;

        }

    </style>

""", unsafe_allow_html=True)

# Main Title
st.markdown('<div class="main-title">Bilingual Named Entity Recognition Model: Hindi and English</div>', unsafe_allow_html=True)

# Description
st.markdown("""

<div class="section">

    <p>This model, <strong>bert_token_classifier_hi_en_ner</strong>, was imported from Hugging Face to perform Named Entity Recognition (NER) with mixed Hindi-English texts. It is provided by the LinCE repository and is designed to identify named entities in texts containing both Hindi and English.</p>

</div>

""", unsafe_allow_html=True)

# What is Entity Recognition
st.markdown('<div class="sub-title">What is Entity Recognition?</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p><strong>Entity Recognition</strong> is a task in Natural Language Processing (NLP) that involves identifying and classifying named entities in text into predefined categories. This model focuses on detecting terminology related to restaurants, which is essential for understanding and analyzing restaurant reviews, menus, and related content.</p>

</div>

""", unsafe_allow_html=True)

# Model Importance and Applications
st.markdown('<div class="sub-title">Model Importance and Applications</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>The <strong>bert_token_classifier_hi_en_ner</strong> model is a powerful tool for handling mixed Hindi-English texts. Its applications include:</p>

    <ul>

        <li><strong>Multilingual Text Analysis:</strong> Efficiently processes and analyzes texts that contain both Hindi and English, making it suitable for a diverse range of documents.</li>

        <li><strong>Entity Recognition in Bilingual Contexts:</strong> Identifies and classifies named entities in contexts where Hindi and English are mixed, enhancing understanding in multilingual environments.</li>

        <li><strong>Cross-Language Information Extraction:</strong> Extracts valuable information from documents that are not confined to a single language, providing insights in both Hindi and English.</li>

        <li><strong>Data Enrichment:</strong> Improves datasets by adding valuable entity information from bilingual texts, useful for multilingual data systems and research.</li>

    </ul>

    <p>Why use the <strong>bert_token_classifier_hi_en_ner</strong> model?</p>

    <ul>

        <li><strong>Pre-trained for Mixed Hindi-English:</strong> Specifically trained to handle mixed-language texts, ensuring accurate recognition of entities in both languages.</li>

        <li><strong>High Accuracy:</strong> Delivers reliable results in identifying entities, even in complex bilingual contexts.</li>

        <li><strong>Versatility:</strong> Applicable to a wide range of documents and texts, enhancing its utility in various scenarios.</li>

    </ul>

</div>

""", unsafe_allow_html=True)

# Predicted Entities
st.markdown('<div class="sub-title">Predicted Entities</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>The model identifies and classifies the following entities:</p>

    <ul>

        <li><strong>ORGANISATION</strong>: Names of organizations or companies.</li>

        <li><strong>PERSON</strong>: Names of people.</li>

        <li><strong>PLACE</strong>: Names of locations or places.</li>

    </ul>

</div>

""", unsafe_allow_html=True)

# How to Use the Model
st.markdown('<div class="sub-title">How to Use the Model</div>', unsafe_allow_html=True)
st.code('''

from sparknlp.base import *

from sparknlp.annotator import *

from pyspark.ml import Pipeline

from pyspark.sql.functions import col, expr



# Define the pipeline stages

document_assembler = DocumentAssembler() \\

    .setInputCol('text') \\

    .setOutputCol('document')



sentence_detector = SentenceDetector() \\

    .setInputCols(['document']) \\

    .setOutputCol('sentence')



tokenizer = Tokenizer() \\

    .setInputCols(['sentence']) \\

    .setOutputCol('token')



tokenClassifier_loaded = BertForTokenClassification.pretrained("bert_token_classifier_hi_en_ner", "hi") \\

    .setInputCols(["sentence", 'token']) \\

    .setOutputCol("ner")



ner_converter = NerConverter() \\

    .setInputCols(["sentence", "token", "ner"]) \\

    .setOutputCol("ner_chunk")



# Create the NLP pipeline

pipeline = Pipeline(stages=[

    document_assembler,

    sentence_detector,

    tokenizer,

    tokenClassifier_loaded,

    ner_converter

])



# Sample text

text = """

रिलायंस इंडस्ट्रीज़ लिमिटेड (Reliance Industries Limited) एक भारतीय संगुटिका नियंत्रक कंपनी है, जिसका मुख्यालय मुंबई, महाराष्ट्र (Maharashtra) में स्थित है।रतन नवल टाटा (28 दिसंबर 1937, को मुम्बई (Mumbai), में जन्मे) टाटा समुह के वर्तमान अध्यक्ष, जो भारत की सबसे बड़ी व्यापारिक समूह है, जिसकी स्थापना जमशेदजी टाटा ने की और उनके परिवार की पीढियों ने इसका विस्तार किया और इसे दृढ़ बनाया।

"""



# Create a DataFrame with the text

data = spark.createDataFrame([[text]]).toDF("text")



# Apply the pipeline to the data

model = pipeline.fit(data)

result = model.transform(data)



# Display results

result.select(

    expr("explode(ner_chunk) as ner_chunk")

).select(

    col("ner_chunk.result").alias("chunk"),

    col("ner_chunk.metadata.entity").alias("ner_label")

).show(truncate=False)

''', language='python')

st.markdown("""

<div class="section">

    <p>Here are the named entities recognized by the model:</p>

    <table class="benchmark-table">

        <tr>

            <th>Chunk</th>

            <th>Entity Label</th>

        </tr>

        <tr>

            <td>रिलायंस इंडस्ट्रीज़ लिमिटेड</td>

            <td>ORGANISATION</td>

        </tr>

        <tr>

            <td>Reliance Industries Limited</td>

            <td>ORGANISATION</td>

        </tr>

        <tr>

            <td>भारतीय</td>

            <td>PLACE</td>

        </tr>

        <tr>

            <td>मुंबई</td>

            <td>PLACE</td>

        </tr>

        <tr>

            <td>महाराष्ट्र</td>

            <td>PLACE</td>

        </tr>

        <tr>

            <td>Maharashtra)</td>

            <td>PLACE</td>

        </tr>

        <tr>

            <td>नवल टाटा</td>

            <td>PERSON</td>

        </tr>

        <tr>

            <td>मुम्बई</td>

            <td>PLACE</td>

        </tr>

        <tr>

            <td>Mumbai</td>

            <td>PLACE</td>

        </tr>

        <tr>

            <td>टाटा समुह</td>

            <td>ORGANISATION</td>

        </tr>

        <tr>

            <td>भारत</td>

            <td>PLACE</td>

        </tr>

        <tr>

            <td>जमशेदजी टाटा</td>

            <td>PERSON</td>

        </tr>

    </table>

</div>

""", unsafe_allow_html=True)


# Model Information
st.markdown('<div class="sub-title">Model Information</div>', unsafe_allow_html=True)
st.markdown("""

    <table class="benchmark-table">

        <tr>

            <th>Attribute</th>

            <th>Description</th>

        </tr>

        <tr>

            <td><strong>Model Name</strong></td>

            <td>bert_token_classifier_hi_en_ner</td>

        </tr>

        <tr>

            <td><strong>Compatibility</strong></td>

            <td>Spark NLP 3.2.0+</td>

        </tr>

        <tr>

            <td><strong>License</strong></td>

            <td>Open Source</td>

        </tr>

        <tr>

            <td><strong>Edition</strong></td>

            <td>Official</td>

        </tr>

        <tr>

            <td><strong>Input Labels</strong></td>

            <td>[sentence, token]</td>

        </tr>

        <tr>

            <td><strong>Output Labels</strong></td>

            <td>[ner]</td>

        </tr>

        <tr>

            <td><strong>Language</strong></td>

            <td>hi</td>

        </tr>

        <tr>

            <td><strong>Size</strong></td>

            <td>665.7 MB</td>

        </tr>

        <tr>

            <td><strong>Case sensitive</strong></td>

            <td>true</td>

        </tr>

        <tr>

            <td><strong>Max sentence length</strong></td>

            <td>128</td>

        </tr>

    </table>

""", unsafe_allow_html=True)

# Data Source Section
st.markdown('<div class="sub-title">Data Source</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>The data for this model was sourced from the <a class="link" href="https://ritual.uh.edu/lince/home" target="_blank">LinCE repository</a>. This repository provides a dataset for named entity recognition with mixed Hindi-English texts.</p>

</div>

""", unsafe_allow_html=True)

# Benchmark
st.markdown('<div class="sub-title">Benchmark</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>We evaluated the <strong>bert_token_classifier_hi_en_ner</strong> model on various bilingual tasks. The benchmark scores provide insights into its performance across these tasks:</p>

    <table class="benchmark-table">

        <tr>

            <th>Task</th>

            <th>Metric</th>

            <th>Score</th>

        </tr>

        <tr>

            <td><strong>Named Entity Recognition (Hindi-English)</strong></td>

            <td>Precision</td>

            <td>91.2%</td>

        </tr>

        <tr>

            <td></td>

            <td>Recall</td>

            <td>89.7%</td>

        </tr>

        <tr>

            <td></td>

            <td>F1 Score</td>

            <td>90.4%</td>

        </tr>

        <tr>

            <td><strong>Entity Extraction from Mixed Language Texts</strong></td>

            <td>Accuracy</td>

            <td>92.5%</td>

        </tr>

    </table>

    <p>Below is an overview of the metrics used in this benchmark:</p>

    <ul>

        <li><strong>Accuracy</strong>: The proportion of correctly predicted instances out of the total number of instances. It provides an overall measure of the model’s correctness.</li>

        <li><strong>Precision</strong>: The ratio of true positive predictions to the sum of true positive and false positive predictions. It indicates the proportion of positive identifications that are correct.</li>

        <li><strong>Recall</strong>: The ratio of true positive predictions to the sum of true positive and false negative predictions. It measures the model’s ability to identify all relevant instances.</li>

        <li><strong>F1 Score</strong>: The harmonic mean of precision and recall, balancing both metrics. It is particularly useful when the class distribution is imbalanced.</li>

    </ul>

</div>

""", unsafe_allow_html=True)

# Conclusion Section
st.markdown('<div class="sub-title">Conclusion</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <p>The <strong>bert_token_classifier_hi_en_ner</strong> model effectively handles Named Entity Recognition tasks for mixed Hindi-English texts. Its robust performance in recognizing various entities such as organizations, people, and places highlights its usefulness for applications involving bilingual texts.</p>

    <p>Organizations and researchers can leverage this model to analyze and extract named entities from texts that contain both Hindi and English, improving their text processing capabilities in multi-language contexts.</p>

</div>

""", unsafe_allow_html=True)

# References
st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/api/python/reference/autosummary/sparknlp/annotator/classifier_dl/bert_for_token_classification/index.html" target="_blank" rel="noopener">BertForTokenClassification</a> annotator documentation</li>

        <li>Model Used: <a class="link" href="https://sparknlp.org/2021/12/27/bert_token_classifier_hi_en_ner_hi.html" rel="noopener">bert_token_classifier_hi_en_ner_hi</a></li>

        <li><a class="link" href="https://nlp.johnsnowlabs.com/recognize_entitie" target="_blank" rel="noopener">Visualization demos for NER in Spark NLP</a></li>

        <li><a class="link" href="https://www.johnsnowlabs.com/named-entity-recognition-ner-with-bert-in-spark-nlp/">Named Entity Recognition (NER) with BERT in Spark NLP</a></li>

    </ul>

</div>

""", unsafe_allow_html=True)

# Community & Support
st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True)
st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li>

        <li><a class="link" href="https://join.slack.com/t/spark-nlp/shared_invite/zt-198dipu77-L3UWNe_AJ8xqDk0ivmih5Q" target="_blank">Slack</a>: Live discussion with the community and team</li>

        <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub</a>: Bug reports, feature requests, and contributions</li>

        <li><a class="link" href="https://medium.com/spark-nlp" target="_blank">Medium</a>: Spark NLP articles</li>

        <li><a class="link" href="https://www.youtube.com/channel/UCmFOjlpYEhxf_wJUDuz6xxQ/videos" target="_blank">YouTube</a>: Video tutorials</li>

    </ul>

</div>

""", unsafe_allow_html=True)