import streamlit as st
import pandas as pd
# Custom CSS for better styling
st.markdown("""
""", unsafe_allow_html=True)
# Main Title
st.markdown('Detect Entities in Urdu (urduvec_140M_300d embeddings)
', unsafe_allow_html=True)
# Introduction
st.markdown("""
Named Entity Recognition (NER) models identify and categorize important entities in a text. This page details a word embeddings-based NER model for Urdu texts, using the urduvec_140M_300d word embeddings. The model is pretrained and available for use with Spark NLP.
""", unsafe_allow_html=True)
# Model Description
st.markdown('Description
', unsafe_allow_html=True)
st.markdown("""
This model uses Urdu word embeddings to find 7 different types of entities in Urdu text. It is trained using urduvec_140M_300d word embeddings, so please use the same embeddings in the pipeline. It can identify the following types of entities:
- PER (Persons)
- LOC (Locations)
- ORG (Organizations)
- DATE (Dates)
- TIME (Times)
- DESIGNATION (Designations)
- NUMBER (Numbers)
""", unsafe_allow_html=True)
# Setup Instructions
st.markdown('Setup
', unsafe_allow_html=True)
st.markdown('To use the model, you need Spark NLP installed. You can install it using pip:
', unsafe_allow_html=True)
st.code("""
pip install spark-nlp
pip install pyspark
""", language="bash")
st.markdown("Then, import Spark NLP and start a Spark session:
", unsafe_allow_html=True)
st.code("""
import sparknlp
# Start Spark Session
spark = sparknlp.start()
""", language='python')
# Example Usage
st.markdown('Example Usage with Urdu NER Model
', unsafe_allow_html=True)
st.markdown("""
Below is an example of how to set up and use the uner_mk_140M_300d model for named entity recognition in Urdu:
""", unsafe_allow_html=True)
st.code('''
from sparknlp.base import *
from sparknlp.annotator import *
from pyspark.ml import Pipeline
# Define the components of the pipeline
documentAssembler = DocumentAssembler() \\
.setInputCol("text") \\
.setOutputCol("document")
sentence_detector = SentenceDetector() \\
.setInputCols(["document"]) \\
.setOutputCol("sentence")
tokenizer = Tokenizer() \\
.setInputCols(["sentence"]) \\
.setOutputCol("token")
word_embeddings = WordEmbeddingsModel.pretrained("urduvec_140M_300d", "ur") \\
.setInputCols(["sentence", "token"]) \\
.setOutputCol("embeddings")
ner = NerDLModel.pretrained("uner_mk_140M_300d", "ur") \\
.setInputCols(["sentence", "token", "embeddings"]) \\
.setOutputCol("ner")
ner_converter = NerConverter().setInputCols(["sentence", "token", "ner"]).setOutputCol("ner_chunk")
# Create the pipeline
pipeline = Pipeline(stages=[documentAssembler, sentence_detector, tokenizer, word_embeddings, ner, ner_converter])
# Create sample data
example = """
بریگیڈیئر ایڈ بٹلر سنہ دوہزارچھ میں ہلمند کے فوجی کمانڈر تھے۔
"""
data = spark.createDataFrame([[example]]).toDF("text")
# Fit and transform data with the pipeline
result = pipeline.fit(data).transform(data)
# Select the result, entity
result.select(
expr("explode(ner_chunk) as ner_chunk")
).select(
col("ner_chunk.result").alias("chunk"),
col("ner_chunk.metadata").getItem("entity").alias("ner_label")
).show(truncate=False)
''', language="python")
import pandas as pd
# Create the data for the DataFrame
data = {
"chunk": [
"بریگیڈیئر",
"ایڈ بٹلر",
"سنہ دوہزارچھ",
"ہلمند"
],
"ner_label": [
"DESIGNATION",
"PERSON",
"DATE",
"LOCATION"
]
}
# Creating the DataFrame
df = pd.DataFrame(data)
df.index += 1
st.dataframe(df)
# Model Information
st.markdown('Model Information
', unsafe_allow_html=True)
st.markdown("""
The uner_mk_140M_300d model details are as follows:
- Model Name: uner_mk_140M_300d
- Type: ner
- Compatibility: Spark NLP 4.0.2+
- License: Open Source
- Edition: Official
- Input Labels: [document, token, word_embeddings]
- Output Labels: [ner]
- Language: ur
- Size: 14.8 MB
""", unsafe_allow_html=True)
# Benchmark Section
st.markdown('Benchmark
', unsafe_allow_html=True)
st.markdown("""
Evaluating the performance of NER models is crucial to understanding their effectiveness in real-world applications. Below are the benchmark results for the uner_mk_140M_300d model, focusing on various named entity categories. The metrics used include precision, recall, and F1-score, which are standard for evaluating classification models.
""", unsafe_allow_html=True)
st.markdown("""
---
| Label | TP | FP | FN | Precision | Recall | F1-Score |
|------------------|-------|-----|-----|-----------|---------|----------|
| I-TIME | 12 | 10 | 1 | 0.545455 | 0.923077| 0.685714 |
| B-PERSON | 2808 | 846 | 535 | 0.768473 | 0.839964| 0.802630 |
| B-DATE | 34 | 6 | 6 | 0.850000 | 0.850000| 0.850000 |
| I-DATE | 45 | 1 | 2 | 0.978261 | 0.957447| 0.967742 |
| B-DESIGNATION | 49 | 30 | 16 | 0.620253 | 0.753846| 0.680556 |
| I-LOCATION | 2110 | 750 | 701 | 0.737762 | 0.750623| 0.744137 |
| B-TIME | 11 | 9 | 3 | 0.550000 | 0.785714| 0.647059 |
| I-ORGANIZATION | 2006 | 772 | 760 | 0.722102 | 0.725235| 0.723665 |
| I-NUMBER | 18 | 6 | 2 | 0.750000 | 0.900000| 0.818182 |
| B-LOCATION | 5428 | 1255| 582 | 0.812210 | 0.903161| 0.855275 |
| B-NUMBER | 194 | 36 | 27 | 0.843478 | 0.877828| 0.860298 |
| B-ORGANIZATION | 4364 | 1092| 990 | 0.799926 | 0.815058| 0.807421 |
| I-DESIGNATION | 57 | 15 | 10 | 0.791667 | 0.850746| 0.820896 |
| B-MISC | 18 | 19 | 13 | 0.486486 | 0.580645| 0.529412 |
| I-MISC | 10 | 11 | 10 | 0.476190 | 0.500000| 0.487805 |
| I-PERSON | 1891 | 689 | 622 | 0.732723 | 0.752499| 0.742486 |
---
""", unsafe_allow_html=True)
st.markdown("""
These results demonstrate the model's ability to accurately identify and classify named entities in Urdu text. Precision measures the accuracy of the positive predictions, recall measures the model's ability to find all relevant instances, and F1-score provides a balance between precision and recall.
""", unsafe_allow_html=True)
# Try the Model
st.markdown('Try the Model
', unsafe_allow_html=True)
st.markdown("""
You can use the LightPipeline to quickly test the model on small texts. Here is an example:
""", unsafe_allow_html=True)
st.code('''
from sparknlp.base import LightPipeline
# Create a LightPipeline
light_pipeline = LightPipeline(pipeline.fit(data))
# Annotate a simple text
example_text = "بریگیڈیئر ایڈ بٹلر سنہ دوہزارچھ میں ہلمند کے فوجی کمانڈر تھے۔"
annotations = light_pipeline.fullAnnotate(example_text)
# Display the annotations
for annotation in annotations[0]['ner_chunk']:
print(annotation.result, "->", annotation.metadata['entity'])
''', language="python")
# Conclusion/Summary
st.markdown('Conclusion
', unsafe_allow_html=True)
st.markdown("""
The uner_mk_140M_300d model demonstrates effective named entity recognition in Urdu texts, with strong performance metrics across various entity types. This model leverages urduvec_140M_300d embeddings to enhance its understanding and accuracy in identifying entities like persons, locations, organizations, and more. Its integration into Spark NLP allows for efficient and scalable processing of Urdu text data, making it a valuable tool for researchers and developers working with Urdu language applications.
""", unsafe_allow_html=True)
# References
st.markdown('References
', unsafe_allow_html=True)
st.markdown("""
""", unsafe_allow_html=True)
# Community & Support
st.markdown('Community & Support
', unsafe_allow_html=True)
st.markdown("""
""", unsafe_allow_html=True)