import streamlit as st
# Custom CSS for better styling
st.markdown("""
""", unsafe_allow_html=True)
# Main Title
st.markdown('Named Entity Recognition for Chinese (BERT-MSRA Dataset)
', unsafe_allow_html=True)
# Description
st.markdown('Description
', unsafe_allow_html=True)
st.markdown("""
This model annotates named entities in a text, which can be used to find features such as names of people, places, and organizations. The model does not read words directly but instead reads word embeddings, which represent words as points such that more semantically similar words are closer together.
This model uses the pre-trained bert_base_chinese embeddings model from BertEmbeddings annotator as an input, so be sure to use the same embeddings in the pipeline.
""", unsafe_allow_html=True)
# Predicted Entities
st.markdown('Predicted Entities
', unsafe_allow_html=True)
st.markdown("""
- Persons-PER
- Locations-LOC
- Organizations-ORG
""", unsafe_allow_html=True)
# How to use
st.markdown('How to use
', unsafe_allow_html=True)
st.markdown("""
To use this model, follow these steps in Python:
""", unsafe_allow_html=True)
st.code("""
from sparknlp.base import *
from sparknlp.annotator import *
from pyspark.ml import Pipeline
# Define the components of the pipeline
document_assembler = DocumentAssembler() \\
.setInputCol("text") \\
.setOutputCol("document")
sentence_detector = SentenceDetector() \\
.setInputCols(["document"]) \\
.setOutputCol("sentence")
word_segmenter = WordSegmenterModel.pretrained("wordseg_large", "zh") \\
.setInputCols(["sentence"]) \\
.setOutputCol("token")
embeddings = BertEmbeddings.pretrained(name='bert_base_chinese', lang='zh') \\
.setInputCols(["document", "token"]) \\
.setOutputCol("embeddings")
ner = NerDLModel.pretrained("ner_msra_bert_768d", "zh") \\
.setInputCols(["document", "token", "embeddings"]) \\
.setOutputCol("ner")
ner_converter = NerConverter() \\
.setInputCols(["sentence", "token", "ner"]) \\
.setOutputCol("entities")
# Create the pipeline
pipeline = Pipeline(stages=[document_assembler, sentence_detector, word_segmenter, embeddings, ner, ner_converter])
# Create sample data
example = spark.createDataFrame([['马云在浙江省杭州市出生，是阿里巴巴集团的主要创始人。']], ["text"])
# Fit and transform data with the pipeline
result = pipeline.fit(example).transform(example)
# Select the result, entity
result.select(
expr("explode(entities) as entity")
).select(
col("entity.result").alias("chunk"),
col("entity.metadata").getItem("entity").alias("ner_label")
).show(truncate=False)
""", language="python")
# Results
import pandas as pd
# Create the data for the DataFrame
data = {
"token": ["马云", "浙江省", "杭州市", "出生", "阿里巴巴集团", "创始人"],
"ner": ["PER", "LOC", "LOC", "ORG", "ORG", "PER"]
}
# Creating the DataFrame
df = pd.DataFrame(data)
df.index += 1
st.dataframe(df)
# Model Information
st.markdown('Model Information
', unsafe_allow_html=True)
st.markdown("""
The ner_msra_bert_768d model details are as follows:
- Model Name: ner_msra_bert_768d
- Type: ner
- Compatibility: Spark NLP 2.7.0+
- License: Open Source
- Edition: Official
- Input Labels: [sentence, token, embeddings]
- Output Labels: [ner]
- Language: zh
""", unsafe_allow_html=True)
# Data Source
st.markdown('Data Source
', unsafe_allow_html=True)
st.markdown("""
The model was trained on the MSRA (Levow, 2006) data set created by “Microsoft Research Asia”.
""", unsafe_allow_html=True)
# Benchmarking
st.markdown('Benchmarking
', unsafe_allow_html=True)
st.markdown("""
Evaluating the performance of NER models is crucial to understanding their effectiveness in real-world applications. Below are the benchmark results for the ner_msra_bert_768d model, focusing on various named entity categories. The metrics used include precision, recall, and F1-score, which are standard for evaluating classification models.
""", unsafe_allow_html=True)
st.markdown("""
---
| ner_tag | precision | recall | f1-score | support |
|--------------|-----------|--------|----------|---------|
| LOC | 0.97 | 0.97 | 0.97 | 2777 |
| O | 1.00 | 1.00 | 1.00 | 146826 |
| ORG | 0.88 | 0.99 | 0.93 | 1292 |
| PER | 0.97 | 0.97 | 0.97 | 1430 |
| accuracy | 1.00 | 152325 | | |
| macro avg | 0.95 | 0.98 | 0.97 | 152325 |
| weighted avg | 1.00 | 1.00 | 1.00 | 152325 |
---
""", unsafe_allow_html=True)
st.markdown("""
These results demonstrate the model's ability to accurately identify and classify named entities in Chinese text. Precision measures the accuracy of the positive predictions, recall measures the model's ability to find all relevant instances, and F1-score provides a balance between precision and recall.
""", unsafe_allow_html=True)
# Conclusion/Summary
st.markdown('Conclusion
', unsafe_allow_html=True)
st.markdown("""
The ner_msra_bert_768d model demonstrates effective named entity recognition in Chinese texts, with high performance metrics across different entity types. This model leverages bert_base_chinese embeddings to enhance its understanding and accuracy in identifying entities like persons, locations, and organizations. Its integration into Spark NLP allows for efficient and scalable processing of Chinese text data, making it a valuable tool for researchers and developers working with Chinese language applications.
""", unsafe_allow_html=True)
# References
st.markdown('References
', unsafe_allow_html=True)
st.markdown("""
""", unsafe_allow_html=True)
# Community & Support
st.markdown('Community & Support
', unsafe_allow_html=True)
st.markdown("""
""", unsafe_allow_html=True)