import streamlit as st # Custom CSS for better styling st.markdown(""" """, unsafe_allow_html=True) # Introduction st.markdown('
Coreference Resolution with BERT-based Models in Spark NLP
', unsafe_allow_html=True) st.markdown("""

Welcome to the Spark NLP Coreference Resolution Demo App! Coreference resolution is a crucial task in Natural Language Processing (NLP) that involves identifying and linking all expressions within a text that refer to the same real-world entity. This can be useful for a wide range of applications, such as text understanding, information extraction, and question answering.

Using Spark NLP, it is possible to perform coreference resolution with high accuracy using BERT-based models. This app demonstrates how to use the SpanBertCoref annotator to resolve coreferences in text data.

""", unsafe_allow_html=True) st.image('images/Coreference-Resolution.png', use_column_width='auto') # About Coreference Resolution st.markdown('
About Coreference Resolution
', unsafe_allow_html=True) st.markdown("""

Coreference resolution is the task of identifying and linking all expressions within a text that refer to the same real-world entity, such as a person, object, or concept. This technique involves analyzing a text and identifying all expressions that refer to a specific entity, such as “he,” “she,” “it,” or “they.” These expressions are then linked together to form a “coreference chain,” representing all the different ways that entity is referred to in the text.

For example, given the sentence, “John went to the store. He bought some groceries,” a coreference resolution model would identify that “John” and “He” both refer to the same entity and produce a cluster of coreferent mentions.

""", unsafe_allow_html=True) # Using SpanBertCoref in Spark NLP st.markdown('
Using SpanBertCoref in Spark NLP
', unsafe_allow_html=True) st.markdown("""

The SpanBertCoref annotator in Spark NLP allows users to perform coreference resolution with high accuracy using BERT-based models. This annotator can identify and link expressions that refer to the same entity in text data, providing valuable insights from unstructured text data.

The SpanBertCoref annotator in Spark NLP offers:

""", unsafe_allow_html=True) st.markdown('

Example Usage in Python

', unsafe_allow_html=True) st.markdown('

Here’s how you can implement coreference resolution using the SpanBertCoref annotator in Spark NLP:

', unsafe_allow_html=True) # Setup Instructions st.markdown('
Setup
', unsafe_allow_html=True) st.markdown('

To install Spark NLP in Python, use your favorite package manager (conda, pip, etc.). For example:

', unsafe_allow_html=True) st.code(""" pip install spark-nlp pip install pyspark """, language="bash") st.markdown("

Then, import Spark NLP and start a Spark session:

", unsafe_allow_html=True) st.code(""" import sparknlp # Start Spark Session spark = sparknlp.start() """, language='python') # Coreference Resolution Example st.markdown('
Example Usage: Coreference Resolution with SpanBertCoref
', unsafe_allow_html=True) st.code(''' from sparknlp.base import DocumentAssembler, Pipeline from sparknlp.annotator import ( SentenceDetector, Tokenizer, SpanBertCorefModel ) import pyspark.sql.functions as F # Step 1: Transforms raw texts to document annotation document = DocumentAssembler() \\ .setInputCol("text") \\ .setOutputCol("document") # Step 2: Sentence Detection sentenceDetector = SentenceDetector() \\ .setInputCols("document") \\ .setOutputCol("sentences") # Step 3: Tokenization token = Tokenizer() \\ .setInputCols("sentences") \\ .setOutputCol("tokens") \\ .setContextChars(["(", ")", "?", "!", ".", ","]) # Step 4: Coreference Resolution corefResolution = SpanBertCorefModel().pretrained("spanbert_base_coref") \\ .setInputCols(["sentences", "tokens"]) \\ .setOutputCol("corefs") \\ .setCaseSensitive(False) # Define the pipeline pipeline = Pipeline(stages=[document, sentenceDetector, token, corefResolution]) # Create the dataframe data = spark.createDataFrame([["Ana is a Graduate Student at UT Dallas. She loves working in Natural Language Processing at the Institute. Her hobbies include blogging, dancing, and singing."]]).toDF("text") # Fit the dataframe to the pipeline to get the model model = pipeline.fit(data) # Transform the data to get predictions result = model.transform(data) # Display the extracted coreferences result.selectExpr("explode(corefs) AS coref").selectExpr("coref.result as token", "coref.metadata").show(truncate=False) ''', language='python') st.text(""" +-------------+----------------------------------------------------------------------------------------+ |token |metadata | +-------------+----------------------------------------------------------------------------------------+ |ana |{head.sentence -> -1, head -> ROOT, head.begin -> -1, head.end -> -1, sentence -> 0} | |she |{head.sentence -> 0, head -> ana, head.begin -> 0, head.end -> 2, sentence -> 1} | |her |{head.sentence -> 0, head -> ana, head.begin -> 0, head.end -> 2, sentence -> 2} | |ut dallas |{head.sentence -> -1, head -> ROOT, head.begin -> -1, head.end -> -1, sentence -> 0} | |the institute|{head.sentence -> 0, head -> ut dallas, head.begin -> 29, head.end -> 37, sentence -> 1}| +-------------+----------------------------------------------------------------------------------------+ """) st.markdown("""

The code snippet demonstrates how to set up a pipeline in Spark NLP to resolve coreferences in text data using the SpanBertCoref annotator. The resulting DataFrame contains the coreferent mentions and their metadata.

""", unsafe_allow_html=True) # One-liner Alternative st.markdown('
One-liner Alternative
', unsafe_allow_html=True) st.markdown("""

In October 2022, John Snow Labs released the open-source johnsnowlabs library that contains all the company products, open-source and licensed, under one common library. This simplified the workflow, especially for users working with more than one of the libraries (e.g., Spark NLP + Healthcare NLP). This new library is a wrapper on all of John Snow Lab’s libraries and can be installed with pip:

pip install johnsnowlabs

To run coreference resolution with one line of code, we can simply:

""", unsafe_allow_html=True) st.code(""" # Import the NLP module which contains Spark NLP and NLU libraries from johnsnowlabs import nlp sample_text = "Ana is a Graduate Student at UT Dallas. She loves working in Natural Language Processing at the Institute. Her hobbies include blogging, dancing, and singing." # Returns a pandas DataFrame, we select the desired columns nlp.load('en.coreference.spanbert').predict(sample_text, output_level='sentence') """, language='python') st.image('images/johnsnowlabs-output.png', use_column_width='auto') st.markdown("""

This approach demonstrates how to use the johnsnowlabs library to perform coreference resolution with a single line of code. The resulting DataFrame contains the coreferent mentions and their metadata.

""", unsafe_allow_html=True) # Conclusion st.markdown("""

Conclusion

In this app, we demonstrated how to use Spark NLP's SpanBertCoref annotator to resolve coreferences in text data. These powerful tools enable users to efficiently process large datasets and identify coreferent mentions, providing deeper insights for various applications. By integrating these annotators into your NLP pipelines, you can enhance the extraction of valuable entity relationships from unstructured text, improving text understanding, information extraction, and question answering.

""", unsafe_allow_html=True) # References and Additional Information st.markdown('
For additional information, please check the following references.
', unsafe_allow_html=True) st.markdown("""
""", unsafe_allow_html=True) st.markdown('
Community & Support
', unsafe_allow_html=True) st.markdown("""
""", unsafe_allow_html=True)