Merge pull request #3 from clclab/feature/blocks

README.md

@@ -4,7 +4,7 @@ emoji: 🌍
 colorFrom: yellow
 colorTo: indigo
 sdk: gradio
-sdk_version: 3.0.24
+sdk_version: 3.3
 app_file: app.py
 pinned: false
 ---

app.py

@@ -68,18 +68,22 @@ dataset = CrowSPairsDataset()
 
 bias_type_sel = gradio.Dropdown(label="Bias Type", choices=dataset.bias_types())
 
-iface = gradio.Interface(
-    fn=run,
-    inputs=bias_type_sel,
-    outputs="html",
-    title="Detecting stereotypes in the GPT-2 language model using CrowS-Pairs",
-    description="""GPT-2 is a language model which can score how likely it is that some text is a valid English sentence: not only grammaticality, but also the 'meaning' of the sentence is part of this score. 
-    CrowS-Pairs is a dataset with pairs of more and less stereotypical examples for different social groups (e.g., gender and nationality stereotypes). 
-    We sample 10 random pairs from CrowS-Pairs and show whether the stereotypical example gets a higher score ('is more likely'). 
-    If GPT-2 systematically prefers the stereotypical examples, it has probably learnt these stereotypes from the training data.
-    **DISCLAIMER: How to measure bias in language models is not trivial and an active area of research. 
-    CrowS-Pairs is only one bias benchmark, and here you can probably find some examples that are nonsensical, with typos, or containing stereotypes that are only relevant in the American cultural context.**
-    """,
-)
+with open("description.md") as fh:
+    desc = fh.read()
+
+with open("notice.md") as fh:
+    notice = fh.read()
+
+with gradio.Blocks() as iface:
+    gradio.Markdown(desc)
+    with gradio.Row(equal_height=True):
+        with gradio.Column(scale=4):
+            inp = gradio.Dropdown(label="Bias Type", choices=dataset.bias_types())
+        with gradio.Column(scale=1):
+            but = gradio.Button("Sample")
+    out = gradio.HTML()
+    but.click(run, inp, out)
+    with gradio.Accordion("A note about explainability models"):
+        gradio.Markdown(notice)
 
 iface.launch()

description.md

@@ -0,0 +1,5 @@
+# Detecting stereotypes in the GPT-2 language model using CrowS-Pairs
+
+GPT-2 is a language model which can score how likely it is that some text is a valid English sentence: not only grammaticality, but also the 'meaning' of the sentence is part of this score. CrowS-Pairs is a dataset with pairs of more and less stereotypical examples for different social groups (e.g., gender and nationality stereotypes). We sample 10 random pairs from CrowS-Pairs and show whether the stereotypical example gets a higher score ('is more likely'). If GPT-2 systematically prefers the stereotypical examples, it has probably learnt these stereotypes from the training data.
+
+The colors indicate whether the $${\color{blue}stereotypical}$$ or the $${\color{pink}less stereotypical}$$ example gets the higher score.

notice.md

@@ -0,0 +1,6 @@
+# Measuring bias in language models is hard!
+How to measure bias in language models is not trivial and still an active area of research.
+First of all, what is bias? As you may have noticed, stereotypes may change across languages and cultures.
+What is problematic in the USA, may not be relevant in the Netherlands---each cultural context requires its own careful evaluation.
+Furthermore, defining good ways to measure it is also difficult.
+For example, [Blodgett et al. (2021)](https://aclanthology.org/2021.acl-long.81/) find that typos, nonsensical examples, and other mistakes threatens the validity of CrowS-Pairs.