Merge branch 'main' of https://huggingface.co/spaces/pico-lm/blimp

README.md

@@ -1,32 +1,29 @@
 ---
-title: Perplexity
-emoji: 🤗
+title: BLiMP
+emoji: 🎈
 colorFrom: blue
 colorTo: red
-sdk: gradio
-sdk_version: 3.19.1
+sdk: static
+sdk_version: 5.20.1
 app_file: app.py
 pinned: false
 tags:
 - evaluate
 - metric
 description: >-
-  Perplexity (PPL) is one of the most common metrics for evaluating language models.
-  It is defined as the exponentiated average negative log-likelihood of a sequence, calculated with exponent base `e`.
-  
-  For more information on perplexity, see [this tutorial](https://huggingface.co/docs/transformers/perplexity).
+  BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets,
+  each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics.
+  The data is automatically generated according to expert-crafted grammars.
+
+  For more information on perplexity, see the [dataset card](https://huggingface.co/datasets/nyu-mll/blimp).
 ---
 
-# Metric Card for Perplexity
+# Metric Card for BLiMP
 
 ## Metric Description
-Given a model and an input text sequence, perplexity measures how likely the model is to generate the input text sequence.
-
-As a metric, it can be used to evaluate how well the model has learned the distribution of the text it was trained on.
-
-In this case, `model_id` should be the trained model to be evaluated, and the input texts should be the text that the model was trained on.
-
-This implementation of perplexity is calculated with log base `e`, as in `perplexity = e**(sum(losses) / num_tokenized_tokens)`, following recent convention in deep learning frameworks.
+BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets,
+each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics.
+The data is automatically generated according to expert-crafted grammars.
 
 ## Intended Uses
 Any language generation task.
@@ -37,20 +34,17 @@ The metric takes a list of text as input, as well as the name of the model used
 
 ```python
 from evaluate import load
-perplexity = load("perplexity", module_type="metric")
-results = perplexity.compute(predictions=predictions, model_id='gpt2')
+blimp = load("pico-lm/blimp", module_type="metric")
+results = blimp.compute(model_id='pico-lm/pico-decoder')
 ```
 
 ### Inputs
-- **model_id** (str): model used for calculating Perplexity. NOTE: Perplexity can only be calculated for causal language models.
-    - This includes models such as gpt2, causal variations of bert, causal versions of t5, and more (the full list can be found in the AutoModelForCausalLM documentation here: https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForCausalLM )
-- **predictions** (list of str): input text, where each separate text snippet is one list entry.
+- **model_id** (str): model used for calculating BLiMP.
 - **batch_size** (int): the batch size to run texts through the model. Defaults to 16.
-- **add_start_token** (bool): whether to add the start token to the texts, so the perplexity can include the probability of the first word. Defaults to True.
 - **device** (str): device to run on, defaults to `cuda` when available
 
 ### Output Values
-This metric outputs a dictionary with the perplexity scores for the text input in the list, and the average perplexity.
+This metric outputs a dictionary with the BLiMP scores for each subdataset.
 If one of the input texts is longer than the max input length of the model, then it is truncated to the max length for the perplexity computation.
 
 ```
@@ -59,9 +53,6 @@ If one of the input texts is longer than the max input length of the model, then
 
 The range of this metric is [0, inf). A lower score is better.
 
-#### Values from Popular Papers
-
-
 ### Examples
 Calculating perplexity on predictions defined here:
 ```python
@@ -94,25 +85,21 @@ print(round(results["perplexities"][0], 2))
 >>>889.28
 ```
 
-## Limitations and Bias
-Note that the output value is based heavily on what text the model was trained on. This means that perplexity scores are not comparable between models or datasets.
-
-See Meister and Cotterell, ["Language Model Evaluation Beyond Perplexity"]( https://arxiv.org/abs/2106.00085) (2021) for more information about alternative model evaluation strategies. 
-
 ## Citation
 
 ```bibtex
-@article{jelinek1977perplexity,
-title={Perplexity—a measure of the difficulty of speech recognition tasks},
-author={Jelinek, Fred and Mercer, Robert L and Bahl, Lalit R and Baker, James K},
-journal={The Journal of the Acoustical Society of America},
-volume={62},
-number={S1},
-pages={S63--S63},
-year={1977},
-publisher={Acoustical Society of America}
+@article{warstadt2020blimp,
+    author = {Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and Peng, Wei and Wang, Sheng-Fu and Bowman, Samuel R.},
+    title = {BLiMP: The Benchmark of Linguistic Minimal Pairs for English},
+    journal = {Transactions of the Association for Computational Linguistics},
+    volume = {8},
+    number = {},
+    pages = {377-392},
+    year = {2020},
+    doi = {10.1162/tacl\_a\_00321},
+    URL = {https://doi.org/10.1162/tacl_a_00321},
+    eprint = {https://doi.org/10.1162/tacl_a_00321},
+    abstract = { We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4\%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands. }
 }
-```
 
-## Further References
-- [Hugging Face Perplexity Blog Post](https://huggingface.co/docs/transformers/perplexity)
+```