make static

README.md

@@ -3,9 +3,7 @@ title: BLiMP
 emoji: 🎈
 colorFrom: blue
 colorTo: red
-sdk: gradio
-sdk_version: 5.20.1
-app_file: app.py
+sdk: static
 pinned: false
 tags:
 - evaluate
@@ -24,11 +22,13 @@ description: >-
 # Metric Card for BLiMP
 
 ## Metric Description
+
 BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets,
 each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics.
 The data is automatically generated according to expert-crafted grammars.
 
 ## Intended Uses
+
 Any language generation task.
 
 ## How to Use
@@ -42,11 +42,13 @@ results = blimp.compute(model_id='pico-lm/pico-decoder')
 ```
 
 ### Inputs
+
 - **model_id** (str): model used for calculating BLiMP.
 - **batch_size** (int): the batch size to run texts through the model. Defaults to 16.
 - **device** (str): device to run on, defaults to `cuda` when available
 
 ### Output Values
+
 This metric outputs a dictionary with the BLiMP scores for each subdataset.
 If one of the input texts is longer than the max input length of the model, then it is truncated to the max length for the perplexity computation.
 
@@ -57,7 +59,9 @@ If one of the input texts is longer than the max input length of the model, then
 The range of this metric is [0, inf). A lower score is better.
 
 ### Examples
+
 Calculating perplexity on predictions defined here:
+
 ```python
 perplexity = evaluate.load("perplexity", module_type="metric")
 input_texts = ["lorem ipsum", "Happy Birthday!", "Bienvenue"]
@@ -71,7 +75,9 @@ print(round(results["mean_perplexity"], 2))
 print(round(results["perplexities"][0], 2))
 >>>32.25
 ```
+
 Calculating perplexity on predictions loaded in from a dataset:
+
 ```python
 perplexity = evaluate.load("perplexity", module_type="metric")
 input_texts = datasets.load_dataset("wikitext",
@@ -105,4 +111,4 @@ print(round(results["perplexities"][0], 2))
     abstract = { We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4\%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands. }
 }
 
-```
+```

app.py

@@ -1,5 +0,0 @@
-import evaluate
-from evaluate.utils import launch_gradio_widget
-
-module = evaluate.load("./blimp.py", module_type="metric")
-launch_gradio_widget(module)