Update README.md
Browse files
README.md
CHANGED
@@ -1,5 +1,5 @@
|
|
1 |
---
|
2 |
-
title:
|
3 |
emoji: 🎈
|
4 |
colorFrom: blue
|
5 |
colorTo: red
|
@@ -18,7 +18,7 @@ description: >-
|
|
18 |
For more information on perplexity, see the [dataset card](https://huggingface.co/datasets/nyu-mll/blimp).
|
19 |
---
|
20 |
|
21 |
-
# Metric Card for
|
22 |
|
23 |
## Metric Description
|
24 |
BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets,
|
@@ -39,12 +39,12 @@ results = blimp.compute(model_id='pico-lm/pico-decoder')
|
|
39 |
```
|
40 |
|
41 |
### Inputs
|
42 |
-
- **model_id** (str): model used for calculating
|
43 |
- **batch_size** (int): the batch size to run texts through the model. Defaults to 16.
|
44 |
- **device** (str): device to run on, defaults to `cuda` when available
|
45 |
|
46 |
### Output Values
|
47 |
-
This metric outputs a dictionary with the
|
48 |
If one of the input texts is longer than the max input length of the model, then it is truncated to the max length for the perplexity computation.
|
49 |
|
50 |
```
|
|
|
1 |
---
|
2 |
+
title: BLiMP
|
3 |
emoji: 🎈
|
4 |
colorFrom: blue
|
5 |
colorTo: red
|
|
|
18 |
For more information on perplexity, see the [dataset card](https://huggingface.co/datasets/nyu-mll/blimp).
|
19 |
---
|
20 |
|
21 |
+
# Metric Card for BLiMP
|
22 |
|
23 |
## Metric Description
|
24 |
BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets,
|
|
|
39 |
```
|
40 |
|
41 |
### Inputs
|
42 |
+
- **model_id** (str): model used for calculating BLiMP.
|
43 |
- **batch_size** (int): the batch size to run texts through the model. Defaults to 16.
|
44 |
- **device** (str): device to run on, defaults to `cuda` when available
|
45 |
|
46 |
### Output Values
|
47 |
+
This metric outputs a dictionary with the BLiMP scores for each subdataset.
|
48 |
If one of the input texts is longer than the max input length of the model, then it is truncated to the max length for the perplexity computation.
|
49 |
|
50 |
```
|