Update Space (evaluate main: 19f1f9a1)

README.md

@@ -1,6 +1,6 @@
 ---
 title: MAUVE
-emoji: 🤗 
+emoji: 🤗
 colorFrom: blue
 colorTo: red
 sdk: gradio
@@ -11,28 +11,23 @@ tags:
 - evaluate
 - metric
 description: >-
-  MAUVE is a library built on PyTorch and HuggingFace Transformers to measure the gap between neural text and human text with the eponymous MAUVE measure.
-  
-  MAUVE summarizes both Type I and Type II errors measured softly using Kullback–Leibler (KL) divergences.
-  
-  For details, see the MAUVE paper: https://arxiv.org/abs/2102.01454 (Neurips, 2021).
-  
-  This metrics is a wrapper around the official implementation of MAUVE:
-  https://github.com/krishnap25/mauve
+  MAUVE is a measure of the statistical gap between two text distributions, e.g., how far the text written by a model is the distribution of human text, using samples from both distributions.
+ 
+  MAUVE is obtained by computing Kullback–Leibler (KL) divergences between the two distributions in a quantized embedding space of a large language model. It can quantify differences in the quality of generated text based on the size of the model, the decoding algorithm, and the length of the generated text. MAUVE was found to correlate the strongest with human evaluations over baseline metrics for open-ended text generation.
+ 
 ---
 
 # Metric Card for MAUVE
 
 ## Metric description
 
-MAUVE is a library built on PyTorch and HuggingFace Transformers to measure the gap between neural text and human text with the eponymous MAUVE measure. It summarizes both Type I and Type II errors measured softly using [Kullback–Leibler (KL) divergences](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence).
+MAUVE is a measure of the gap between neural text and human text. It is computed using the [Kullback–Leibler (KL) divergences](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) between the two distributions of text in a quantized embedding space of a large language model. MAUVE can identify differences in quality arising from model sizes and decoding algorithms.
 
 This metric is a wrapper around the [official implementation](https://github.com/krishnap25/mauve) of MAUVE.
 
 For more details, consult the [MAUVE paper](https://arxiv.org/abs/2102.01454).
 
-
-## How to use 
+## How to use
 
 The metric takes two lists of strings of tokens separated by spaces: one representing `predictions` (i.e. the text generated by the model) and the second representing `references` (a reference text for each prediction):
 
@@ -41,16 +36,16 @@ from evaluate import load
 mauve = load('mauve')
 predictions = ["hello world", "goodnight moon"]
 references = ["hello world",  "goodnight moon"]
-mauve_results = mauve.compute(predictions=predictions, references=references) 
+mauve_results = mauve.compute(predictions=predictions, references=references)
 ```
 
 It also has several optional arguments:
 
 `num_buckets`: the size of the histogram to quantize P and Q. Options: `auto` (default) or an integer.
 
-`pca_max_data`: the number data points to use for PCA dimensionality reduction prior to clustering. If -1, use all the data. The default is `-1`.
+`pca_max_data`: the number of data points to use for PCA dimensionality reduction prior to clustering. If -1, use all the data. The default is `-1`.
 
-`kmeans_explained_var`: amount of variance of the data to keep in dimensionality reduction by PCA. The default is `0.9`.
+`kmeans_explained_var`: the amount of variance of the data to keep in dimensionality reduction by PCA. The default is `0.9`.
 
 `kmeans_num_redo`: number of times to redo k-means clustering (the best objective is kept). The default is `5`.
 
@@ -89,10 +84,10 @@ This metric outputs a dictionary with 5 key-value pairs:
 
 ### Values from popular papers
 
-The [original MAUVE paper](https://arxiv.org/abs/2102.01454) reported values ranging from 0.88 to 0.94 for open-ended text generation using a text completion task in the web text domain. The authors found that bigger models resulted in higher MAUVE scores, and that MAUVE is correlated with human judgments.
+The [original MAUVE paper](https://arxiv.org/abs/2102.01454) reported values ranging from 0.88 to 0.94 for open-ended text generation using a text completion task in the web text domain. The authors found that bigger models resulted in higher MAUVE scores and that MAUVE is correlated with human judgments.
 
 
-## Examples 
+## Examples
 
 Perfect match between prediction and reference:
 
@@ -101,7 +96,7 @@ from evaluate import load
 mauve = load('mauve')
 predictions = ["hello world", "goodnight moon"]
 references = ["hello world",  "goodnight moon"]
-mauve_results = mauve.compute(predictions=predictions, references=references) 
+mauve_results = mauve.compute(predictions=predictions, references=references)
 print(mauve_results.mauve)
 1.0
 ```
@@ -113,7 +108,7 @@ from evaluate import load
 mauve = load('mauve')
 predictions = ["hello world", "goodnight moon"]
 references = ["hello there", "general kenobi"]
-mauve_results = mauve.compute(predictions=predictions, references=references) 
+mauve_results = mauve.compute(predictions=predictions, references=references)
 print(mauve_results.mauve)
 0.27811372536724027
 ```
@@ -122,7 +117,15 @@ print(mauve_results.mauve)
 
 The [original MAUVE paper](https://arxiv.org/abs/2102.01454) did not analyze the inductive biases present in different embedding models, but related work has shown different kinds of biases exist in many popular generative language models including GPT-2 (see [Kirk et al., 2021](https://arxiv.org/pdf/2102.04130.pdf), [Abid et al., 2021](https://arxiv.org/abs/2101.05783)). The extent to which these biases can impact the MAUVE score has not been quantified.
 
-Also, calculating the MAUVE metric involves downloading the model from which features are obtained -- the default model, `gpt2-large`, takes over 3GB of storage space and downloading it can take a significant amount of time depending on the speed of your internet connection. If this is an issue, choose a smaller model; for instance `gpt` is 523MB.
+Also, calculating the MAUVE metric involves downloading the model from which features are obtained -- the default model, `gpt2-large`, takes over 3GB of storage space and downloading it can take a significant amount of time depending on the speed of your internet connection. If this is an issue, choose a smaller model; for instance, `gpt` is 523MB.
+
+It is a good idea to use at least 1000 samples for each distribution to compute MAUVE (the original paper uses 5000).
+
+MAUVE is unable to identify very small differences between different settings of generation (e.g., between top-p sampling with p=0.95 versus 0.96). It is important, therefore, to account for the randomness inside the generation (e.g., due to sampling) and within the MAUVE estimation procedure (see the `seed` parameter above). Concretely, it is a good idea to obtain generations using multiple random seeds and/or to use rerun MAUVE with multiple values of the parameter `seed`.
+
+For MAUVE to be large, the model distribution must be close to the human text distribution as seen by the embeddings. It is possible to have high-quality model text that still has a small MAUVE score (i.e., large gap) if it contains text about different topics/subjects, or uses a different writing style or vocabulary, or contains texts of a different length distribution. MAUVE summarizes the statistical gap (as measured by the large language model embeddings) --- this includes all these factors in addition to the quality-related aspects such as grammaticality.
+
+See the [official implementation](https://github.com/krishnap25/mauve#best-practices-for-mauve) for more details about best practices. 
 
 
 ## Citation
@@ -132,10 +135,10 @@ Also, calculating the MAUVE metric involves downloading the model from which fea
   title={MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers},
   author={Pillutla, Krishna and Swayamdipta, Swabha and Zellers, Rowan and Thickstun, John and Welleck, Sean and Choi, Yejin and Harchaoui, Zaid},
   booktitle = {NeurIPS},
-  year      = {2021}
+  year  	= {2021}
 }
 ```
 
-## Further References 
+## Further References
 - [Official MAUVE implementation](https://github.com/krishnap25/mauve)
 - [Hugging Face Tasks - Text Generation](https://huggingface.co/tasks/text-generation)

requirements.txt

@@ -1,4 +1,4 @@
-git+https://github.com/huggingface/evaluate@28144191c9b78b67c15f50527b55a18d9cb6e1e6
+git+https://github.com/huggingface/evaluate@19f1f9a1e76aa7aa3c8ee50022a33a200a3467b0
 faiss-cpu
 scikit-learn
 mauve-text