Add my new, shiny module.

README.md

@@ -1,50 +1,95 @@
 ---
 title: PR AUC
-datasets:
--  
-tags:
-- evaluate
-- metric
-description: "TODO: add a description here"
+emoji: 🤗 
+colorFrom: blue
+colorTo: red
 sdk: gradio
 sdk_version: 3.19.1
 app_file: app.py
 pinned: false
+tags:
+- evaluate
+- metric
+description: "This metric computes the area under the curve (AUC) for the Precision-Recall Curve (PR). summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight."
 ---
 
 # Metric Card for PR AUC
-
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
-
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+This metric computes the area under the curve (AUC) for the Precision-Recall Curve (PR). summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight.
+
+You should use this metric:
+- when your data is heavily imbalanced. As mentioned before, it was discussed extensively in this article by Takaya Saito and Marc Rehmsmeier. The intuition is the following: since PR AUC focuses mainly on the positive class (PPV and TPR) it cares less about the frequent negative class.
+- when you care more about positive than negative class. If you care more about the positive class and hence PPV and TPR you should go with Precision-Recall curve and PR AUC (average precision).
 
 ## How to Use
 *Give general statement of how to use the metric*
-
-*Provide simplest possible example for using the metric*
+This metric requires references and prediction scores:
+```python
+>>> average_precision_score = evaluate.load("pr_auc")
+>>> refs = np.array([0, 0, 1, 1])
+>>> pred_scores = np.array([0.1, 0.4, 0.35, 0.8])
+>>> results = average_precision_score.compute(references=refs, prediction_scores=pred_scores)
+>>> print(round(results['average_precision'], 2))
+0.83
+```
+Default implementation of this metric is binary. If using multiclass, see examples below.
 
 ### Inputs
-*List all input arguments in the format below*
 - **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
-
+Args:
+- **`references`** (array-like of shape (n_samples,) or (n_samples, n_classes)): True binary labels or binary label indicators.
+- prediction_scores (array-like of shape (n_samples,) or (n_samples, n_classes)): Model predictions. Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by decision_function on some classifiers).
+- **`average`** (`str`): Type of average, and is ignored in the binary use case. Defaults to `'macro'`. Options are:
+    - `'micro'`: Calculate metrics globally by considering each element of the label indicator matrix as a label.
+    - `'macro'`: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
+    - `'weighted'`: Calculate metrics for each label, and find their average, weighted by support (i.e. the number of true instances for each label).
+    - `'samples'`: Calculate metrics for each instance, and find their average. Only works with the multilabel use case.
+    - `None`:  No average is calculated, and scores for each class are returned. Only works with the multilabels use case.
+- **`pos_label`** (`int`, `float`, `bool` or `str`): The label of the positive class. Only applied to binary y_true. For multilabel-indicator y_true, pos_label is fixed to 1.
+- **`sample_weight`** (array-like of shape (n_samples,)): Sample weights. Defaults to None.
 ### Output Values
+This metric returns a dict containing the `average_precision` score. The score is a `float`.
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+The output therefore generally takes the following format:
+```python
+{'average_precision': 0.778}
+```
 
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
+PR AUC scores can take on any value between `0` and `1`, inclusive.
 
 #### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
 
 ### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+Example 1, the **binary** use case:
+```python
+>>> average_precision_score = evaluate.load("pr_auc")
+>>> refs = np.array([0, 0, 1, 1])
+>>> pred_scores = np.array([0.1, 0.4, 0.35, 0.8])
+>>> results = average_precision_score.compute(references=refs, prediction_scores=pred_scores)
+>>> print(round(results['average_precision'], 2))
+0.83
+```
+
+Example 2, the **multiclass** use case:
+```python
+>>> average_precision_score = evaluate.load("pr_auc")
+>>> refs = np.array([0, 0, 1, 1, 2, 2])
+>>> pred_scores = np.array([[0.7, 0.2, 0.1],
+...                         [0.4, 0.3, 0.3],
+...                         [0.1, 0.8, 0.1],
+...                         [0.2, 0.3, 0.5],
+...                         [0.4, 0.4, 0.2],
+...                         [0.1, 0.2, 0.7]])
+>>> results = average_precision_score.compute(references=refs, prediction_scores=pred_scores)
+>>> print(round(results['average_precision'], 2))
+0.77
+```
 
 ## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+
 
 ## Citation
-*Cite the source where this metric was introduced.*
+
 
 ## Further References
-*Add any useful further references.*
+This implementation is a wrapper around the [Scikit-learn implementation]("https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html"). Much of the documentation here was adapted from their existing documentation, as well.

pr_auc.py

@@ -1,4 +1,4 @@
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2023 The HuggingFace Datasets Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -11,85 +11,93 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""TODO: Add a description here."""
+"""Accuracy metric."""
 
-import evaluate
 import datasets
+from sklearn.metrics import average_precision_score
 
+import evaluate
 
-# TODO: Add BibTeX citation
-_CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
-}
-"""
 
-# TODO: Add description of the module here
-_DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
-"""
+_DESCRIPTION = """
+This metric computes the area under the curve (AUC) for the Precision-Recall Curve (PR). summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight.
 
+You should use this metric:
+- when your data is heavily imbalanced. As mentioned before, it was discussed extensively in this article by Takaya Saito and Marc Rehmsmeier. The intuition is the following: since PR AUC focuses mainly on the positive class (PPV and TPR) it cares less about the frequent negative class.
+- when you care more about positive than negative class. If you care more about the positive class and hence PPV and TPR you should go with Precision-Recall curve and PR AUC (average precision).
+"""
 
-# TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates how good are predictions given some references, using certain scores
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
+- references (array-like of shape (n_samples,) or (n_samples, n_classes)): True binary labels or binary label indicators.
+- prediction_scores (array-like of shape (n_samples,) or (n_samples, n_classes)): Model predictions. Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by decision_function on some classifiers).
+- average (`str`): Type of average, and is ignored in the binary use case. Defaults to 'macro'. Options are:
+    - `'micro'`: Calculate metrics globally by considering each element of the label indicator matrix as a label.
+    - `'macro'`: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
+    - `'weighted'`: Calculate metrics for each label, and find their average, weighted by support (i.e. the number of true instances for each label).
+    - `'samples'`: Calculate metrics for each instance, and find their average. Only works with the multilabel use case.
+    - `None`:  No average is calculated, and scores for each class are returned. Only works with the multilabels use case.
+- pos_label (int, float, bool or str): The label of the positive class. Only applied to binary y_true. For multilabel-indicator y_true, pos_label is fixed to 1.
+- sample_weight (array-like of shape (n_samples,)): Sample weights. Defaults to None.
 Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
-Examples:
-    Examples should be written in doctest format, and should illustrate how
-    to use the function.
+    average_precision (`float` or array-like of shape (n_classes,)): Returns `float` of average precision score.
+Example:
+    Example 1:
+        >>> average_precision_score = evaluate.load("pr_auc")
+        >>> refs = np.array([0, 0, 1, 1])
+        >>> pred_scores = np.array([0.1, 0.4, 0.35, 0.8])
+        >>> results = average_precision_score.compute(references=refs, prediction_scores=pred_scores)
+        >>> print(round(results['average_precision'], 2))
+        0.83
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'accuracy': 1.0}
+    Example 2:
+        >>> average_precision_score = evaluate.load("pr_auc")
+        >>> refs = np.array([0, 0, 1, 1, 2, 2])
+        >>> pred_scores = np.array([[0.7, 0.2, 0.1],
+        ...                         [0.4, 0.3, 0.3],
+        ...                         [0.1, 0.8, 0.1],
+        ...                         [0.2, 0.3, 0.5],
+        ...                         [0.4, 0.4, 0.2],
+        ...                         [0.1, 0.2, 0.7]])
+        >>> results = average_precision_score.compute(references=refs, prediction_scores=pred_scores)
+        >>> print(round(results['roc_auc'], 2))
+        0.77
 """
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
+_CITATION = """
+"""
 
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class PRAUC(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
-
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
-            # This is the description that will appear on the modules page.
-            module_type="metric",
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
-            # This defines the format of each prediction and reference
-            features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
-            }),
-            # Homepage of the module for documentation
-            homepage="http://module.homepage",
-            # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            features=datasets.Features(
+                {
+                    "prediction_scores": datasets.Sequence(datasets.Value("float")),
+                    "references": datasets.Value("int32"),
+                }
+            ),
+            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html"],
         )
 
-    def _download_and_prepare(self, dl_manager):
-        """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
-        pass
-
-    def _compute(self, predictions, references):
-        """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
+    def _compute(
+        self,
+        references,
+        prediction_scores,
+        average="macro",
+        sample_weight=None,
+        pos_label=1,
+    ):
         return {
-            "accuracy": accuracy,
+            "average_precision": average_precision_score(
+                references,
+                prediction_scores,
+                average=average,
+                sample_weight=sample_weight,
+                pos_labels=pos_label,
+            )
         }

requirements.txt

@@ -1 +1,2 @@
-git+https://github.com/huggingface/evaluate@main
+git+https://github.com/huggingface/evaluate@main
+scikit-learn