initial commit.

README.md

@@ -5,46 +5,89 @@ datasets:
 tags:
 - evaluate
 - metric
-description: "TODO: add a description here"
+- accuracy
+
+description: "computes the accuracy at k for a set of predictions as labels"
 sdk: gradio
 sdk_version: 3.0.2
 app_file: app.py
 pinned: false
 ---
 
-# Metric Card for accuracyk
-
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
+# accuracyk
 
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+Computes the accuracy at k for a set of predictions. The accuracy at k is the number of instances where the real label is in the set of the k most probable
+classes.
+The parameter k is inferred from the shape of the array passed. If you want the accuracy at 5 the shape needs to be (N, 5) where N is the number of examples.
 
 ## How to Use
-*Give general statement of how to use the metric*
-
-*Provide simplest possible example for using the metric*
+```
+predictions = np.array([
+    [0, 7, 1, 3, 5],
+    [0, 2, 9, 8, 4],
+    [8, 4, 0, 1, 3],
+])
+references = np.array([
+    3, 
+    5, 
+    0
+])
+results = accuracyk.compute(predictions=predictions, references=references)
+# 2/3 of the labels are in the corresponding rows
+# the shape of the array predictions is (3, 5) so accuracy at 5 has been computed
+# { accuracy: 0.6 } 
+```
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+- **predictions**: An array of shape (N, K) where N is the number of examples and K is the desired k (5 for accuracy at 5)
+- **references**: An array of the true labels for the examples
 
 ### Output Values
-
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
-
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
-
-#### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
+The metric returns outputs between 0 and 1. With 0 being that no value is in its corresponding row and 1 being that every value occurs in its row (higher is better).
 
 ### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
-
-## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+```python
+>>> accuracyk = evaluate.load("KevinSpaghetti/accuracyk")
+>>> # with numpy arrays
+>>> predictions = np.array([
+>>>     [0, 7, 1, 3, 5],
+>>>     [0, 2, 9, 8, 4],
+>>>     [8, 4, 0, 1, 3],
+>>> ])
+>>> references = np.array([
+>>>     3, 
+>>>     4, 
+>>>     0
+>>> ])
+>>> results = accuracyk.compute(predictions=predictions, references=references)
+{ accuracy: 1 } # every label is in its row 
 
-## Citation
-*Cite the source where this metric was introduced.*
+>>> # With lists
+>>> predictions = [
+>>>     [0, 7, 1, 3, 5],
+>>>     [0, 2, 9, 8, 4],
+>>>     [8, 4, 0, 1, 3],
+>>> ]
+>>> references = [
+>>>     3, 
+>>>     5, 
+>>>     0
+>>> ]
+>>> results = accuracyk.compute(predictions=predictions, references=references)
+{ accuracy: 0.6 } 
+>>> # 3 is in the first row, 
+>>> # 5 is not in the second row,
+>>> # 0 is in the third row
+    
+>>> # with numpy for a batch of examples
+>>> k=5
+>>> # get the 5 highest probabilities
+>>> top5_probs = np.argpartition(logits, -k, axis=-1)[:, -k:]
+>>> results = accuracyk.compute(references=top5_probs, predictions=labels)
 
-## Further References
-*Add any useful further references.*
+>>> # computing the accuracy at 1
+>>> predictions = np.array([ 3, 8, 1 ])
+>>> references = np.array([ 3, 4, 0 ])
+>>> results = accuracyk.compute(predictions=np.expand_dims(predictions, axis=1), references=references)
+```

accuracyk.py

@@ -1,68 +1,81 @@
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""TODO: Add a description here."""
+"""Computes the accuracy at k for a set of labels"""
 
 import evaluate
 import datasets
+import typing
 
+_CITATION = ""
 
-# TODO: Add BibTeX citation
-_CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
-}
-"""
-
-# TODO: Add description of the module here
 _DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
+Computes the accuracy at k for a set of predictions. The accuracy at k is the \
+number of instances where the real label is in the set of the k most probable 
+classes.
+The parameter k is inferred from the shape of the array passed. If you want the accuracy \
+at 5 the shape needs to be (N, 5) where N is the number of examples.
 """
 
-
 # TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates how good are predictions given some references, using certain scores
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
+    predictions: An array of shape (N, K) where N is the number of examples
+                 and K is the desired k (5 for accuracy at 5)
+    references: An array of the true labels for the examples
 Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
+    accuracy: the accuracy at k for the inputs
 Examples:
-    Examples should be written in doctest format, and should illustrate how
-    to use the function.
+    
+    >>> accuracyk = evaluate.load("KevinSpaghetti/accuracyk")
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'accuracy': 1.0}
-"""
+    >>> #with numpy arrays
+    >>> predictions = np.array([
+    >>>     [0, 7, 1, 3, 5],
+    >>>     [0, 2, 9, 8, 4],
+    >>>     [8, 4, 0, 1, 3],
+    >>> ])
+    >>> references = np.array([
+    >>>     3, 
+    >>>     4, 
+    >>>     0
+    >>> ])
+    >>> results = accuracyk.compute(predictions=predictions, references=references)
+    >>> #\{ accuracy: 1 \} # every label is in its row 
+
+    >>> #With lists
+    >>> predictions = [
+    >>>     [0, 7, 1, 3, 5],
+    >>>     [0, 2, 9, 8, 4],
+    >>>     [8, 4, 0, 1, 3],
+    >>> ]
+    >>> references = [
+    >>>     3, 
+    >>>     5, 
+    >>>     0
+    >>> ]
+    >>> results = accuracyk.compute(predictions=predictions, references=references)
+    >>> #\{ accuracy: 0.6 \} 
+    >>> # 3 is in the first row, 
+    >>> # 5 is not in the second row,
+    >>> # 0 is in the third row
+    
+    >>> #with numpy for a batch of examples
+    >>> k=5
+    >>> # get the 5 highest probabilities
+    >>> top5_probs = np.argpartition(logits, -k, axis=-1)[:, -k:]
+    >>> results = accuracyk.compute(references=top5_probs, predictions=labels)
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
+    >>> # computing the accuracy at 1
+    >>> predictions = np.array([ 3, 8, 1 ])
+    >>> references = np.array([ 3, 4, 0 ])
+    >>> results = accuracyk.compute(predictions=np.expand_dims(predictions, axis=1), references=references)
+    >>> print(results)
 
+"""
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class accuracyk(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
+    """Computes the accuracy at k for an array of shape (N, k) and correct labels"""
 
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
             # This is the description that will appear on the modules page.
             module_type="metric",
@@ -71,25 +84,24 @@ class accuracyk(evaluate.Metric):
             inputs_description=_KWARGS_DESCRIPTION,
             # This defines the format of each prediction and reference
             features=datasets.Features({
-                'predictions': datasets.Value('int64'),
+                'predictions': datasets.Sequence(datasets.Value("int64")),
                 'references': datasets.Value('int64'),
             }),
-            # Homepage of the module for documentation
-            homepage="http://module.homepage",
-            # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            codebase_urls=[],
+            reference_urls=[]
         )
 
     def _download_and_prepare(self, dl_manager):
-        """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
-        pass
+        ...
 
     def _compute(self, predictions, references):
-        """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
-        return {
-            "accuracy": accuracy,
-        }
+        """Returns the accuracy at k"""
+        if isinstance(predictions, list):
+            accuracyk = sum(
+                [reference in kpredictions for kpredictions, reference in zip(predictions, references)]
+            ) / len(references)    
+        else:
+            accuracyk = (
+                references[:, None] == predictions[:, :]
+            ).any(axis=1).sum() / len(references)
+        return dict(accuracy=accuracyk)

app.py

@@ -1,6 +1,5 @@
 import evaluate
 from evaluate.utils import launch_gradio_widget
 
-
 module = evaluate.load("KevinSpaghetti/accuracyk")
 launch_gradio_widget(module)

tests.py

@@ -1,17 +1,33 @@
-test_cases = [
-    {
-        "predictions": [0, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0}
-    },
-    {
-        "predictions": [1, 1],
-        "references": [1, 1],
-        "result": {"metric_score": 1}
-    },
-    {
-        "predictions": [1, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0.5}
-    }
-]
+import evaluate
+import numpy as np
+
+accuracyk = evaluate.load("./accuracyk.py", use_auth_token=True)
+
+predictions = np.array([
+    [0, 7, 1, 3, 5],
+    [0, 2, 9, 8, 4],
+    [8, 4, 0, 1, 3],
+])
+
+references = np.array([
+    3, 
+    4, 
+    0
+])
+results = accuracyk.compute(predictions=predictions, references=references)
+print(results)
+
+predictions = [
+    [0, 7, 1, 3, 5],
+    [0, 2, 9, 8, 4],
+    [8, 4, 0, 1, 3],
+]
+references = [3, 5, 0]
+results = accuracyk.compute(predictions=predictions, references=references)
+
+predictions = np.array([ 3, 8, 1 ])
+references = np.array([ 3, 4, 0 ])
+
+results = accuracyk.compute(predictions=np.expand_dims(predictions, axis=1), references=references)
+print(results)
+