Filled out the model card

README.md

@@ -1,11 +1,11 @@
 ---
 title: RestrictedPython Code Eval
 datasets:
--  
+-  N/A (eval module only)
 tags:
 - evaluate
 - metric
-description: "TODO: add a description here"
+description: "Same logic as the built-in `code_eval`, but compiling and running the code using `RestrictedPython`"
 sdk: gradio
 sdk_version: 3.19.1
 app_file: app.py
@@ -14,37 +14,141 @@ pinned: false
 
 # Metric Card for RestrictedPython Code Eval
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
-
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+A code-based evaluation metric, with the same logic as [`code_eval`](https://huggingface.co/spaces/evaluate-metric/code_eval).
 
 ## How to Use
-*Give general statement of how to use the metric*
+```python
+from evaluate import load
+code_eval = load("RestrictedPython_code_eval")
+test_cases = ["assert add(2,3)==5"]
+candidates = [["def add(a,b): return a*b", "def add(a, b): return a+b"]]
+pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1, 2], use_safe_builtins=True)
+```
 
-*Provide simplest possible example for using the metric*
+N.B.
+This metric exists to run untrusted model-generated code. Users are strongly encouraged not to do so outside of a robust security sandbox. Before running this metric and once you've taken the necessary precautions, you will need to set the `HF_ALLOW_CODE_EVAL` environment variable. Use it at your own risk:
+```python
+import os
+os.environ["HF_ALLOW_CODE_EVAL"] = "1"` 
+```
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+The following arguments are inherited from the basic `code_eval`: 
+
+- **`predictions`** (`List[List[str]]`): a list of candidates to evaluate. Each candidate should be a list of strings with several code candidates to solve the problem.
+
+- **`references`** (`List[str]`): a list with a test for each prediction. Each test should evaluate the correctness of a code candidate.
+
+- **`k`** (`List[int]`): number of code candidates to consider in the evaluation. The default value is `[1, 10, 100]`.
+
+- **`num_workers`** (`int`): the number of workers used to evaluate the candidate programs (The default value is `4`).
+
+- **`timeout`** (`float`): The maximum time taken to produce a prediction before it is considered a "timeout". The default value is `3.0` (i.e. 3 seconds).
+
+In addition, this metric supports three additional arguments, specifying which default imports should be made available: 
+
+- **`use_safe_builtins`** (`bool`): Whether or not to allow the usage of [`RestrictedPython.safe_builtins`](https://github.com/zopefoundation/RestrictedPython/blob/c31c133844ac2308f5cc930e934a7227a2a6a77b/src/RestrictedPython/Guards.py#L23), defaults to True
+
+- **`use_limited_builtins`** (`bool`): Whether or not to allow the usage of [`RestrictedPython.limited_builtins`](https://github.com/zopefoundation/RestrictedPython/blob/c31c133844ac2308f5cc930e934a7227a2a6a77b/src/RestrictedPython/Limits.py#L14), which provides limited implementations of `range`, `list`, and `tuple`.  defaults to True. 
+
+- **`use_utility_builtins`** (`bool`): Whether or not to allow the usage of [`RestrictedPython.utility_builtins`](https://github.com/zopefoundation/RestrictedPython/blob/c31c133844ac2308f5cc930e934a7227a2a6a77b/src/RestrictedPython/Utilities.py#L19), which includes the `string`, `math`, `random`, and `set` packages, among others. Defaults to True. 
+
+As the additional arguments are optional, this could be used as a drop-in replacement for `code_eval`.
 
 ### Output Values
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+Identical to `code_eval`:
+
+The Code Eval metric outputs two things:
 
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
+`pass_at_k`: a dictionary with the pass rates for each k value defined in the arguments.
 
-#### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
+`results`: a dictionary with granular results of each unit test.
+
+### Values from popular papers
+The [original CODEX paper](https://arxiv.org/pdf/2107.03374.pdf) reported that the CODEX-12B model had a pass@k score of 28.8% at `k=1`, 46.8% at `k=10` and 72.3% at `k=100`. However, since the CODEX model is not open source, it is hard to verify these numbers. 
 
 ### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+Copied from the `code_eval` model card:
+
+Full match at `k=1`:
+
+```python
+from evaluate import load
+code_eval = load("code_eval")
+test_cases = ["assert add(2,3)==5"]
+candidates = [["def add(a, b): return a+b"]]
+pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1])
+print(pass_at_k)
+{'pass@1': 1.0}
+```
+
+No match for k = 1:
+
+```python
+from evaluate import load
+code_eval = load("code_eval")
+test_cases = ["assert add(2,3)==5"]
+candidates = [["def add(a,b): return a*b"]]
+pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1])
+print(pass_at_k)
+{'pass@1': 0.0}
+```
+
+Partial match at k=1, full match at k=2:
+
+```python
+from evaluate import load
+code_eval = load("code_eval")
+test_cases = ["assert add(2,3)==5"]
+candidates = [["def add(a, b): return a+b", "def add(a,b): return a*b"]]
+pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1, 2])
+print(pass_at_k)
+{'pass@1': 0.5, 'pass@2': 1.0}
+```
 
 ## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+From the original `code_eval` model card:
+
+As per the warning included in the metric code itself:
+> This program exists to execute untrusted model-generated code. Although it is highly unlikely that model-generated code will do something overtly malicious in response to this test suite, model-generated code may act destructively due to a lack of model capability or alignment. Users are strongly encouraged to sandbox this evaluation suite so that it does not perform destructive actions on their host or network. For more  information on how OpenAI sandboxes its code, see the accompanying paper. Once you have read this disclaimer and taken appropriate precautions, uncomment the following line and proceed at your own risk:
+
+More information about the limitations of the code can be found on the [Human Eval Github repository](https://github.com/openai/human-eval).
+
+Additionally, this metric does not currently allow for custom `RestrictedPython` policies -- so any code that depends on non-default libraries or packages may fail for that reason. 
+
+**TODO**: Add a `use_custom_builtins` argument that allows users to specify their own `RestrictedPython` policy. See the RestrictedPython [documentation](https://restrictedpython.readthedocs.io/en/latest/usage/policy.html#implementing-a-policy) for additional details.
+
 
 ## Citation
-*Cite the source where this metric was introduced.*
+Based on the original `code_eval` metric, which cites:
+```bibtex
+@misc{chen2021evaluating,
+      title={Evaluating Large Language Models Trained on Code},
+      author={Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan \
+and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards \
+and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray \
+and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf \
+and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray \
+and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser \
+and Mohammad Bavarian and Clemens Winter and Philippe Tillet \
+and Felipe Petroski Such and Dave Cummings and Matthias Plappert \
+and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss \
+and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak \
+and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain \
+and William Saunders and Christopher Hesse and Andrew N. Carr \
+and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa \
+and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati \
+and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei \
+and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},
+      year={2021},
+      eprint={2107.03374},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+```
 
 ## Further References
-*Add any useful further references.*
+- The original `code_eval` metric: https://huggingface.co/spaces/evaluate-metric/code_eval
+- RestrictedPython: https://restrictedpython.readthedocs.io/en/latest/index.html

restrictedpython_code_eval.py

@@ -47,7 +47,7 @@ year={2020}
 
 # TODO: Add description of the module here
 _DESCRIPTION = """\
-This module tries to extend the built in `code_eval` module to use restricted python.
+This module implements the same logic as the baseline `code_eval` module but using RestrictedPython.
 """
 
 
@@ -69,7 +69,7 @@ Returns:
     pass_at_k: dict with pass rates for each k
     results: dict with granular results of each unittest
 Examples:
-    >>> code_eval = evaluate.load("code_eval")
+    >>> code_eval = evaluate.load("RestrictedPython_code_eval")
     >>> test_cases = ["assert add(2,3)==5"]
     >>> candidates = [["def add(a,b): return a*b", "def add(a, b): return a+b"]]
     >>> pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1, 2])