Note great overview and a lot of additional references! frequently updated list: https://github.com/codefuse-ai/Awesome-Code-LLM

Note introduces HumanEval and pass@k

Note has already been used for marketing: Devin Most promising benchmark currently. They model the actual software engineer by using github issues as inputs, the whole repository as resource. This benchmark tests systems - not just models. So you can have agent like managers and retrieval systems. rapidly advancing leaderboard: https://www.swebench.com/

Note an older work that consists of several easy task for encoder and decoder models. For example line completion or min/max cloze test

Note human judgement doesn't agree with static metrics (BLEU, ChrF, RUBY ...)

Note perturbations on docstrings/prompts

Note phi-1 model, novel evaluation problems that combine two tasks to make data contamination less likely.

Note Annotate problems by months and spot potential contamination

Note model context is whole file up untill funciton header, ground turth the source file, first AST, then Linter, errors in context cause errors in generation, undefined/name error most common, EOF error due to generation length

Note introduces Mostly Basic Programming Problems (MBPP)

Note has been renamed "Python Saga". Figure 6 is a great sign of saturation.

Note "non functional" -> focusses on instruct models, has classification tasks and non functional requirements like efficiency, security and also maintability. Still relies on gold labels using DiffBLEU. filters examples larger than 3k tokens using starcoder tokenizer GPT4 seems to be really good on these tasks. might be a lot of prompt engineering tho

Note this paper has been withdrawn!

Note HumanEvalPlus with additional test cases has a great figure to rank HumanEval problems on passrate, showcasing that some of them are much easier than others.

Note Verilog generation. creativity here means ". This term refers to the capacity to think innovatively—the ability to formulate new solutions or connections that are effective and unconventional [11]." where the reference is: https://doi.org/10.1080/10400419.2012.650092 "fluency" is how many of the pass@k variants are unique?(but this will be skewed towards larger models, right - they sample wider)

Note NeoCoder: denial prompting to get novel approaches, even outside of "historical human solutions". (cont)

Note contamination directly: every humaneval problem is at least 43 times on github contamination indirectly: high similarity with synthetic rephrases for instruction learning overfitting on benchmarks: humaneval and MBPP should be considered "dev sets" also add LBPP (less basic ....)