30-seconds-of-code-eval

An evaluation harness for 30-seconds-of-code snippets, in similar to HumanEval problem solving dataset described in the paper "Evaluating Large Language Models Trained on Code".

In contrast to HumanEval, a dataset of 164 hand-written python programming problems, we use 30-seconds-of-code to build few short code snippets datasets, similar in structure to HumanEval, in Python , React , Go and PHP.

Quick Setup

Clone project

$ git clone [email protected]:talmago/30-seconds-of-code-eval.git

Install dependencies

$ pip install -r requirements.txt

Notice: python 3.7 or later is a requirement. If you use conda, follow these instructions.

Usage

Build a problem set

$ python 01_download_problem_set.py --language_name python

Output will be saved to data/30_seconds_of_python_problem.jsonl .

Collect samples (via Codex)

$ OPENAI_API_KEY=XXXX python 02_get_completions_from_codex.py data/30_seconds_of_python_problem.jsonl \
  --model_name davinci-codex \
  --max_length 70 \
  --temperature 0.8

Output will be saved to data/30_seconds_of_python_samples.jsonl .

Preview

$ python 03_preview.py data/30_seconds_of_python_samples.jsonl

Output will be saved to examples/30_seconds_of_python.md

Evaluation

Coming soon ...

References

[1] Evaluating Large Language Models Trained on Code

@article{chen2021codex,
  title={Evaluating Large Language Models Trained on Code},
  author={Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser and Mohammad Bavarian and Clemens Winter and Philippe Tillet and Felipe Petroski Such and Dave Cummings and Matthias Plappert and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain and William Saunders and Christopher Hesse and Andrew N. Carr and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},
  year={2021},
  eprint={2107.03374},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}