| --- |
| language: |
| - code |
| license: apache-2.0 |
| tags: |
| - code |
| - gpt2 |
| - generation |
| datasets: |
| - "codeparrot/codeparrot-clean" |
| - "openai_humaneval" |
| metrics: |
| - "evaluate-metric/code_eval" |
| --- |
| |
| # CodeParrot 🦜 (small) |
|
|
| CodeParrot 🦜 is a GPT-2 model (110M parameters) trained to generate Python code. |
|
|
| ## Usage |
|
|
| You can load the CodeParrot model and tokenizer directly in `transformers`: |
|
|
| ```Python |
| from transformers import AutoTokenizer, AutoModelWithLMHead |
| |
| tokenizer = AutoTokenizer.from_pretrained("codeparrot/codeparrot-small") |
| model = AutoModelWithLMHead.from_pretrained("codeparrot/codeparrot-small") |
| |
| inputs = tokenizer("def hello_world():", return_tensors="pt") |
| outputs = model(**inputs) |
| ``` |
|
|
| or with a `pipeline`: |
|
|
| ```Python |
| from transformers import pipeline |
| |
| pipe = pipeline("text-generation", model="codeparrot/codeparrot-small") |
| outputs = pipe("def hello_world():") |
| ``` |
|
|
| ## Training |
|
|
| The model was trained on the cleaned [CodeParrot 🦜 dataset](https://huggingface.co/datasets/codeparrot/codeparrot-clean) with the following settings: |
|
|
| |Config|Value| |
| |-------|-----| |
| |Batch size| 192 | |
| |Context size| 1024 | |
| |Training steps| 150'000| |
| |Gradient accumulation| 1| |
| |Gradient checkpointing| False| |
| |Learning rate| 5e-4 | |
| |Weight decay | 0.1 | |
| |Warmup steps| 2000 | |
| |Schedule| Cosine | |
|
|
| The training was executed on 16 x A100 (40GB) GPUs. This setting amounts to roughly 29 billion tokens. |
|
|
| ## Performance |
|
|
| We evaluated the model on OpenAI's [HumanEval](https://huggingface.co/datasets/openai_humaneval) benchmark which consists of programming challenges: |
|
|
| | Metric | Value | |
| |-------|-----| |
| |pass@1 | 3.80% | |
| |pass@10 | 6.57% | |
| |pass@100 | 12.78% | |
|
|
| The [pass@k metric](https://huggingface.co/metrics/code_eval) tells the probability that at least one out of k generations passes the tests. |
|
|
| ## Resources |
|
|
| - Dataset: [full](https://huggingface.co/datasets/codeparrot/codeparrot-clean), [train](https://huggingface.co/datasets/codeparrot/codeparrot-clean-train), [valid](https://huggingface.co/datasets/codeparrot/codeparrot-clean-valid) |
| - Code: [repository](https://github.com/huggingface/transformers/tree/master/examples/research_projects/codeparrot) |
| - Spaces: [generation](), [highlighting]() |
