| [CodeGen](https://huggingface.co/Salesforce/codegen-16B-mono) architecture follows a standard transformer decoder with left-to-right causal masking. With rotary position embedding for the positional encoding [(Su et al., 2021)](https://arxiv.org/abs/2104.09864), and a context length of 2048. CodeGen models are trained in various sizes. |
|
|
| <div align="center"> |
|
|
| |Model | # parameters | |
| | - | - | |
| | Decoder | 350M | |
| | Decoder | 2.7B | |
| | Decoder | 6.1B | |
| | Decoder | 16.1B | |
|
|
| </div> |
|
|
|
|
| You can load the model and tokenizer directly from [`transformers`](https://huggingface.co/docs/transformers/index): |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
| tokenizer = AutoTokenizer.from_pretrained('Salesforce/codegen-16B-mono') |
| model = AutoModelForCausalLM.from_pretrained('Salesforce/codegen-16B-mono') |
|
|
| inputs = tokenizer("def hello_world():", return_tensors="pt") |
| outputs = model(**inputs) |
| ``` |
