| | --- |
| | license: apache-2.0 |
| | datasets: |
| | - lambdasec/cve-single-line-fixes |
| | - lambdasec/gh-top-1000-projects-vulns |
| | language: |
| | - code |
| | tags: |
| | - code |
| | programming_language: |
| | - Java |
| | - JavaScript |
| | - Python |
| | inference: false |
| | model-index: |
| | - name: SantaFixer |
| | results: |
| | - task: |
| | type: text-generation |
| | dataset: |
| | type: openai/human-eval-infilling |
| | name: HumanEval |
| | metrics: |
| | - name: single-line infilling pass@1 |
| | type: pass@1 |
| | value: 0.47 |
| | verified: false |
| | - name: single-line infilling pass@10 |
| | type: pass@10 |
| | value: 0.74 |
| | verified: false |
| | - task: |
| | type: text-generation |
| | dataset: |
| | type: lambdasec/gh-top-1000-projects-vulns |
| | name: GH Top 1000 Projects Vulnerabilities |
| | metrics: |
| | - name: pass@1 (Java) |
| | type: pass@1 |
| | value: 0.26 |
| | verified: false |
| | - name: pass@10 (Java) |
| | type: pass@10 |
| | value: 0.48 |
| | verified: false |
| | - name: pass@1 (Python) |
| | type: pass@1 |
| | value: 0.31 |
| | verified: false |
| | - name: pass@10 (Python) |
| | type: pass@10 |
| | value: 0.56 |
| | verified: false |
| | - name: pass@1 (JavaScript) |
| | type: pass@1 |
| | value: 0.36 |
| | verified: false |
| | - name: pass@10 (JavaScript) |
| | type: pass@10 |
| | value: 0.62 |
| | verified: false |
| | --- |
| | |
| | # Model Card for SantaFixer |
| |
|
| | <!-- Provide a quick summary of what the model is/does. --> |
| |
|
| | This is a LLM for code that is focussed on generating bug fixes using infilling. |
| |
|
| | ## Model Details |
| |
|
| | ### Model Description |
| |
|
| | <!-- Provide a longer summary of what this model is. --> |
| |
|
| |
|
| |
|
| | - **Developed by:** [codelion](https://huggingface.co/codelion) |
| | - **Model type:** GPT-2 |
| | - **Finetuned from model:** [bigcode/santacoder](https://huggingface.co/bigcode/santacoder) |
| |
|
| |
|
| | ## How to Get Started with the Model |
| |
|
| | Use the code below to get started with the model. |
| |
|
| | ```python |
| | # pip install -q transformers |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | checkpoint = "lambdasec/santafixer" |
| | device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') |
| | |
| | tokenizer = AutoTokenizer.from_pretrained(checkpoint) |
| | model = AutoModelForCausalLM.from_pretrained(checkpoint, |
| | trust_remote_code=True).to(device) |
| | |
| | input_text = "<fim-prefix>def print_hello_world():\n |
| | <fim-suffix>\n print('Hello world!') |
| | <fim-middle>" |
| | inputs = tokenizer.encode(input_text, return_tensors="pt").to(device) |
| | outputs = model.generate(inputs) |
| | print(tokenizer.decode(outputs[0])) |
| | ``` |
| |
|
| | ## Training Details |
| |
|
| | - **GPU:** Tesla P100 |
| | - **Time:** ~5 hrs |
| |
|
| | ### Training Data |
| |
|
| | The model was fine-tuned on the [CVE single line fixes dataset](https://huggingface.co/datasets/lambdasec/cve-single-line-fixes) |
| |
|
| | ### Training Procedure |
| |
|
| | Supervised Fine Tuning (SFT) |
| |
|
| | #### Training Hyperparameters |
| |
|
| | - **optim:** adafactor |
| | - **gradient_accumulation_steps:** 4 |
| | - **gradient_checkpointing:** true |
| | - **fp16:** false |
| | |
| | ## Evaluation |
| | |
| | The model was tested with the [GitHub top 1000 projects vulnerabilities dataset](https://huggingface.co/datasets/lambdasec/gh-top-1000-projects-vulns) |
