| | --- |
| | datasets: |
| | - ML4SE2023-G1-WizardCoder/EvolInstruct-SCoT-1k |
| | language: |
| | - en |
| | tags: |
| | - code |
| | --- |
| | |
| | # ML4SE23_G1_WizardCoder-SCoT-1B-V1.0 |
| |
|
| | IN4334 ML4SE |
| |
|
| | Group1 WizardCoder |
| |
|
| | This model is the result of the fine-tunign of the WizardCoder-1B-V1.0 model using Structured Chain-of-Though (S-CoT) enhanced instructions. |
| | S-CoT is used to enhance a sample of about 1200 entries from the Evol-Instruct 80k dataset. |
| | The resulting dataset is then used for the training task. |
| | The current WizardCoder model and the new S-CoT fine-tuned one are compared on both versions of HumanEval and MBPP (S-CoT enhanced and not) on the pass@1 metric. |
| | The S-CoT enhancement of the evaluation datasets allows to study its effect when used just as a prompting technique, independently of the S-CoT fine-tuning of the model. |
| |
|
| | ## Fine-tuning Details |
| |
|
| | | Hyperparameter | [WizardCoder-1B-V1.0](https://huggingface.co/WizardLM/WizardCoder-1B-V1.0) | |
| | |----------------|---------------------| |
| | | Batch size | 16 | |
| | | Learning rate | 2e-5 | |
| | | Epochs | 3 | |
| | | Max length | 2048 | |
| | | Warmup step | 30 | |
| | | LR scheduler | cosine | |
| | | Dataset | [ML4SE23_G1_EvolInstruct-SCoT-1k](https://huggingface.co/datasets/ML4SE2023-G1-WizardCoder/ML4SE23_G1_EvolInstruct-SCoT-1k) | |
| |
|
| | The hardware consisted on a GPU instance rented from [DataCrunch](https://datacrunch.io/) with the following specifications: |
| |
|
| | | NVidia RTX A6000 48GB 1A6000.10V | |
| | |----------------------------------| |
| | | 2 GPUs | |
| | | 48GB VRAM per GPU | |
| | | 60 GB RAM | |
| | | 10 CPUs | |
| | | 100GB SSD Storage | |
| | | Ubuntu 20.04 | |
| | | CUDA 11.6 | |
| |
|
| | ## Results |
| |
|
| | Results of pass@1(%) on HumanEval and MBPP compared to HumanEval-SCoT and MBPP-SCoT using WizardCoder-1B, WizardCoder-SCoT-1B and WizardCoder-15B. |
| |
|
| | | **Dataset** | **WizardCoder-1B-V1.0** | **WizardCoder-SCoT-1B-V1.0** | **WizardCoder-15B-V1.0** | |
| | |----------------|-------------------------|------------------------------|--------------------------| |
| | | HumanEval | 23.78 | **17.68** | 57.3 | |
| | | HumanEval-SCoT | **44.51** | **27.44** | **57.3** | |
| | | MBPP | 23.4 | **19.4** | 51.8 | |
| | | MBPP-SCoT | **40** | **28** | **45.6** | |
| |
|
