Create README.md

README.md

@@ -0,0 +1,23 @@
+## A Lossless Syntax Tree Generator with Zero-shot Error Correction
+
+- We follow [jam](https://huggingface.co/apcl/jam)'s pretraining procedure and use the same data to pretrain except we also use srcml to pretrain the models.
+- In the finetuning stage, we finetune our models for 3 epochs.
+- Our [GitHub repo](https://github.com/apcl-research/autorepair) contains the code for reproduction using the same [data](https://huggingface.co/datasets/apcl/autorepair).
+
+
+## Pretrained model parameters
+| Hyperparameter | Description | Value |
+| ----------- | ----------- |------------|
+|e | embedding dimensions              | 1024 |		 
+|L | number of layers 			  		| 24 | 		 
+|h | attention heads             		| 16 |		 
+|c | block size / context length       | 256 |  		 
+|b | batch size                        | 4  | 		 
+|a | accumulation steps				| 32 |		 	 
+|r | learning rate                     | 3e-5 |		 
+|y | weight decay						| 1e-5 |
+|iter | iterations						| 570000  |
+
+- Note that you can adjust the batch size and accumulation steps based on your GPU memory. But, the batch size * accumulation steps should be 128.
+- If you finetune your models with multiple GPUs, you can turn down accumulation steps. For example, if you finetune with 2 GPUs, you will need to half the accumulation steps.
+