Create README.md

README.md

@@ -0,0 +1,23 @@
+## Jam-Contextsum
+
+Jam-Contextsum is a GPT2-like model finetuned to generate summary on why the method exists.
+
+## Jam-Contextsum Training Details
+
+- ckpt_pretrain is the file that we use to finetune the model for generating the summary on why the method exists
+- Our [GitHub repo](https://github.com/apcl-research/jam-contextsum) contains the code for reproduction using the same [data](https://huggingface.co/datasets/apcl/jam_contextsum).
+
+
+## ckpt_pretrain.pt
+| Hyperparameter | Description | Value |
+| ----------- | ----------- |------------|
+|e | embedding dimensions              | 512 |		 
+|L | number of layers 			  		| 4 | 		 
+|h | attention heads             		| 4 |		 
+|c | block size / context length       | 1,024 |  		 
+|b | batch size                        | 4  | 		 
+|a | accumulation steps				| 32 |		 
+|d | dropout							| 0.20 |		 
+|r | learning rate                     | 3e-5 |		 
+|y | iterations						| 1e-5 |	
+|iter | number of iterations after pretraing						| 137,900  |