Update README.md

README.md

@@ -20,6 +20,7 @@ Jam-CGPT is a GPT2-like model that follows [jam](https://huggingface.co/apcl/jam
 |d | dropout							| 0.20 |		 
 |r | learning rate                     | 3e-5 |		 
 |y | weight decay						| 1e-5 |	
+|iter | number of iterations after pretraing						| 757,000 |	
 
 ## Jam-CGPT 110 million parameters model
 | Hyperparameter | Description | Value |
@@ -33,6 +34,7 @@ Jam-CGPT is a GPT2-like model that follows [jam](https://huggingface.co/apcl/jam
 |d | dropout							| 0.20 |		 
 |r | learning rate                     | 3e-5 |		 
 |y | weight decay						| 1e-5 |	
+|iter | number of iterations after pretraing						| 762,000 |	
 
 
 ## Jam-CGPT 350 million parameters model
@@ -46,8 +48,9 @@ Jam-CGPT is a GPT2-like model that follows [jam](https://huggingface.co/apcl/jam
 |a | accumulation steps				| 32 |		 
 |d | dropout							| 0.20 |		 
 |r | learning rate                     | 3e-5 |		 
-|y | weight decay						| 1e-5 |	
+|y | weight decay						| 1e-5 |
+|iter | weight decay						| 272,000 |
 
 - Note that you can adjust the batch size and accumulation steps based on your GPU memory. But, the batch size * accumulation steps should be 128.
 - If you finetune your models with multiple GPUs, you can turn down accumulation steps. For example, if you finetune with 2 GPUs, you will need to half the accumulation steps.
-
+- We pretrained 38m and 110m models for 3 epochs.