End of training

README.md

@@ -16,14 +16,14 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 216.0
-- eval_frwikippl: 876.0
-- eval_zhwikippl: 173.0
-- eval_tinystoriesppl: 178.0
-- eval_loss: 1.2693
-- eval_runtime: 25.4344
-- eval_samples_per_second: 98.292
-- eval_steps_per_second: 12.306
+- eval_enwikippl: 208.0
+- eval_frwikippl: 796.0
+- eval_zhwikippl: 166.0
+- eval_tinystoriesppl: 166.0
+- eval_loss: 1.2497
+- eval_runtime: 25.4899
+- eval_samples_per_second: 98.078
+- eval_steps_per_second: 12.279
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -53,80 +53,80 @@ The following hyperparameters were used during training:
 - eval_batch_size: 8
 - seed: 42
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
+- lr_scheduler_type: cosine
 - lr_scheduler_warmup_ratio: 0.5
 - num_epochs: 1.0
 
 ### Resource Usage
-Peak GPU Memory: 7.2012 GB
+Peak GPU Memory: 7.2014 GB
 
 ### Eval-Phase Metrics
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 43.25 | 61.25 |  |  |  |  | 11.6875 | 19.125 |
-| 0 | 0 | 1606317768704.0 | 119297011613696.0 | 20.7344 | 25.4007 | 98.423 | 12.323 | 5066719232.0 | 19104014532608.0 |
-| 1000 | 0.0162 | 81408.0 | 958464.0 | 5.1263 | 25.4848 | 98.098 | 12.282 | 16384.0 | 1253376.0 |
-| 2000 | 0.0323 | 4032.0 | 47360.0 | 3.2154 | 25.5066 | 98.014 | 12.271 | 2368.0 | 256000.0 |
-| 3000 | 0.0485 | 1104.0 | 5792.0 | 2.3506 | 25.4684 | 98.161 | 12.29 | 724.0 | 8960.0 |
-| 4000 | 0.0646 | 632.0 | 3984.0 | 1.9871 | 25.4984 | 98.046 | 12.275 | 410.0 | 2368.0 |
-| 5000 | 0.0808 | 418.0 | 2096.0 | 1.6792 | 25.4486 | 98.237 | 12.299 | 310.0 | 572.0 |
-| 6000 | 0.0970 | 310.0 | 1384.0 | 1.5115 | 25.5353 | 97.904 | 12.258 | 241.0 | 284.0 |
-| 7000 | 0.1131 | 256.0 | 1040.0 | 1.3917 | 25.5364 | 97.899 | 12.257 | 203.0 | 205.0 |
-| 8000 | 0.1293 | 229.0 | 892.0 | 1.3296 | 25.4687 | 98.16 | 12.29 | 183.0 | 224.0 |
-| 9000 | 0.1455 | 216.0 | 876.0 | 1.2693 | 25.4344 | 98.292 | 12.306 | 178.0 | 173.0 |
-| 10000 | 0.1616 | 192.0 | 716.0 | 1.1774 | 25.4691 | 98.158 | 12.289 | 165.0 | 198.0 |
-| 11000 | 0.1778 | 255.0 | 824.0 | 1.0750 | 25.5641 | 97.793 | 12.244 | 296.0 | 169.0 |
-| 12000 | 0.1939 | 264.0 | 744.0 | 0.9937 | 25.5508 | 97.844 | 12.25 | 292.0 | 147.0 |
-| 13000 | 0.2101 | 231.0 | 608.0 | 0.9388 | 25.6185 | 97.586 | 12.218 | 192.0 | 158.0 |
-| 14000 | 0.2263 | 221.0 | 552.0 | 0.8904 | 25.5619 | 97.802 | 12.245 | 178.0 | 127.5 |
-| 15000 | 0.2424 | 221.0 | 608.0 | 0.8620 | 25.5139 | 97.986 | 12.268 | 145.0 | 128.0 |
-| 16000 | 0.2586 | 194.0 | 564.0 | 0.8357 | 25.4869 | 98.09 | 12.281 | 123.0 | 152.0 |
-| 17000 | 0.2747 | 143.0 | 468.0 | 0.8056 | 25.4845 | 98.099 | 12.282 | 88.0 | 131.0 |
-| 18000 | 0.2909 | 135.0 | 512.0 | 0.7808 | 25.6085 | 97.624 | 12.222 | 81.5 | 121.5 |
-| 19000 | 0.3071 | 130.0 | 484.0 | 0.8070 | 25.6119 | 97.611 | 12.221 | 80.5 | 123.5 |
-| 20000 | 0.3232 | 102.5 | 414.0 | 0.7258 | 25.5902 | 97.694 | 12.231 | 73.0 | 136.0 |
-| 21000 | 0.3394 | 92.5 | 370.0 | 0.6690 | 25.484 | 98.101 | 12.282 | 72.5 | 117.0 |
-| 22000 | 0.3556 | 89.5 | 320.0 | 0.6363 | 25.4959 | 98.055 | 12.277 | 72.0 | 109.5 |
-| 23000 | 0.3717 | 93.5 | 302.0 | 0.6073 | 25.5285 | 97.93 | 12.261 | 76.0 | 105.5 |
-| 24000 | 0.3879 | 93.5 | 306.0 | 0.5868 | 25.52 | 97.962 | 12.265 | 76.5 | 106.0 |
-| 25000 | 0.4040 | 126.5 | 336.0 | 0.5717 | 25.5413 | 97.881 | 12.255 | 108.0 | 132.0 |
-| 26000 | 0.4202 | 72.0 | 250.0 | 0.5578 | 25.5841 | 97.717 | 12.234 | 61.25 | 114.0 |
-| 27000 | 0.4364 | 87.5 | 262.0 | 0.5462 | 25.5236 | 97.948 | 12.263 | 63.75 | 116.0 |
-| 28000 | 0.4525 | 72.0 | 228.0 | 0.5431 | 25.5362 | 97.9 | 12.257 | 57.75 | 107.0 |
-| 29000 | 0.4687 | 75.5 | 247.0 | 0.5524 | 25.5375 | 97.895 | 12.256 | 56.75 | 105.5 |
-| 30000 | 0.4848 | 72.0 | 222.0 | 0.5459 | 25.5244 | 97.945 | 12.263 | 53.25 | 114.5 |
-| 31000 | 0.5010 | 71.5 | 230.0 | 0.5460 | 25.4562 | 98.208 | 12.296 | 54.25 | 151.0 |
-| 32000 | 0.5172 | 69.5 | 246.0 | 0.5391 | 25.5258 | 97.94 | 12.262 | 54.5 | 194.0 |
-| 33000 | 0.5333 | 70.5 | 242.0 | 0.5283 | 25.4905 | 98.076 | 12.279 | 53.75 | 117.5 |
-| 34000 | 0.5495 | 76.5 | 246.0 | 0.5147 | 25.4567 | 98.206 | 12.295 | 62.75 | 114.0 |
-| 35000 | 0.5657 | 68.0 | 216.0 | 0.5014 | 25.4939 | 98.063 | 12.277 | 50.25 | 87.0 |
-| 36000 | 0.5818 | 71.0 | 225.0 | 0.4986 | 25.4871 | 98.089 | 12.281 | 52.75 | 90.0 |
-| 37000 | 0.5980 | 79.5 | 231.0 | 0.4939 | 25.509 | 98.005 | 12.27 | 54.0 | 112.5 |
-| 38000 | 0.6141 | 75.5 | 232.0 | 0.4880 | 25.4997 | 98.04 | 12.275 | 58.75 | 116.5 |
-| 39000 | 0.6303 | 82.0 | 241.0 | 0.4879 | 25.519 | 97.966 | 12.265 | 61.75 | 123.0 |
-| 40000 | 0.6465 | 73.5 | 220.0 | 0.4759 | 25.4982 | 98.046 | 12.275 | 55.25 | 126.0 |
-| 41000 | 0.6626 | 70.5 | 212.0 | 0.4738 | 25.5094 | 98.003 | 12.27 | 55.5 | 102.0 |
-| 42000 | 0.6788 | 79.0 | 227.0 | 0.4699 | 25.527 | 97.935 | 12.262 | 59.25 | 82.5 |
-| 43000 | 0.6949 | 82.0 | 227.0 | 0.4613 | 25.5453 | 97.865 | 12.253 | 59.25 | 92.5 |
-| 44000 | 0.7111 | 78.0 | 210.0 | 0.4227 | 25.5427 | 97.875 | 12.254 | 61.5 | 79.0 |
-| 45000 | 0.7273 | 72.0 | 192.0 | 0.4068 | 25.4354 | 98.288 | 12.306 | 55.5 | 72.0 |
-| 46000 | 0.7434 | 72.0 | 185.0 | 0.3952 | 25.5194 | 97.965 | 12.265 | 56.0 | 53.0 |
-| 47000 | 0.7596 | 68.0 | 178.0 | 0.3928 | 25.5018 | 98.032 | 12.274 | 51.25 | 54.25 |
-| 48000 | 0.7758 | 71.5 | 183.0 | 0.3873 | 25.5018 | 98.032 | 12.274 | 54.5 | 63.0 |
-| 49000 | 0.7919 | 71.0 | 177.0 | 0.3853 | 25.5149 | 97.982 | 12.267 | 52.5 | 66.5 |
-| 50000 | 0.8081 | 71.0 | 174.0 | 0.3802 | 25.4786 | 98.122 | 12.285 | 54.75 | 61.75 |
-| 51000 | 0.8242 | 73.5 | 184.0 | 0.3797 | 25.4575 | 98.203 | 12.295 | 56.25 | 59.0 |
-| 52000 | 0.8404 | 75.5 | 187.0 | 0.3784 | 25.4298 | 98.31 | 12.308 | 57.5 | 68.5 |
-| 53000 | 0.8566 | 74.5 | 185.0 | 0.3741 | 25.5041 | 98.023 | 12.273 | 55.75 | 59.75 |
-| 54000 | 0.8727 | 73.0 | 178.0 | 0.3686 | 25.5431 | 97.874 | 12.254 | 54.75 | 56.25 |
-| 55000 | 0.8889 | 72.0 | 180.0 | 0.3649 | 25.447 | 98.244 | 12.3 | 55.0 | 55.0 |
-| 56000 | 0.9051 | 71.5 | 179.0 | 0.3636 | 25.5222 | 97.954 | 12.264 | 53.75 | 53.0 |
-| 57000 | 0.9212 | 71.5 | 176.0 | 0.3619 | 25.4887 | 98.083 | 12.28 | 54.25 | 50.5 |
-| 58000 | 0.9374 | 72.0 | 177.0 | 0.3605 | 25.4451 | 98.251 | 12.301 | 55.0 | 50.25 |
-| 59000 | 0.9535 | 72.0 | 177.0 | 0.3598 | 25.5041 | 98.023 | 12.273 | 54.5 | 50.5 |
-| 60000 | 0.9697 | 72.0 | 177.0 | 0.3589 | 25.5086 | 98.006 | 12.27 | 54.0 | 50.5 |
-| 61000 | 0.9859 | 72.0 | 177.0 | 0.3588 | 25.4042 | 98.409 | 12.321 | 54.5 | 50.5 |
-| 61875 | 1.0 | 72.0 | 177.0 | 0.3587 | 25.56 | 97.809 | 12.246 | 54.25 | 50.25 |
+| 0 | 0 | 854698491904.0 | 94557999988736.0 | 18.8626 | 25.4462 | 98.246 | 12.3 | 2550136832.0 | 34497177321472.0 |
+| 1000 | 0.0162 | 55040.0 | 1228800.0 | 4.9530 | 25.546 | 97.863 | 12.252 | 10688.0 | 3080192.0 |
+| 2000 | 0.0323 | 4048.0 | 37888.0 | 3.1978 | 25.5234 | 97.949 | 12.263 | 2816.0 | 327680.0 |
+| 3000 | 0.0485 | 1096.0 | 6464.0 | 2.3355 | 25.567 | 97.782 | 12.242 | 752.0 | 7296.0 |
+| 4000 | 0.0646 | 624.0 | 3904.0 | 1.9682 | 25.5721 | 97.763 | 12.24 | 418.0 | 1656.0 |
+| 5000 | 0.0808 | 416.0 | 2128.0 | 1.6585 | 25.4891 | 98.081 | 12.28 | 290.0 | 406.0 |
+| 6000 | 0.0970 | 312.0 | 1424.0 | 1.4638 | 25.6034 | 97.643 | 12.225 | 243.0 | 234.0 |
+| 7000 | 0.1131 | 260.0 | 872.0 | 1.3739 | 25.6059 | 97.634 | 12.224 | 198.0 | 210.0 |
+| 8000 | 0.1293 | 232.0 | 768.0 | 1.2985 | 25.5335 | 97.911 | 12.258 | 181.0 | 197.0 |
+| 9000 | 0.1455 | 208.0 | 796.0 | 1.2497 | 25.4899 | 98.078 | 12.279 | 166.0 | 166.0 |
+| 10000 | 0.1616 | 180.0 | 644.0 | 1.1413 | 25.5166 | 97.975 | 12.267 | 151.0 | 145.0 |
+| 11000 | 0.1778 | 155.0 | 552.0 | 1.0405 | 25.5119 | 97.994 | 12.269 | 123.5 | 161.0 |
+| 12000 | 0.1939 | 143.0 | 510.0 | 0.9691 | 25.5958 | 97.672 | 12.229 | 114.0 | 136.0 |
+| 13000 | 0.2101 | 127.0 | 446.0 | 0.9089 | 25.5873 | 97.705 | 12.233 | 101.0 | 153.0 |
+| 14000 | 0.2263 | 117.5 | 480.0 | 0.8634 | 25.5335 | 97.911 | 12.258 | 90.5 | 126.5 |
+| 15000 | 0.2424 | 113.5 | 466.0 | 0.8469 | 25.5278 | 97.932 | 12.261 | 85.5 | 108.0 |
+| 16000 | 0.2586 | 107.0 | 434.0 | 0.8045 | 25.5947 | 97.677 | 12.229 | 85.0 | 124.5 |
+| 17000 | 0.2747 | 103.5 | 398.0 | 0.7731 | 25.593 | 97.683 | 12.23 | 81.0 | 135.0 |
+| 18000 | 0.2909 | 99.0 | 378.0 | 0.7506 | 25.5316 | 97.918 | 12.259 | 81.5 | 122.0 |
+| 19000 | 0.3071 | 102.5 | 414.0 | 0.7708 | 25.6052 | 97.636 | 12.224 | 90.5 | 134.0 |
+| 20000 | 0.3232 | 89.5 | 376.0 | 0.6979 | 25.702 | 97.269 | 12.178 | 75.5 | 201.0 |
+| 21000 | 0.3394 | 82.5 | 328.0 | 0.6418 | 25.5751 | 97.751 | 12.238 | 69.5 | 119.5 |
+| 22000 | 0.3556 | 81.0 | 294.0 | 0.6043 | 25.5312 | 97.92 | 12.26 | 65.5 | 117.5 |
+| 23000 | 0.3717 | 76.5 | 276.0 | 0.5839 | 25.5256 | 97.941 | 12.262 | 63.5 | 117.5 |
+| 24000 | 0.3879 | 74.5 | 260.0 | 0.5720 | 25.5583 | 97.816 | 12.247 | 60.25 | 115.0 |
+| 25000 | 0.4040 | 72.5 | 252.0 | 0.5538 | 25.7608 | 97.047 | 12.15 | 57.0 | 112.5 |
+| 26000 | 0.4202 | 70.0 | 238.0 | 0.5516 | 25.6446 | 97.486 | 12.205 | 56.0 | 144.0 |
+| 27000 | 0.4364 | 70.5 | 236.0 | 0.5374 | 25.5599 | 97.809 | 12.246 | 56.25 | 166.0 |
+| 28000 | 0.4525 | 69.5 | 232.0 | 0.5255 | 25.5888 | 97.699 | 12.232 | 54.25 | 87.5 |
+| 29000 | 0.4687 | 70.5 | 245.0 | 0.5362 | 25.5487 | 97.852 | 12.251 | 55.0 | 123.5 |
+| 30000 | 0.4848 | 70.5 | 240.0 | 0.5324 | 25.8213 | 96.819 | 12.122 | 56.25 | 83.0 |
+| 31000 | 0.5010 | 74.5 | 236.0 | 0.5325 | 25.7081 | 97.246 | 12.175 | 56.25 | 105.5 |
+| 32000 | 0.5172 | 70.5 | 238.0 | 0.5273 | 25.5231 | 97.95 | 12.263 | 56.25 | 126.5 |
+| 33000 | 0.5333 | 71.0 | 256.0 | 0.5217 | 25.7437 | 97.111 | 12.158 | 54.0 | 109.0 |
+| 34000 | 0.5495 | 71.5 | 248.0 | 0.5136 | 25.6807 | 97.349 | 12.188 | 58.5 | 201.0 |
+| 35000 | 0.5657 | 70.5 | 230.0 | 0.5122 | 25.544 | 97.87 | 12.253 | 53.0 | 108.5 |
+| 36000 | 0.5818 | 70.0 | 234.0 | 0.5048 | 25.597 | 97.668 | 12.228 | 54.5 | 101.0 |
+| 37000 | 0.5980 | 71.5 | 220.0 | 0.4963 | 25.6094 | 97.62 | 12.222 | 56.0 | 96.0 |
+| 38000 | 0.6141 | 64.5 | 201.0 | 0.4774 | 25.5539 | 97.833 | 12.249 | 52.75 | 76.0 |
+| 39000 | 0.6303 | 66.5 | 204.0 | 0.4773 | 25.5882 | 97.701 | 12.232 | 49.5 | 88.0 |
+| 40000 | 0.6465 | 65.0 | 209.0 | 0.4727 | 25.5505 | 97.845 | 12.25 | 48.0 | 90.5 |
+| 41000 | 0.6626 | 63.25 | 209.0 | 0.4634 | 25.5287 | 97.929 | 12.261 | 47.75 | 144.0 |
+| 42000 | 0.6788 | 64.5 | 213.0 | 0.4608 | 25.5578 | 97.818 | 12.247 | 48.25 | 118.0 |
+| 43000 | 0.6949 | 62.25 | 191.0 | 0.4536 | 25.5708 | 97.768 | 12.241 | 48.75 | 73.0 |
+| 44000 | 0.7111 | 62.25 | 191.0 | 0.4533 | 25.5689 | 97.775 | 12.241 | 47.75 | 80.0 |
+| 45000 | 0.7273 | 59.75 | 180.0 | 0.4149 | 25.6074 | 97.628 | 12.223 | 42.75 | 61.25 |
+| 46000 | 0.7434 | 58.0 | 168.0 | 0.3961 | 25.5788 | 97.737 | 12.237 | 42.0 | 54.0 |
+| 47000 | 0.7596 | 56.75 | 159.0 | 0.3844 | 25.5179 | 97.97 | 12.266 | 41.25 | 48.75 |
+| 48000 | 0.7758 | 56.5 | 157.0 | 0.3789 | 25.5851 | 97.713 | 12.234 | 41.0 | 53.5 |
+| 49000 | 0.7919 | 56.5 | 154.0 | 0.3724 | 25.5249 | 97.944 | 12.263 | 40.0 | 43.75 |
+| 50000 | 0.8081 | 55.25 | 153.0 | 0.3689 | 25.5064 | 98.015 | 12.271 | 39.5 | 44.75 |
+| 51000 | 0.8242 | 54.75 | 151.0 | 0.3632 | 25.5055 | 98.018 | 12.272 | 38.5 | 47.5 |
+| 52000 | 0.8404 | 54.0 | 149.0 | 0.3584 | 25.5897 | 97.696 | 12.231 | 38.25 | 45.5 |
+| 53000 | 0.8566 | 54.0 | 148.0 | 0.3561 | 25.6987 | 97.281 | 12.18 | 38.25 | 43.25 |
+| 54000 | 0.8727 | 54.5 | 150.0 | 0.3546 | 25.5535 | 97.834 | 12.249 | 38.25 | 41.0 |
+| 55000 | 0.8889 | 54.0 | 147.0 | 0.3527 | 25.6836 | 97.338 | 12.187 | 38.25 | 40.75 |
+| 56000 | 0.9051 | 54.0 | 147.0 | 0.3517 | 25.5234 | 97.949 | 12.263 | 37.75 | 40.75 |
+| 57000 | 0.9212 | 54.0 | 148.0 | 0.3516 | 25.5369 | 97.897 | 12.257 | 37.75 | 40.5 |
+| 58000 | 0.9374 | 53.75 | 147.0 | 0.3508 | 25.6411 | 97.5 | 12.207 | 38.0 | 40.0 |
+| 59000 | 0.9535 | 53.75 | 147.0 | 0.3509 | 25.6546 | 97.448 | 12.201 | 37.75 | 40.25 |
+| 60000 | 0.9697 | 53.75 | 147.0 | 0.3510 | 25.6875 | 97.324 | 12.185 | 37.75 | 40.25 |
+| 61000 | 0.9859 | 53.75 | 147.0 | 0.3511 | 25.583 | 97.721 | 12.235 | 37.75 | 40.25 |
+| 61875 | 1.0 | 53.75 | 147.0 | 0.3509 | 25.5525 | 97.838 | 12.249 | 37.75 | 40.25 |
 
 ### Framework versions
 - Distily 0.2.0

logs/lr_scheduler_type=cosine, warmup_ratio=0.5/events.out.tfevents.1724162963.f383272e719b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88e2fe0f04ae24dd718df32fc4bdd2828320943b13eb023a66c9097c84eb90cb
+size 312