End of training

README.md

@@ -17,7 +17,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on the wikitext dataset.
 It achieves the following results on the evaluation set:
-- Loss: 7.1789
+- Loss: 6.0287
 
 ## Model description
 
@@ -36,37 +36,25 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 0.005
-- train_batch_size: 8
-- eval_batch_size: 8
+- learning_rate: 0.0001
+- train_batch_size: 128
+- eval_batch_size: 128
 - seed: 42
 - gradient_accumulation_steps: 8
-- total_train_batch_size: 64
+- total_train_batch_size: 1024
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 100
-- num_epochs: 10
+- num_epochs: 5
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss |
 |:-------------:|:-----:|:----:|:---------------:|
-| 7.8933        | 0.07  | 50   | 7.4310          |
-| 7.3323        | 0.14  | 100  | 7.3016          |
-| 7.2569        | 0.21  | 150  | 7.2390          |
-| 7.2126        | 0.27  | 200  | 7.2150          |
-| 7.1929        | 0.34  | 250  | 7.2001          |
-| 7.1759        | 0.41  | 300  | 7.1929          |
-| 7.2085        | 0.48  | 350  | 7.1945          |
-| 7.1944        | 0.55  | 400  | 7.1941          |
-| 7.1889        | 0.62  | 450  | 7.1847          |
-| 7.1626        | 0.69  | 500  | 7.1791          |
-| 7.1635        | 0.75  | 550  | 7.1777          |
-| 7.1868        | 0.82  | 600  | 7.1842          |
-| 7.1766        | 0.89  | 650  | 7.1725          |
-| 7.1767        | 0.96  | 700  | 7.1743          |
-| 7.1778        | 1.03  | 750  | 7.1728          |
-| 7.1497        | 1.1   | 800  | 7.1789          |
+| 9.0662        | 1.1   | 50   | 7.8866          |
+| 7.1297        | 2.19  | 100  | 6.6448          |
+| 6.4229        | 3.29  | 150  | 6.2367          |
+| 6.0864        | 4.38  | 200  | 6.0287          |
 
 
 ### Framework versions