Update README.md

README.md

@@ -9,4 +9,19 @@ language:
 - en
 ---
 
-3.2B parameter base model trained for ~64B tokens from the FineWeb dataset
+3.2B parameter base model trained for ~64B tokens from the FineWeb dataset
+
+uses gpt2 tokenizer from tiktoken
+
+[wandb training metrics](https://api.wandb.ai/links/teammapo-mapo-labs/zooq3iig)
+- note: increased batch size from 8 to 512 at step 2,160,000
+- Final checkpoint: step 2,187,000, val_loss: 2.7489
+- Trained on a 8xH100 80GB node using data parallel
+
+```
+"d_head": 128,
+"d_model": 8192,
+"n_heads": 64,
+"n_layers": 3,
+"n_vocab": 50257
+```