LH-Tech-AI commited on
Commit
035115c
·
verified ·
1 Parent(s): 0c47749

Update train.py

Browse files

The official pretraining script. Use it like this
```bash
python3 train.py \
--dataset=fineweb-edu \
--n_layer=24 \
--n_head=16 \
--n_embd=1024 \
--block_size=1024 \
--batch_size=8 \
--gradient_accumulation_steps=16 \
--learning_rate=6e-4 \
--max_iters=300000 \
--eval_interval=1000 \
--eval_iters=100 \
--log_interval=5 \
--weight_decay=0.1 \
--warmup_iters=2000 \
--lr_decay_iters=300000 \
--min_lr=6e-5 \
--dtype=bfloat16 \
--compile=True \
--always_save_checkpoint=True \
--init_from=scratch \
--out_dir=/home/user/350m_fineweb
```

Files changed (1) hide show
  1. train.py +1 -1
train.py CHANGED
@@ -386,6 +386,6 @@ while True:
386
  # termination conditions
387
  if iter_num > max_iters:
388
  break
389
-
390
  if ddp:
391
  destroy_process_group()
 
386
  # termination conditions
387
  if iter_num > max_iters:
388
  break
389
+
390
  if ddp:
391
  destroy_process_group()