log

train.log

@@ -15,3 +15,9 @@
 [p1] starting training: max_steps=2000
 [transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 248044, 'bos_token_id': None, 'pad_token_id': 248044}.
 
  0%|          | 0/2000 [00:00<?, ?it/s][transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
+
  0%|          | 1/2000 [08:17<276:11:04, 497.38s/it]
  0%|          | 2/2000 [08:20<114:39:30, 206.59s/it]
  0%|          | 3/2000 [08:23<63:07:55, 113.81s/it] 
  0%|          | 4/2000 [08:27<39:03:41, 70.45s/it] 
  0%|          | 5/2000 [08:30<25:31:49, 46.07s/it]
  0%|          | 6/2000 [08:33<17:23:40, 31.40s/it]
  0%|          | 7/2000 [08:36<12:15:31, 22.14s/it]
  0%|          | 8/2000 [08:39<8:51:14, 16.00s/it] 
  0%|          | 9/2000 [08:42<6:34:21, 11.88s/it]
  0%|          | 10/2000 [08:45<5:01:27,  9.09s/it][p1][NaN-ABORT] grad_norm=nan at step 10
+
                                                   
+
  0%|          | 10/2000 [08:45<5:01:27,  9.09s/it]
                                                   
+
  0%|          | 10/2000 [08:45<5:01:27,  9.09s/it]
  0%|          | 10/2000 [08:45<29:02:02, 52.52s/it]
+[p1] training finished in 8.8m
+[p1] NaN-aborted. Reason: grad_norm=nan step=10. NOT pushing model.