F16 NaN in 10 steps workaround
#2
by foss22 - opened
grad underflow → NaN in 10 steps. structurally unusable
Continued pretraining in f16 was possible after first 100-200 steps in f32 (Pascal P102 in my case @10 TFLOPs). Continued in f16 (Turing ~22 TFLOPS)
NaNs occur only when gradients vary much, no lang undersanding in a random weigts initialized LM.
I moved on from F16 so there's no point trying it, BF16 is better.
But I had lots of issues with the LR on my very specific hardware + torch stack on Linux.