| In the next frame we have Dropout which renormalizes |
| the weights, after it zeroed some of the elements, which pushes the absolute max value to more than 64K, and we get an |
| overflow (inf). |
| As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 |
| numbers. |
| Let's match the report to the code from models/t5/modeling_t5.py: |
| thon |
| class T5DenseGatedGeluDense(nn.Module): |
| def init(self, config): |
| super().init() |
| self.wi_0 = nn.Linear(config.d_model, config.d_ff, bias=False) |
| self.wi_1 = nn.Linear(config.d_model, config.d_ff, bias=False) |
| self.wo = nn.Linear(config.d_ff, config.d_model, bias=False) |
| self.dropout = nn.Dropout(config.dropout_rate) |
| self.gelu_act = ACT2FN["gelu_new"] |
| def forward(self, hidden_states): |
| hidden_gelu = self.gelu_act(self.wi_0(hidden_states)) |
| hidden_linear = self.wi_1(hidden_states) |
| hidden_states = hidden_gelu * hidden_linear |
| hidden_states = self.dropout(hidden_states) |
| hidden_states = self.wo(hidden_states) |
| return hidden_states |
|
|
| Now it's easy to see the dropout call, and all the previous calls as well. |
| Since the detection is happening in a forward hook, these reports are printed immediately after each forward |
| returns. |
| Going back to the full report, to act on it and to fix the problem, we need to go a few frames up where the numbers |
| started to go up and most likely switch to the fp32 mode here, so that the numbers don't overflow when multiplied |
| or summed up. |
