| In such a case you can use the |
| detect_overflow helper function to inject the detector where you want it, for example: |
| thon |
| from debug_utils import detect_overflow |
| class T5LayerFF(nn.Module): |
| [] |
| def forward(self, hidden_states): |
| forwarded_states = self.layer_norm(hidden_states) |
| detect_overflow(forwarded_states, "after layer_norm") |
| forwarded_states = self.DenseReluDense(forwarded_states) |
| detect_overflow(forwarded_states, "after DenseReluDense") |
| return hidden_states + self.dropout(forwarded_states) |
|
|
| You can see that we added 2 of these and now we track if inf or nan for forwarded_states was detected |
| somewhere in between. |
| Actually, the detector already reports these because each of the calls in the example above is a nn.Module, but |
| let's say if you had some local direct calculations this is how you'd do that. |
| Additionally, if you're instantiating the debugger in your own code, you can adjust the number of frames printed from |
| its default, e.g.: |
| thon |
| from transformers.debug_utils import DebugUnderflowOverflow |
| debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100) |
|
|
| Specific batch absolute min and max value tracing |
| The same debugging class can be used for per-batch tracing with the underflow/overflow detection feature turned off. |
| Let's say you want to watch the absolute min and max values for all the ingredients of each forward call of a given |
| batch, and only do that for batches 1 and 3. |
