Update README.md

README.md

@@ -12,6 +12,14 @@ The runs were all performed training a smaller ViT (`vit_wee_patch16_reg1_gap_25
 
 So far I have results for `adamw`, `laprop`, and `mars`. You can find full results in sub-folders by optimizer names. In all of these runs, the experiments with 'c' prefix in the name have caution enabled.
 
+This is what the 'caution' addition looks like in an optimizer:
+```python
+    mask = (exp_avg * grad > 0).to(grad.dtype)
+    mask.div_(mask.mean().clamp_(min=1e-3))
+    exp_avg = exp_avg * mask
+```
+
+
 # LaProp
 
 |optim                       |best_epoch|train_loss        |eval_loss         |eval_top1        |eval_top5        |lr                    |