Auto upload 2026-08-06T21:51:09.116436 (part 3)

wandb/run-20260806_214555-al1jdpoz/files/output.log

@@ -1,6 +1,95 @@
 [transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
 /mnt/data/zainulabideen/zain-exp/notebooks/Activation/exp.py:464: UserWarning: std(): degrees of freedom is <= 0. Correction should be strictly less than the reduction factor (input numel divided by output numel). (Triggered internally at /pytorch/aten/src/ATen/native/ReduceOps.cpp:1831.)
   "std": tensor.std().item(),
-  3%|█                                       | 52/2000 [00:10<05:26,  5.97it/s]
+ 25%|█████████▊                             | 500/2000 [01:50<04:16,  5.85it/s][transformers] TinyLlamaForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
 {'loss': '7.777', 'grad_norm': '1.258', 'learning_rate': '0.0003', 'epoch': '0.001348', 'train/total_time_seconds': '3.406', 'train/time_per_step_avg': '0.1703', 'train/epoch_time_elapsed': '4.64', 'train/estimated_remaining_minutes': '5.619', 'train/global/act/norm': '2.193e+04', 'train/global/act/mean': '-0.005418', 'train/global/act/std': '0', 'train/global/act/max_abs': '8.336', 'train/global/act/frac_near_dtype_limit': '0', 'train/global/act/frac_near_user_limit': '0', 'train/global/grad/norm': '1.648', 'train/global/grad/mean': '4.181e-07', 'train/global/grad/std': '0.0005802', 'train/global/grad/max_abs': '0.09131', 'train/global/grad/frac_near_dtype_limit': '0', 'train/global/grad/frac_near_user_limit': '0', 'train/global/param/norm': '123', 'train/global/param/mean': '0.001476', 'train/global/param/std': '0.04326', 'train/global/param/max_abs': '1', 'train/global/param/frac_near_dtype_limit': '0', 'train/global/param/frac_near_user_limit': '0', 'train/layer_model_layers_7/act/norm': '3156', 'train/layer_model_layers_7/act/mean': '-0.006166', 'train/layer_model_layers_7/act/std': '0.4118', 'train/layer_model_layers_7/act/max_abs': '5.156', 'train/layer_model_layers_7/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/act/frac_near_user_limit': '0', 'train/layer_model_layers_7/grad/norm': '0.2329', 'train/layer_model_layers_7/grad/mean': '-4.304e-07', 'train/layer_model_layers_7/grad/std': '0.0005749', 'train/layer_model_layers_7/grad/max_abs': '0.01324', 'train/layer_model_layers_7/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/grad/frac_near_user_limit': '0', 'train/layer_model_layers_0/act/norm': '3144', 'train/layer_model_layers_0/act/mean': '-0.001883', 'train/layer_model_layers_0/act/std': '0.4109', 'train/layer_model_layers_0/act/max_abs': '4.531', 'train/layer_model_layers_0/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/act/frac_near_user_limit': '0', 'train/layer_model_layers_0/grad/norm': '0.6564', 'train/layer_model_layers_0/grad/mean': '-7.783e-08', 'train/layer_model_layers_0/grad/std': '0.00162', 'train/layer_model_layers_0/grad/max_abs': '0.03638', 'train/layer_model_layers_0/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/grad/frac_near_user_limit': '0', 'train/layer__model_layers_41/param/norm': '17.93', 'train/layer__model_layers_41/param/mean': '0.001577', 'train/layer__model_layers_41/param/std': '0.04424', 'train/layer__model_layers_41/param/max_abs': '1', 'train/layer__model_layers_41/param/frac_near_dtype_limit': '0', 'train/layer__model_layers_41/param/frac_near_user_limit': '0', 'train/layer_model_layers_42/act/norm': '3209', 'train/layer_model_layers_42/act/mean': '-0.003321', 'train/layer_model_layers_42/act/std': '0.4188', 'train/layer_model_layers_42/act/max_abs': '4.5', 'train/layer_model_layers_42/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/act/frac_near_user_limit': '0', 'train/layer_model_layers_42/grad/norm': '0.0991', 'train/layer_model_layers_42/grad/mean': '-4.48e-08', 'train/layer_model_layers_42/grad/std': '0.0002446', 'train/layer_model_layers_42/grad/max_abs': '0.004333', 'train/layer_model_layers_42/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/grad/frac_near_user_limit': '0', 'train/layer_model_layers_9/act/norm': '3157', 'train/layer_model_layers_9/act/mean': '-0.003039', 'train/layer_model_layers_9/act/std': '0.412', 'train/layer_model_layers_9/act/max_abs': '4.781', 'train/layer_model_layers_9/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/act/frac_near_user_limit': '0', 'train/layer_model_layers_9/grad/norm': '0.186', 'train/layer_model_layers_9/grad/mean': '-1.37e-06', 'train/layer_model_layers_9/grad/std': '0.000459', 'train/layer_model_layers_9/grad/max_abs': '0.008118', 'train/layer_model_layers_9/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/grad/frac_near_user_limit': '0', 'train/layer_model_layers_39/act/norm': '3208', 'train/layer_model_layers_39/act/mean': '-0.0051', 'train/layer_model_layers_39/act/std': '0.4187', 'train/layer_model_layers_3
 {'loss': '7.018', 'grad_norm': '1.172', 'learning_rate': '0.0003', 'epoch': '0.002696', 'train/total_time_seconds': '5.748', 'train/time_per_step_avg': '0.1437', 'train/epoch_time_elapsed': '8.088', 'train/estimated_remaining_minutes': '4.694'}
+{'loss': '6.487', 'grad_norm': '0.9297', 'learning_rate': '0.0003', 'epoch': '0.004044', 'train/total_time_seconds': '8.098', 'train/time_per_step_avg': '0.135', 'train/epoch_time_elapsed': '11.44', 'train/estimated_remaining_minutes': '4.364'}
+{'loss': '6.161', 'grad_norm': '0.6875', 'learning_rate': '0.0003', 'epoch': '0.005393', 'train/total_time_seconds': '10.42', 'train/time_per_step_avg': '0.1303', 'train/epoch_time_elapsed': '14.74', 'train/estimated_remaining_minutes': '4.169'}
+{'loss': '5.966', 'grad_norm': '0.6406', 'learning_rate': '0.0003', 'epoch': '0.006741', 'train/total_time_seconds': '12.76', 'train/time_per_step_avg': '0.1276', 'train/epoch_time_elapsed': '18.1', 'train/estimated_remaining_minutes': '4.039'}
+{'loss': '5.725', 'grad_norm': '0.9766', 'learning_rate': '0.0003', 'epoch': '0.008089', 'train/total_time_seconds': '15.08', 'train/time_per_step_avg': '0.1168', 'train/epoch_time_elapsed': '21.41', 'train/estimated_remaining_minutes': '3.939'}
+{'loss': '5.336', 'grad_norm': '1.047', 'learning_rate': '0.0003', 'epoch': '0.009437', 'train/total_time_seconds': '17.44', 'train/time_per_step_avg': '0.1169', 'train/epoch_time_elapsed': '24.75', 'train/estimated_remaining_minutes': '3.861'}
+{'loss': '4.942', 'grad_norm': '1.023', 'learning_rate': '0.0003', 'epoch': '0.01079', 'train/total_time_seconds': '19.77', 'train/time_per_step_avg': '0.1168', 'train/epoch_time_elapsed': '28.05', 'train/estimated_remaining_minutes': '3.79'}
+{'loss': '4.514', 'grad_norm': '1.156', 'learning_rate': '0.0003', 'epoch': '0.01213', 'train/total_time_seconds': '22.1', 'train/time_per_step_avg': '0.1168', 'train/epoch_time_elapsed': '31.35', 'train/estimated_remaining_minutes': '3.725'}
+{'loss': '4.053', 'grad_norm': '1.242', 'learning_rate': '0.0003', 'epoch': '0.01348', 'train/total_time_seconds': '24.43', 'train/time_per_step_avg': '0.1167', 'train/epoch_time_elapsed': '34.64', 'train/estimated_remaining_minutes': '3.664'}
+  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
+{'eval_loss': '3.849', 'eval_runtime': '12.18', 'eval_samples_per_second': '781.9', 'eval_steps_per_second': '12.23', 'epoch': '0.01348', 'train/total_time_seconds': '24.43', 'train/time_per_step_avg': '0.1167', 'train/epoch_time_elapsed': '46.83', 'train/estimated_remaining_minutes': '3.664'}
+{'loss': '3.665', 'grad_norm': '1.516', 'learning_rate': '0.0003', 'epoch': '0.01483', 'train/total_time_seconds': '27.04', 'train/time_per_step_avg': '0.1195', 'train/epoch_time_elapsed': '50.59', 'train/estimated_remaining_minutes': '3.646', 'train/global/act/norm': '3.706e+04', 'train/global/act/mean': '-0.0428', 'train/global/act/std': '0', 'train/global/act/max_abs': '7.344', 'train/global/act/frac_near_dtype_limit': '0', 'train/global/act/frac_near_user_limit': '0', 'train/global/grad/norm': '2.125', 'train/global/grad/mean': '-1.823e-07', 'train/global/grad/std': '0.0007479', 'train/global/grad/max_abs': '0.05908', 'train/global/grad/frac_near_dtype_limit': '0', 'train/global/grad/frac_near_user_limit': '0', 'train/global/param/norm': '126.2', 'train/global/param/mean': '0.00138', 'train/global/param/std': '0.04441', 'train/global/param/max_abs': '1', 'train/global/param/frac_near_dtype_limit': '0', 'train/global/param/frac_near_user_limit': '0', 'train/layer_model_layers_7/act/norm': '3444', 'train/layer_model_layers_7/act/mean': '0.01866', 'train/layer_model_layers_7/act/std': '0.4489', 'train/layer_model_layers_7/act/max_abs': '4.125', 'train/layer_model_layers_7/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/act/frac_near_user_limit': '0', 'train/layer_model_layers_7/grad/norm': '0.2414', 'train/layer_model_layers_7/grad/mean': '2.571e-06', 'train/layer_model_layers_7/grad/std': '0.0005958', 'train/layer_model_layers_7/grad/max_abs': '0.0155', 'train/layer_model_layers_7/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/grad/frac_near_user_limit': '0', 'train/layer_model_layers_0/act/norm': '3910', 'train/layer_model_layers_0/act/mean': '0.02449', 'train/layer_model_layers_0/act/std': '0.5099', 'train/layer_model_layers_0/act/max_abs': '4.531', 'train/layer_model_layers_0/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/act/frac_near_user_limit': '0', 'train/layer_model_layers_0/grad/norm': '0.7774', 'train/layer_model_layers_0/grad/mean': '-1.117e-06', 'train/layer_model_layers_0/grad/std': '0.001917', 'train/layer_model_layers_0/grad/max_abs': '0.0332', 'train/layer_model_layers_0/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/grad/frac_near_user_limit': '0', 'train/layer__model_layers_41/param/norm': '18.23', 'train/layer__model_layers_41/param/mean': '0.001606', 'train/layer__model_layers_41/param/std': '0.04497', 'train/layer__model_layers_41/param/max_abs': '1', 'train/layer__model_layers_41/param/frac_near_dtype_limit': '0', 'train/layer__model_layers_41/param/frac_near_user_limit': '0', 'train/layer_model_layers_42/act/norm': '5995', 'train/layer_model_layers_42/act/mean': '0.02654', 'train/layer_model_layers_42/act/std': '0.7819', 'train/layer_model_layers_42/act/max_abs': '5.219', 'train/layer_model_layers_42/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/act/frac_near_user_limit': '0', 'train/layer_model_layers_42/grad/norm': '0.3595', 'train/layer_model_layers_42/grad/mean': '4.694e-07', 'train/layer_model_layers_42/grad/std': '0.0008873', 'train/layer_model_layers_42/grad/max_abs': '0.01233', 'train/layer_model_layers_42/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/grad/frac_near_user_limit': '0', 'train/layer_model_layers_9/act/norm': '3415', 'train/layer_model_layers_9/act/mean': '0.01543', 'train/layer_model_layers_9/act/std': '0.4456', 'train/layer_model_layers_9/act/max_abs': '4.25', 'train/layer_model_layers_9/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/act/frac_near_user_limit': '0', 'train/layer_model_layers_9/grad/norm': '0.2054', 'train/layer_model_layers_9/grad/mean': '-3.612e-07', 'train/layer_model_layers_9/grad/std': '0.0005069', 'train/layer_model_layers_9/grad/max_abs': '0.01044', 'train/layer_model_layers_9/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/grad/frac_near_user_limit': '0', 'train/layer_model_layers_39/act/norm': '6339', 'train/layer_model_layers_39/act/mean': '0.03071', 'train/layer_model_layers_39/act/std': '0.8269', 'train/layer_model_layers_39/act/m
+{'loss': '3.326', 'grad_norm': '1.266', 'learning_rate': '0.0003', 'epoch': '0.01618', 'train/total_time_seconds': '29.37', 'train/time_per_step_avg': '0.1193', 'train/epoch_time_elapsed': '54', 'train/estimated_remaining_minutes': '3.589'}
+{'loss': '3.032', 'grad_norm': '1.867', 'learning_rate': '0.0003', 'epoch': '0.01753', 'train/total_time_seconds': '31.73', 'train/time_per_step_avg': '0.1195', 'train/epoch_time_elapsed': '57.37', 'train/estimated_remaining_minutes': '3.539'}
+{'loss': '2.754', 'grad_norm': '1.984', 'learning_rate': '0.0003', 'epoch': '0.01887', 'train/total_time_seconds': '34.05', 'train/time_per_step_avg': '0.1195', 'train/epoch_time_elapsed': '60.66', 'train/estimated_remaining_minutes': '3.486'}
+{'loss': '2.488', 'grad_norm': '2.172', 'learning_rate': '0.0003', 'epoch': '0.02022', 'train/total_time_seconds': '36.39', 'train/time_per_step_avg': '0.1196', 'train/epoch_time_elapsed': '63.98', 'train/estimated_remaining_minutes': '3.436'}
+{'loss': '2.249', 'grad_norm': '1.641', 'learning_rate': '0.0003', 'epoch': '0.02157', 'train/total_time_seconds': '38.71', 'train/time_per_step_avg': '0.1168', 'train/epoch_time_elapsed': '67.26', 'train/estimated_remaining_minutes': '3.387'}
+{'loss': '2.03', 'grad_norm': '1.594', 'learning_rate': '0.0003', 'epoch': '0.02292', 'train/total_time_seconds': '41.05', 'train/time_per_step_avg': '0.1168', 'train/epoch_time_elapsed': '70.57', 'train/estimated_remaining_minutes': '3.34'}
+{'loss': '1.831', 'grad_norm': '1.484', 'learning_rate': '0.0003', 'epoch': '0.02427', 'train/total_time_seconds': '43.39', 'train/time_per_step_avg': '0.1166', 'train/epoch_time_elapsed': '73.91', 'train/estimated_remaining_minutes': '3.295'}
+{'loss': '1.649', 'grad_norm': '1.781', 'learning_rate': '0.0003', 'epoch': '0.02562', 'train/total_time_seconds': '45.72', 'train/time_per_step_avg': '0.1166', 'train/epoch_time_elapsed': '77.2', 'train/estimated_remaining_minutes': '3.248'}
+{'loss': '1.476', 'grad_norm': '1.82', 'learning_rate': '0.0003', 'epoch': '0.02696', 'train/total_time_seconds': '48.08', 'train/time_per_step_avg': '0.1169', 'train/epoch_time_elapsed': '80.54', 'train/estimated_remaining_minutes': '3.205'}
+{'eval_loss': '1.399', 'eval_runtime': '12.24', 'eval_samples_per_second': '778.1', 'eval_steps_per_second': '12.17', 'epoch': '0.02696', 'train/total_time_seconds': '48.08', 'train/time_per_step_avg': '0.1169', 'train/epoch_time_elapsed': '92.79', 'train/estimated_remaining_minutes': '3.205'}
+{'loss': '1.318', 'grad_norm': '1.562', 'learning_rate': '0.0003', 'epoch': '0.02831', 'train/total_time_seconds': '50.67', 'train/time_per_step_avg': '0.1196', 'train/epoch_time_elapsed': '96.47', 'train/estimated_remaining_minutes': '3.177', 'train/global/act/norm': '3.616e+04', 'train/global/act/mean': '-0.04917', 'train/global/act/std': '0', 'train/global/act/max_abs': '8.75', 'train/global/act/frac_near_dtype_limit': '0', 'train/global/act/frac_near_user_limit': '0', 'train/global/grad/norm': '1.333', 'train/global/grad/mean': '-4.247e-07', 'train/global/grad/std': '0.0004691', 'train/global/grad/max_abs': '0.05396', 'train/global/grad/frac_near_dtype_limit': '0', 'train/global/grad/frac_near_user_limit': '0', 'train/global/param/norm': '129', 'train/global/param/mean': '0.00137', 'train/global/param/std': '0.04541', 'train/global/param/max_abs': '1', 'train/global/param/frac_near_dtype_limit': '0', 'train/global/param/frac_near_user_limit': '0', 'train/layer_model_layers_7/act/norm': '3344', 'train/layer_model_layers_7/act/mean': '0.01441', 'train/layer_model_layers_7/act/std': '0.4361', 'train/layer_model_layers_7/act/max_abs': '4.375', 'train/layer_model_layers_7/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/act/frac_near_user_limit': '0', 'train/layer_model_layers_7/grad/norm': '0.174', 'train/layer_model_layers_7/grad/mean': '-8.502e-07', 'train/layer_model_layers_7/grad/std': '0.0004298', 'train/layer_model_layers_7/grad/max_abs': '0.01508', 'train/layer_model_layers_7/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/grad/frac_near_user_limit': '0', 'train/layer_model_layers_0/act/norm': '3705', 'train/layer_model_layers_0/act/mean': '0.01718', 'train/layer_model_layers_0/act/std': '0.4832', 'train/layer_model_layers_0/act/max_abs': '4.062', 'train/layer_model_layers_0/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/act/frac_near_user_limit': '0', 'train/layer_model_layers_0/grad/norm': '0.4122', 'train/layer_model_layers_0/grad/mean': '-4.22e-06', 'train/layer_model_layers_0/grad/std': '0.001016', 'train/layer_model_layers_0/grad/max_abs': '0.01721', 'train/layer_model_layers_0/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/grad/frac_near_user_limit': '0', 'train/layer__model_layers_41/param/norm': '18.54', 'train/layer__model_layers_41/param/mean': '0.001614', 'train/layer__model_layers_41/param/std': '0.04575', 'train/layer__model_layers_41/param/max_abs': '1', 'train/layer__model_layers_41/param/frac_near_dtype_limit': '0', 'train/layer__model_layers_41/param/frac_near_user_limit': '0', 'train/layer_model_layers_42/act/norm': '5423', 'train/layer_model_layers_42/act/mean': '0.03866', 'train/layer_model_layers_42/act/std': '0.7067', 'train/layer_model_layers_42/act/max_abs': '4.281', 'train/layer_model_layers_42/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/act/frac_near_user_limit': '0', 'train/layer_model_layers_42/grad/norm': '0.1341', 'train/layer_model_layers_42/grad/mean': '-6.338e-07', 'train/layer_model_layers_42/grad/std': '0.0003308', 'train/layer_model_layers_42/grad/max_abs': '0.004822', 'train/layer_model_layers_42/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/grad/frac_near_user_limit': '0', 'train/layer_model_layers_9/act/norm': '3327', 'train/layer_model_layers_9/act/mean': '0.008129', 'train/layer_model_layers_9/act/std': '0.4337', 'train/layer_model_layers_9/act/max_abs': '4.594', 'train/layer_model_layers_9/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/act/frac_near_user_limit': '0', 'train/layer_model_layers_9/grad/norm': '0.1618', 'train/layer_model_layers_9/grad/mean': '1.63e-06', 'train/layer_model_layers_9/grad/std': '0.0003994', 'train/layer_model_layers_9/grad/max_abs': '0.01428', 'train/layer_model_layers_9/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/grad/frac_near_user_limit': '0', 'train/layer_model_layers_39/act/norm': '5906', 'train/layer_model_layers_39/act/mean': '0.04663', 'train/layer_model_layers_39/act/std': '0.7686', 'train/layer_model_layers_39/act/
+{'loss': '1.194', 'grad_norm': '1.062', 'learning_rate': '0.0003', 'epoch': '0.02966', 'train/total_time_seconds': '53', 'train/time_per_step_avg': '0.1195', 'train/epoch_time_elapsed': '99.87', 'train/estimated_remaining_minutes': '3.132'}
+{'loss': '1.068', 'grad_norm': '1.461', 'learning_rate': '0.0003', 'epoch': '0.03101', 'train/total_time_seconds': '55.34', 'train/time_per_step_avg': '0.1195', 'train/epoch_time_elapsed': '103.2', 'train/estimated_remaining_minutes': '3.088'}
+{'loss': '0.9639', 'grad_norm': '1.188', 'learning_rate': '0.0003', 'epoch': '0.03236', 'train/total_time_seconds': '57.67', 'train/time_per_step_avg': '0.1195', 'train/epoch_time_elapsed': '106.5', 'train/estimated_remaining_minutes': '3.044'}
+{'loss': '0.8594', 'grad_norm': '0.6992', 'learning_rate': '0.0003', 'epoch': '0.0337', 'train/total_time_seconds': '60.01', 'train/time_per_step_avg': '0.1193', 'train/epoch_time_elapsed': '109.8', 'train/estimated_remaining_minutes': '3.001'}
+  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
+  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
+Writing model shards: 100%|██████████████████████| 1/1 [00:00<00:00, 26.46it/s]
+ 30%|███████████▋                           | 600/2000 [02:19<03:50,  6.06it/s]/mnt/data/zainulabideen/zain-exp/notebooks/Activation/exp.py:464: UserWarning: std(): degrees of freedom is <= 0. Correction should be strictly less than the reduction factor (input numel divided by output numel). (Triggered internally at /pytorch/aten/src/ATen/native/ReduceOps.cpp:1831.)
+{'loss': '0.7737', 'grad_norm': '0.4141', 'learning_rate': '0.0003', 'epoch': '0.03505', 'train/total_time_seconds': '62.37', 'train/time_per_step_avg': '0.117', 'train/epoch_time_elapsed': '113.4', 'train/estimated_remaining_minutes': '2.959'}
+{'loss': '0.6989', 'grad_norm': '0.4199', 'learning_rate': '0.0003', 'epoch': '0.0364', 'train/total_time_seconds': '64.7', 'train/time_per_step_avg': '0.1171', 'train/epoch_time_elapsed': '116.7', 'train/estimated_remaining_minutes': '2.916'}
+{'loss': '0.6477', 'grad_norm': '0.4023', 'learning_rate': '0.0003', 'epoch': '0.03775', 'train/total_time_seconds': '67.06', 'train/time_per_step_avg': '0.1173', 'train/epoch_time_elapsed': '120', 'train/estimated_remaining_minutes': '2.874'}
+{'loss': '0.6122', 'grad_norm': '1.742', 'learning_rate': '0.0003', 'epoch': '0.0391', 'train/total_time_seconds': '69.39', 'train/time_per_step_avg': '0.1172', 'train/epoch_time_elapsed': '123.3', 'train/estimated_remaining_minutes': '2.832'}
+{'loss': '0.5977', 'grad_norm': '1.305', 'learning_rate': '0.0003', 'epoch': '0.04044', 'train/total_time_seconds': '71.72', 'train/time_per_step_avg': '0.1171', 'train/epoch_time_elapsed': '126.6', 'train/estimated_remaining_minutes': '2.789'}
+  "std": tensor.std().item(),                                                  
+{'eval_loss': '0.5825', 'eval_runtime': '12.22', 'eval_samples_per_second': '779.7', 'eval_steps_per_second': '12.2', 'epoch': '0.04044', 'train/total_time_seconds': '71.72', 'train/time_per_step_avg': '0.1171', 'train/epoch_time_elapsed': '138.9', 'train/estimated_remaining_minutes': '2.789'}
+ 50%|███████████████████                   | 1000/2000 [03:51<02:42,  6.17it/s][transformers] TinyLlamaForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
+{'loss': '0.5631', 'grad_norm': '0.4023', 'learning_rate': '0.0003', 'epoch': '0.04179', 'train/total_time_seconds': '74.33', 'train/time_per_step_avg': '0.1197', 'train/epoch_time_elapsed': '142.6', 'train/estimated_remaining_minutes': '2.758', 'train/global/act/norm': '3.588e+04', 'train/global/act/mean': '-0.04831', 'train/global/act/std': '0', 'train/global/act/max_abs': '10.44', 'train/global/act/frac_near_dtype_limit': '0', 'train/global/act/frac_near_user_limit': '0', 'train/global/grad/norm': '0.9172', 'train/global/grad/mean': '-3.194e-07', 'train/global/grad/std': '0.0003229', 'train/global/grad/max_abs': '0.02808', 'train/global/grad/frac_near_dtype_limit': '0', 'train/global/grad/frac_near_user_limit': '0', 'train/global/param/norm': '132', 'train/global/param/mean': '0.001367', 'train/global/param/std': '0.04644', 'train/global/param/max_abs': '1', 'train/global/param/frac_near_dtype_limit': '0', 'train/global/param/frac_near_user_limit': '0', 'train/layer_model_layers_7/act/norm': '3328', 'train/layer_model_layers_7/act/mean': '0.01525', 'train/layer_model_layers_7/act/std': '0.4341', 'train/layer_model_layers_7/act/max_abs': '4.531', 'train/layer_model_layers_7/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/act/frac_near_user_limit': '0', 'train/layer_model_layers_7/grad/norm': '0.1289', 'train/layer_model_layers_7/grad/mean': '-8.371e-07', 'train/layer_model_layers_7/grad/std': '0.0003182', 'train/layer_model_layers_7/grad/max_abs': '0.009888', 'train/layer_model_layers_7/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/grad/frac_near_user_limit': '0', 'train/layer_model_layers_0/act/norm': '3626', 'train/layer_model_layers_0/act/mean': '0.01291', 'train/layer_model_layers_0/act/std': '0.4731', 'train/layer_model_layers_0/act/max_abs': '3.688', 'train/layer_model_layers_0/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/act/frac_near_user_limit': '0', 'train/layer_model_layers_0/grad/norm': '0.2576', 'train/layer_model_layers_0/grad/mean': '-4.276e-06', 'train/layer_model_layers_0/grad/std': '0.0006359', 'train/layer_model_layers_0/grad/max_abs': '0.01007', 'train/layer_model_layers_0/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/grad/frac_near_user_limit': '0', 'train/layer__model_layers_41/param/norm': '18.68', 'train/layer__model_layers_41/param/mean': '0.001606', 'train/layer__model_layers_41/param/std': '0.04609', 'train/layer__model_layers_41/param/max_abs': '1', 'train/layer__model_layers_41/param/frac_near_dtype_limit': '0', 'train/layer__model_layers_41/param/frac_near_user_limit': '0', 'train/layer_model_layers_42/act/norm': '5080', 'train/layer_model_layers_42/act/mean': '0.04069', 'train/layer_model_layers_42/act/std': '0.6615', 'train/layer_model_layers_42/act/max_abs': '4.406', 'train/layer_model_layers_42/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/act/frac_near_user_limit': '0', 'train/layer_model_layers_42/grad/norm': '0.1045', 'train/layer_model_layers_42/grad/mean': '-3.143e-07', 'train/layer_model_layers_42/grad/std': '0.0002577', 'train/layer_model_layers_42/grad/max_abs': '0.003998', 'train/layer_model_layers_42/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/grad/frac_near_user_limit': '0', 'train/layer_model_layers_9/act/norm': '3306', 'train/layer_model_layers_9/act/mean': '0.008007', 'train/layer_model_layers_9/act/std': '0.431', 'train/layer_model_layers_9/act/max_abs': '4.625', 'train/layer_model_layers_9/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/act/frac_near_user_limit': '0', 'train/layer_model_layers_9/grad/norm': '0.1078', 'train/layer_model_layers_9/grad/mean': '7.34e-07', 'train/layer_model_layers_9/grad/std': '0.000266', 'train/layer_model_layers_9/grad/max_abs': '0.007935', 'train/layer_model_layers_9/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/grad/frac_near_user_limit': '0', 'train/layer_model_layers_39/act/norm': '5731', 'train/layer_model_layers_39/act/mean': '0.05006', 'train/layer_model_layers_39/act/std': '0.7458', 'train/layer_model_layers
+{'loss': '0.5362', 'grad_norm': '0.3359', 'learning_rate': '0.0003', 'epoch': '0.04314', 'train/total_time_seconds': '76.66', 'train/time_per_step_avg': '0.1195', 'train/epoch_time_elapsed': '146', 'train/estimated_remaining_minutes': '2.715'}
+{'loss': '0.5065', 'grad_norm': '0.3125', 'learning_rate': '0.0003', 'epoch': '0.04449', 'train/total_time_seconds': '79', 'train/time_per_step_avg': '0.1194', 'train/epoch_time_elapsed': '149.3', 'train/estimated_remaining_minutes': '2.673'}
+{'loss': '0.4883', 'grad_norm': '0.3223', 'learning_rate': '0.0003', 'epoch': '0.04584', 'train/total_time_seconds': '81.66', 'train/time_per_step_avg': '0.1227', 'train/epoch_time_elapsed': '153', 'train/estimated_remaining_minutes': '2.642'}
+{'loss': '0.4688', 'grad_norm': '0.3125', 'learning_rate': '0.0003', 'epoch': '0.04719', 'train/total_time_seconds': '83.98', 'train/time_per_step_avg': '0.1226', 'train/epoch_time_elapsed': '156.2', 'train/estimated_remaining_minutes': '2.599'}
+{'loss': '0.455', 'grad_norm': '0.334', 'learning_rate': '0.0003', 'epoch': '0.04853', 'train/total_time_seconds': '86.3', 'train/time_per_step_avg': '0.1197', 'train/epoch_time_elapsed': '159.5', 'train/estimated_remaining_minutes': '2.557'}
+{'loss': '0.4373', 'grad_norm': '0.3516', 'learning_rate': '0.0003', 'epoch': '0.04988', 'train/total_time_seconds': '88.63', 'train/time_per_step_avg': '0.1198', 'train/epoch_time_elapsed': '162.8', 'train/estimated_remaining_minutes': '2.515'}
+{'loss': '0.4296', 'grad_norm': '0.3652', 'learning_rate': '0.0003', 'epoch': '0.05123', 'train/total_time_seconds': '90.96', 'train/time_per_step_avg': '0.1196', 'train/epoch_time_elapsed': '166.1', 'train/estimated_remaining_minutes': '2.473'}
+{'loss': '0.453', 'grad_norm': '1.93', 'learning_rate': '0.0003', 'epoch': '0.05258', 'train/total_time_seconds': '93.29', 'train/time_per_step_avg': '0.1163', 'train/epoch_time_elapsed': '169.4', 'train/estimated_remaining_minutes': '2.432'}
+{'loss': '0.4267', 'grad_norm': '0.5391', 'learning_rate': '0.0003', 'epoch': '0.05393', 'train/total_time_seconds': '95.61', 'train/time_per_step_avg': '0.1163', 'train/epoch_time_elapsed': '172.7', 'train/estimated_remaining_minutes': '2.39'}
+  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
+{'eval_loss': '0.4185', 'eval_runtime': '12.23', 'eval_samples_per_second': '779', 'eval_steps_per_second': '12.18', 'epoch': '0.05393', 'train/total_time_seconds': '95.61', 'train/time_per_step_avg': '0.1163', 'train/epoch_time_elapsed': '184.9', 'train/estimated_remaining_minutes': '2.39'}
+{'loss': '0.4098', 'grad_norm': '0.3477', 'learning_rate': '0.0003', 'epoch': '0.05527', 'train/total_time_seconds': '98.2', 'train/time_per_step_avg': '0.1189', 'train/epoch_time_elapsed': '188.6', 'train/estimated_remaining_minutes': '2.355', 'train/global/act/norm': '3.558e+04', 'train/global/act/mean': '-0.04731', 'train/global/act/std': '0', 'train/global/act/max_abs': '10.75', 'train/global/act/frac_near_dtype_limit': '0', 'train/global/act/frac_near_user_limit': '0', 'train/global/grad/norm': '0.5428', 'train/global/grad/mean': '-2.132e-07', 'train/global/grad/std': '0.0001911', 'train/global/grad/max_abs': '0.02563', 'train/global/grad/frac_near_dtype_limit': '0', 'train/global/grad/frac_near_user_limit': '0', 'train/global/param/norm': '133.2', 'train/global/param/mean': '0.001369', 'train/global/param/std': '0.04687', 'train/global/param/max_abs': '1', 'train/global/param/frac_near_dtype_limit': '0', 'train/global/param/frac_near_user_limit': '0', 'train/layer_model_layers_7/act/norm': '3326', 'train/layer_model_layers_7/act/mean': '0.01555', 'train/layer_model_layers_7/act/std': '0.4335', 'train/layer_model_layers_7/act/max_abs': '4.625', 'train/layer_model_layers_7/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/act/frac_near_user_limit': '0', 'train/layer_model_layers_7/grad/norm': '0.05889', 'train/layer_model_layers_7/grad/mean': '-5.156e-07', 'train/layer_model_layers_7/grad/std': '0.0001453', 'train/layer_model_layers_7/grad/max_abs': '0.003647', 'train/layer_model_layers_7/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/grad/frac_near_user_limit': '0', 'train/layer_model_layers_0/act/norm': '3624', 'train/layer_model_layers_0/act/mean': '0.01294', 'train/layer_model_layers_0/act/std': '0.4728', 'train/layer_model_layers_0/act/max_abs': '3.375', 'train/layer_model_layers_0/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/act/frac_near_user_limit': '0', 'train/layer_model_layers_0/grad/norm': '0.1504', 'train/layer_model_layers_0/grad/mean': '-1.352e-06', 'train/layer_model_layers_0/grad/std': '0.0003713', 'train/layer_model_layers_0/grad/max_abs': '0.009094', 'train/layer_model_layers_0/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/grad/frac_near_user_limit': '0', 'train/layer__model_layers_41/param/norm': '18.74', 'train/layer__model_layers_41/param/mean': '0.0016', 'train/layer__model_layers_41/param/std': '0.04624', 'train/layer__model_layers_41/param/max_abs': '1', 'train/layer__model_layers_41/param/frac_near_dtype_limit': '0', 'train/layer__model_layers_41/param/frac_near_user_limit': '0', 'train/layer_model_layers_42/act/norm': '4945', 'train/layer_model_layers_42/act/mean': '0.04031', 'train/layer_model_layers_42/act/std': '0.6438', 'train/layer_model_layers_42/act/max_abs': '4.469', 'train/layer_model_layers_42/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/act/frac_near_user_limit': '0', 'train/layer_model_layers_42/grad/norm': '0.04575', 'train/layer_model_layers_42/grad/mean': '-2.636e-07', 'train/layer_model_layers_42/grad/std': '0.000113', 'train/layer_model_layers_42/grad/max_abs': '0.001686', 'train/layer_model_layers_42/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/grad/frac_near_user_limit': '0', 'train/layer_model_layers_9/act/norm': '3304', 'train/layer_model_layers_9/act/mean': '0.008383', 'train/layer_model_layers_9/act/std': '0.4309', 'train/layer_model_layers_9/act/max_abs': '4.844', 'train/layer_model_layers_9/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/act/frac_near_user_limit': '0', 'train/layer_model_layers_9/grad/norm': '0.05856', 'train/layer_model_layers_9/grad/mean': '3.848e-07', 'train/layer_model_layers_9/grad/std': '0.0001445', 'train/layer_model_layers_9/grad/max_abs': '0.004242', 'train/layer_model_layers_9/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/grad/frac_near_user_limit': '0', 'train/layer_model_layers_39/act/norm': '5664', 'train/layer_model_layers_39/act/mean': '0.05114', 'train/layer_model_layers_39/act/std': '0.7369', 'train/layer_model_l
+{'loss': '0.3982', 'grad_norm': '0.291', 'learning_rate': '0.0003', 'epoch': '0.05662', 'train/total_time_seconds': '100.5', 'train/time_per_step_avg': '0.119', 'train/epoch_time_elapsed': '192', 'train/estimated_remaining_minutes': '2.314'}
+{'loss': '0.3863', 'grad_norm': '0.2773', 'learning_rate': '0.0003', 'epoch': '0.05797', 'train/total_time_seconds': '103', 'train/time_per_step_avg': '0.1199', 'train/epoch_time_elapsed': '195.4', 'train/estimated_remaining_minutes': '2.274'}
+{'loss': '0.3765', 'grad_norm': '0.2773', 'learning_rate': '0.0003', 'epoch': '0.05932', 'train/total_time_seconds': '105.4', 'train/time_per_step_avg': '0.1214', 'train/epoch_time_elapsed': '198.9', 'train/estimated_remaining_minutes': '2.236'}
+{'loss': '0.3731', 'grad_norm': '0.2754', 'learning_rate': '0.0003', 'epoch': '0.06067', 'train/total_time_seconds': '107.7', 'train/time_per_step_avg': '0.1213', 'train/epoch_time_elapsed': '202.2', 'train/estimated_remaining_minutes': '2.195'}
+{'loss': '0.367', 'grad_norm': '0.2832', 'learning_rate': '0.0003', 'epoch': '0.06202', 'train/total_time_seconds': '110.1', 'train/time_per_step_avg': '0.1193', 'train/epoch_time_elapsed': '205.6', 'train/estimated_remaining_minutes': '2.155'}
+{'loss': '0.3599', 'grad_norm': '0.2793', 'learning_rate': '0.0003', 'epoch': '0.06336', 'train/total_time_seconds': '112.4', 'train/time_per_step_avg': '0.119', 'train/epoch_time_elapsed': '208.9', 'train/estimated_remaining_minutes': '2.113'}
+{'loss': '0.3525', 'grad_norm': '0.2812', 'learning_rate': '0.0003', 'epoch': '0.06471', 'train/total_time_seconds': '114.8', 'train/time_per_step_avg': '0.1182', 'train/epoch_time_elapsed': '212.2', 'train/estimated_remaining_minutes': '2.072'}
+{'loss': '0.3484', 'grad_norm': '0.2715', 'learning_rate': '0.0003', 'epoch': '0.06606', 'train/total_time_seconds': '117.1', 'train/time_per_step_avg': '0.1165', 'train/epoch_time_elapsed': '215.5', 'train/estimated_remaining_minutes': '2.031'}
+{'loss': '0.3458', 'grad_norm': '0.3047', 'learning_rate': '0.0003', 'epoch': '0.06741', 'train/total_time_seconds': '119.4', 'train/time_per_step_avg': '0.1164', 'train/epoch_time_elapsed': '218.7', 'train/estimated_remaining_minutes': '1.99'}
+{'eval_loss': '0.3437', 'eval_runtime': '12.21', 'eval_samples_per_second': '780', 'eval_steps_per_second': '12.2', 'epoch': '0.06741', 'train/total_time_seconds': '119.4', 'train/time_per_step_avg': '0.1164', 'train/epoch_time_elapsed': '230.9', 'train/estimated_remaining_minutes': '1.99'}
+  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
+  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
+Writing model shards: 100%|██████████████████████| 1/1 [00:00<00:00, 32.75it/s]
+/mnt/data/zainulabideen/zain-exp/notebooks/Activation/exp.py:464: UserWarning: std(): degrees of freedom is <= 0. Correction should be strictly less than the reduction factor (input numel divided by output numel). (Triggered internally at /pytorch/aten/src/ATen/native/ReduceOps.cpp:1831.)
+  "std": tensor.std().item(),
+ 70%|██████████████████████████▌           | 1400/2000 [05:10<01:38,  6.06it/s]
+{'loss': '0.3385', 'grad_norm': '0.2832', 'learning_rate': '0.0003', 'epoch': '0.06876', 'train/total_time_seconds': '122', 'train/time_per_step_avg': '0.1186', 'train/epoch_time_elapsed': '234.8', 'train/estimated_remaining_minutes': '1.953', 'train/global/act/norm': '3.609e+04', 'train/global/act/mean': '-0.04334', 'train/global/act/std': '0', 'train/global/act/max_abs': '10.94', 'train/global/act/frac_near_dtype_limit': '0', 'train/global/act/frac_near_user_limit': '0', 'train/global/grad/norm': '0.2968', 'train/global/grad/mean': '-2.121e-07', 'train/global/grad/std': '0.0001045', 'train/global/grad/max_abs': '0.02356', 'train/global/grad/frac_near_dtype_limit': '0', 'train/global/grad/frac_near_user_limit': '0', 'train/global/param/norm': '133.9', 'train/global/param/mean': '0.001371', 'train/global/param/std': '0.04711', 'train/global/param/max_abs': '1', 'train/global/param/frac_near_dtype_limit': '0', 'train/global/param/frac_near_user_limit': '0', 'train/layer_model_layers_7/act/norm': '3331', 'train/layer_model_layers_7/act/mean': '0.01644', 'train/layer_model_layers_7/act/std': '0.4343', 'train/layer_model_layers_7/act/max_abs': '4.719', 'train/layer_model_layers_7/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/act/frac_near_user_limit': '0', 'train/layer_model_layers_7/grad/norm': '0.02292', 'train/layer_model_layers_7/grad/mean': '1.5e-07', 'train/layer_model_layers_7/grad/std': '5.656e-05', 'train/layer_model_layers_7/grad/max_abs': '0.001968', 'train/layer_model_layers_7/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/grad/frac_near_user_limit': '0', 'train/layer_model_layers_0/act/norm': '3621', 'train/layer_model_layers_0/act/mean': '0.01225', 'train/layer_model_layers_0/act/std': '0.4722', 'train/layer_model_layers_0/act/max_abs': '3.422', 'train/layer_model_layers_0/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/act/frac_near_user_limit': '0', 'train/layer_model_layers_0/grad/norm': '0.04145', 'train/layer_model_layers_0/grad/mean': '-3.202e-07', 'train/layer_model_layers_0/grad/std': '0.0001023', 'train/layer_model_layers_0/grad/max_abs': '0.001839', 'train/layer_model_layers_0/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/grad/frac_near_user_limit': '0', 'train/layer__model_layers_41/param/norm': '18.77', 'train/layer__model_layers_41/param/mean': '0.0016', 'train/layer__model_layers_41/param/std': '0.04632', 'train/layer__model_layers_41/param/max_abs': '1', 'train/layer__model_layers_41/param/frac_near_dtype_limit': '0', 'train/layer__model_layers_41/param/frac_near_user_limit': '0', 'train/layer_model_layers_42/act/norm': '4901', 'train/layer_model_layers_42/act/mean': '0.04131', 'train/layer_model_layers_42/act/std': '0.6378', 'train/layer_model_layers_42/act/max_abs': '4.5', 'train/layer_model_layers_42/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/act/frac_near_user_limit': '0', 'train/layer_model_layers_42/grad/norm': '0.01446', 'train/layer_model_layers_42/grad/mean': '-2.823e-08', 'train/layer_model_layers_42/grad/std': '3.571e-05', 'train/layer_model_layers_42/grad/max_abs': '0.0003929', 'train/layer_model_layers_42/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/grad/frac_near_user_limit': '0', 'train/layer_model_layers_9/act/norm': '3312', 'train/layer_model_layers_9/act/mean': '0.008287', 'train/layer_model_layers_9/act/std': '0.4317', 'train/layer_model_layers_9/act/max_abs': '4.75', 'train/layer_model_layers_9/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/act/frac_near_user_limit': '0', 'train/layer_model_layers_9/grad/norm': '0.02125', 'train/layer_model_layers_9/grad/mean': '1.382e-07', 'train/layer_model_layers_9/grad/std': '5.248e-05', 'train/layer_model_layers_9/grad/max_abs': '0.001633', 'train/layer_model_layers_9/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/grad/frac_near_user_limit': '0', 'train/layer_model_layers_39/act/norm': '5792', 'train/layer_model_layers_39/act/mean': '0.05425', 'train/layer_model_layers_39/act/std': '0.7538', 'train/layer_model_layer
+{'loss': '0.3322', 'grad_norm': '0.2637', 'learning_rate': '0.0003', 'epoch': '0.0701', 'train/total_time_seconds': '124.3', 'train/time_per_step_avg': '0.1187', 'train/epoch_time_elapsed': '238.2', 'train/estimated_remaining_minutes': '1.912'}
+{'loss': '0.3301', 'grad_norm': '0.2715', 'learning_rate': '0.0003', 'epoch': '0.07145', 'train/total_time_seconds': '126.6', 'train/time_per_step_avg': '0.1187', 'train/epoch_time_elapsed': '241.5', 'train/estimated_remaining_minutes': '1.872'}
+{'loss': '0.324', 'grad_norm': '0.2832', 'learning_rate': '0.0003', 'epoch': '0.0728', 'train/total_time_seconds': '129', 'train/time_per_step_avg': '0.1189', 'train/epoch_time_elapsed': '244.7', 'train/estimated_remaining_minutes': '1.831'}
+{'loss': '0.3225', 'grad_norm': '0.2871', 'learning_rate': '0.0003', 'epoch': '0.07415', 'train/total_time_seconds': '131.3', 'train/time_per_step_avg': '0.1191', 'train/epoch_time_elapsed': '248', 'train/estimated_remaining_minutes': '1.79'}
+{'loss': '0.3179', 'grad_norm': '0.3105', 'learning_rate': '0.0003', 'epoch': '0.0755', 'train/total_time_seconds': '133.6', 'train/time_per_step_avg': '0.1162', 'train/epoch_time_elapsed': '251.3', 'train/estimated_remaining_minutes': '1.75'}
+{'loss': '0.3123', 'grad_norm': '0.3125', 'learning_rate': '0.0003', 'epoch': '0.07685', 'train/total_time_seconds': '135.9', 'train/time_per_step_avg': '0.116', 'train/epoch_time_elapsed': '254.6', 'train/estimated_remaining_minutes': '1.709'}
+{'loss': '0.3115', 'grad_norm': '0.3398', 'learning_rate': '0.0003', 'epoch': '0.07819', 'train/total_time_seconds': '138.2', 'train/time_per_step_avg': '0.1159', 'train/epoch_time_elapsed': '257.8', 'train/estimated_remaining_minutes': '1.668'}
+{'loss': '0.3398', 'grad_norm': '1.773', 'learning_rate': '0.0003', 'epoch': '0.07954', 'train/total_time_seconds': '140.6', 'train/time_per_step_avg': '0.116', 'train/epoch_time_elapsed': '261.2', 'train/estimated_remaining_minutes': '1.628'}
+{'loss': '0.3267', 'grad_norm': '0.4434', 'learning_rate': '0.0003', 'epoch': '0.08089', 'train/total_time_seconds': '142.9', 'train/time_per_step_avg': '0.116', 'train/epoch_time_elapsed': '264.5', 'train/estimated_remaining_minutes': '1.588'}
+ 42%|█████████████████▎                       | 63/149 [00:05<00:06, 12.35it/s]
+{'eval_loss': '0.3165', 'eval_runtime': '12.23', 'eval_samples_per_second': '779.2', 'eval_steps_per_second': '12.19', 'epoch': '0.08089', 'train/total_time_seconds': '142.9', 'train/time_per_step_avg': '0.116', 'train/epoch_time_elapsed': '276.7', 'train/estimated_remaining_minutes': '1.588'}
+{'loss': '0.3096', 'grad_norm': '0.2773', 'learning_rate': '0.0003', 'epoch': '0.08224', 'train/total_time_seconds': '145.5', 'train/time_per_step_avg': '0.1187', 'train/epoch_time_elapsed': '280.4', 'train/estimated_remaining_minutes': '1.55', 'train/global/act/norm': '3.523e+04', 'train/global/act/mean': '-0.04307', 'train/global/act/std': '0', 'train/global/act/max_abs': '11', 'train/global/act/frac_near_dtype_limit': '0', 'train/global/act/frac_near_user_limit': '0', 'train/global/grad/norm': '0.4376', 'train/global/grad/mean': '-2.371e-07', 'train/global/grad/std': '0.0001539', 'train/global/grad/max_abs': '0.02356', 'train/global/grad/frac_near_dtype_limit': '0', 'train/global/grad/frac_near_user_limit': '0', 'train/global/param/norm': '134.4', 'train/global/param/mean': '0.001372', 'train/global/param/std': '0.04728', 'train/global/param/max_abs': '1', 'train/global/param/frac_near_dtype_limit': '0', 'train/global/param/frac_near_user_limit': '0', 'train/layer_model_layers_7/act/norm': '3327', 'train/layer_model_layers_7/act/mean': '0.01492', 'train/layer_model_layers_7/act/std': '0.4338', 'train/layer_model_layers_7/act/max_abs': '4.375', 'train/layer_model_layers_7/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/act/frac_near_user_limit': '0', 'train/layer_model_layers_7/grad/norm': '0.05124', 'train/layer_model_layers_7/grad/mean': '-4.278e-07', 'train/layer_model_layers_7/grad/std': '0.0001265', 'train/layer_model_layers_7/grad/max_abs': '0.004425', 'train/layer_model_layers_7/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_7/grad/frac_near_user_limit': '0', 'train/layer_model_layers_0/act/norm': '3624', 'train/layer_model_layers_0/act/mean': '0.01191', 'train/layer_model_layers_0/act/std': '0.4728', 'train/layer_model_layers_0/act/max_abs': '3.391', 'train/layer_model_layers_0/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/act/frac_near_user_limit': '0', 'train/layer_model_layers_0/grad/norm': '0.09987', 'train/layer_model_layers_0/grad/mean': '-3.174e-07', 'train/layer_model_layers_0/grad/std': '0.0002471', 'train/layer_model_layers_0/grad/max_abs': '0.004303', 'train/layer_model_layers_0/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_0/grad/frac_near_user_limit': '0', 'train/layer__model_layers_41/param/norm': '18.83', 'train/layer__model_layers_41/param/mean': '0.001599', 'train/layer__model_layers_41/param/std': '0.04643', 'train/layer__model_layers_41/param/max_abs': '1', 'train/layer__model_layers_41/param/frac_near_dtype_limit': '0', 'train/layer__model_layers_41/param/frac_near_user_limit': '0', 'train/layer_model_layers_42/act/norm': '4811', 'train/layer_model_layers_42/act/mean': '0.0405', 'train/layer_model_layers_42/act/std': '0.6263', 'train/layer_model_layers_42/act/max_abs': '4.281', 'train/layer_model_layers_42/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/act/frac_near_user_limit': '0', 'train/layer_model_layers_42/grad/norm': '0.03404', 'train/layer_model_layers_42/grad/mean': '3.376e-08', 'train/layer_model_layers_42/grad/std': '8.393e-05', 'train/layer_model_layers_42/grad/max_abs': '0.001472', 'train/layer_model_layers_42/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_42/grad/frac_near_user_limit': '0', 'train/layer_model_layers_9/act/norm': '3308', 'train/layer_model_layers_9/act/mean': '0.008058', 'train/layer_model_layers_9/act/std': '0.4313', 'train/layer_model_layers_9/act/max_abs': '4.656', 'train/layer_model_layers_9/act/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/act/frac_near_user_limit': '0', 'train/layer_model_layers_9/grad/norm': '0.04549', 'train/layer_model_layers_9/grad/mean': '-2.148e-07', 'train/layer_model_layers_9/grad/std': '0.0001123', 'train/layer_model_layers_9/grad/max_abs': '0.003876', 'train/layer_model_layers_9/grad/frac_near_dtype_limit': '0', 'train/layer_model_layers_9/grad/frac_near_user_limit': '0', 'train/layer_model_layers_39/act/norm': '5605', 'train/layer_model_layers_39/act/mean': '0.05267', 'train/layer_model_layers_39/act/std': '0.7302', 'train/layer_model_l
+{'loss': '0.305', 'grad_norm': '0.2539', 'learning_rate': '0.0003', 'epoch': '0.08359', 'train/total_time_seconds': '147.8', 'train/time_per_step_avg': '0.1189', 'train/epoch_time_elapsed': '283.7', 'train/estimated_remaining_minutes': '1.51'}
+{'loss': '0.3022', 'grad_norm': '0.25', 'learning_rate': '0.0003', 'epoch': '0.08493', 'train/total_time_seconds': '150.1', 'train/time_per_step_avg': '0.1188', 'train/epoch_time_elapsed': '287', 'train/estimated_remaining_minutes': '1.469'}
+{'loss': '0.2971', 'grad_norm': '0.2598', 'learning_rate': '0.0003', 'epoch': '0.08628', 'train/total_time_seconds': '152.5', 'train/time_per_step_avg': '0.1189', 'train/epoch_time_elapsed': '290.3', 'train/estimated_remaining_minutes': '1.429'}
+{'loss': '0.2963', 'grad_norm': '0.2559', 'learning_rate': '0.0003', 'epoch': '0.08763', 'train/total_time_seconds': '154.8', 'train/time_per_step_avg': '0.119', 'train/epoch_time_elapsed': '293.6', 'train/estimated_remaining_minutes': '1.389'}
+{'loss': '0.293', 'grad_norm': '0.2461', 'learning_rate': '0.0003', 'epoch': '0.08898', 'train/total_time_seconds': '157.1', 'train/time_per_step_avg': '0.1163', 'train/epoch_time_elapsed': '296.9', 'train/estimated_remaining_minutes': '1.349'}
+{'loss': '0.2908', 'grad_norm': '0.2461', 'learning_rate': '0.0003', 'epoch': '0.09033', 'train/total_time_seconds': '159.4', 'train/time_per_step_avg': '0.1162', 'train/epoch_time_elapsed': '300.2', 'train/estimated_remaining_minutes': '1.309'}
+{'loss': '0.2876', 'grad_norm': '0.25', 'learning_rate': '0.0003', 'epoch': '0.09168', 'train/total_time_seconds': '161.7', 'train/time_per_step_avg': '0.1162', 'train/epoch_time_elapsed': '303.5', 'train/estimated_remaining_minutes': '1.268'}
+{'loss': '0.287', 'grad_norm': '0.25', 'learning_rate': '0.0003', 'epoch': '0.09302', 'train/total_time_seconds': '164.1', 'train/time_per_step_avg': '0.1162', 'train/epoch_time_elapsed': '306.8', 'train/estimated_remaining_minutes': '1.229'}
+{'loss': '0.284', 'grad_norm': '0.2539', 'learning_rate': '0.0003', 'epoch': '0.09437', 'train/total_time_seconds': '166.4', 'train/time_per_step_avg': '0.116', 'train/epoch_time_elapsed': '310', 'train/estimated_remaining_minutes': '1.188'}

wandb/run-20260806_214555-al1jdpoz/logs/debug-internal.log

@@ -21,3 +21,45 @@
 {"time":"2026-08-06T21:46:00.914572518Z","level":"INFO","msg":"flowcontrol: unblocked","totalOffloaded":1}
 {"time":"2026-08-06T21:46:00.915317306Z","level":"INFO","msg":"flowcontrol: backed up, offloading to disk","recordNumber":7591}
 {"time":"2026-08-06T21:46:00.915467961Z","level":"INFO","msg":"flowcontrol: unblocked","totalOffloaded":41}
+{"time":"2026-08-06T21:46:11.332458678Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":0,"history_lines":4,"events_offset":0,"events_lines":2,"console_offset":1,"console_lines":7,"uploaded_len":2}
+{"time":"2026-08-06T21:46:11.935198121Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:46:26.323909563Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":4,"history_lines":4,"events_offset":2,"events_lines":2,"console_offset":3,"console_lines":1}
+{"time":"2026-08-06T21:46:27.038060991Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:46:41.321447829Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":8,"history_lines":2,"events_offset":4,"events_lines":2,"console_offset":3,"console_lines":1}
+{"time":"2026-08-06T21:46:42.010289914Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:46:56.331197338Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":10,"history_lines":4,"events_offset":6,"events_lines":2,"console_offset":3,"console_lines":1}
+{"time":"2026-08-06T21:46:57.128117767Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:47:11.322643037Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":14,"history_lines":5,"events_offset":8,"events_lines":2,"console_offset":3,"console_lines":1}
+{"time":"2026-08-06T21:47:12.006917359Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:47:26.321890114Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":19,"history_lines":2,"events_offset":10,"events_lines":2,"console_offset":3,"console_lines":1}
+{"time":"2026-08-06T21:47:26.942653062Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:47:41.330635829Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":21,"history_lines":4,"events_offset":12,"events_lines":2,"console_offset":3,"console_lines":1}
+{"time":"2026-08-06T21:47:42.016663843Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:47:56.324947348Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":25,"history_lines":5,"events_offset":14,"events_lines":2,"console_offset":3,"console_lines":1}
+{"time":"2026-08-06T21:47:57.025765428Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:48:11.3207974Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":30,"history_lines":2,"events_offset":16,"events_lines":2,"console_offset":8,"console_lines":34}
+{"time":"2026-08-06T21:48:11.974950188Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:48:26.33095143Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":32,"history_lines":4,"events_offset":18,"events_lines":2,"console_offset":35,"console_lines":1}
+{"time":"2026-08-06T21:48:27.171925366Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:48:41.320713553Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":36,"history_lines":4,"events_offset":20,"events_lines":2,"console_offset":41,"console_lines":10}
+{"time":"2026-08-06T21:48:41.972140855Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:48:56.32434026Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":40,"history_lines":3,"events_offset":22,"events_lines":2,"console_offset":43,"console_lines":1}
+{"time":"2026-08-06T21:48:57.00754703Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:49:11.329558219Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":43,"history_lines":3,"events_offset":24,"events_lines":2,"console_offset":43,"console_lines":1}
+{"time":"2026-08-06T21:49:12.221996052Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:49:26.321276219Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":46,"history_lines":5,"events_offset":26,"events_lines":2,"console_offset":43,"console_lines":1}
+{"time":"2026-08-06T21:49:26.947048225Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:49:41.321446831Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":51,"history_lines":3,"events_offset":28,"events_lines":2,"console_offset":43,"console_lines":1}
+{"time":"2026-08-06T21:49:42.207808877Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:49:56.328135621Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":54,"history_lines":3,"events_offset":30,"events_lines":2,"console_offset":43,"console_lines":1}
+{"time":"2026-08-06T21:49:57.059087387Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:50:11.321093144Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":57,"history_lines":5,"events_offset":32,"events_lines":2,"console_offset":51,"console_lines":29}
+{"time":"2026-08-06T21:50:12.115113025Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:50:26.324861928Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":62,"history_lines":3,"events_offset":34,"events_lines":2,"console_offset":72,"console_lines":1}
+{"time":"2026-08-06T21:50:26.957856664Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:50:41.32998699Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":65,"history_lines":3,"events_offset":36,"events_lines":2,"console_offset":72,"console_lines":1}
+{"time":"2026-08-06T21:50:42.16856512Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:50:56.325439661Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":68,"history_lines":4,"events_offset":38,"events_lines":2,"console_offset":72,"console_lines":1}
+{"time":"2026-08-06T21:50:56.99841207Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
+{"time":"2026-08-06T21:51:11.322882894Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":72,"history_lines":4,"events_offset":40,"events_lines":2,"console_offset":72,"console_lines":1}
+{"time":"2026-08-06T21:51:11.989138277Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}

wandb/run-20260806_214555-al1jdpoz/run-al1jdpoz.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:53a75cd66ba2cc26defae1b14a3a8183670c5aa8abb235bc00bf5cec01db7e50
-size 3244032
+oid sha256:ee49619546f22a4c1bf97eb2813046ce3e187b042fe685b52257ef146a4c62d9
+size 17170432