AbstractPhil's picture
coco256: replication arm (the original model)
ede1230 verified
Raw History Blame Contribute Delete
6.02 kB
REPLICATION ARM: the original model, width 256, latent [32, 32, 32]; 2,239,527,242 trained parameters (denoiser only, as its trainer); batch 64; at most 2000 steps or 25 minutes; NVIDIA RTX PRO 6000 Blackwell Server Edition
step 1: loss 2.0323 cos 0.0008 | grad norm 632.9 before clipping | no gradient 244 / 573 | non-finite 0 | 1.343 s/step | spent 0:00:01
step 50: loss 1.9703 cos 0.1329 | grad norm 0.1 before clipping | no gradient 244 / 573 | non-finite 0 | 0.262 s/step | spent 0:00:14
step 100: loss 1.9129 cos 0.2087 | grad norm 0.1 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:00:27
step 150: loss 1.8979 cos 0.2336 | grad norm 0.1 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:00:41
step 200: loss 1.8673 cos 0.2517 | grad norm 0.1 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:00:54
step 250: loss 1.8700 cos 0.2582 | grad norm 0.1 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:01:08
step 300: loss 1.8433 cos 0.2718 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:01:21
step 350: loss 1.8293 cos 0.3005 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:01:34
step 400: loss 1.8111 cos 0.2979 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.267 s/step | spent 0:01:48
step 450: loss 1.8069 cos 0.2932 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:02:01
step 500: loss 1.8200 cos 0.2982 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:02:14
step 550: loss 1.8177 cos 0.2946 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:02:28
step 600: loss 1.8105 cos 0.3103 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:02:41
step 650: loss 1.7977 cos 0.3154 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:02:55
step 700: loss 1.7755 cos 0.3245 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:03:08
step 750: loss 1.7791 cos 0.3255 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:03:21
step 800: loss 1.7793 cos 0.3265 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.267 s/step | spent 0:03:35
step 850: loss 1.7806 cos 0.3346 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:03:48
step 900: loss 1.7680 cos 0.3283 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.267 s/step | spent 0:04:01
step 950: loss 1.7754 cos 0.3324 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.267 s/step | spent 0:04:15
step 1000: loss 1.7834 cos 0.3267 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.267 s/step | spent 0:04:28
step 1050: loss 1.7658 cos 0.3427 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:04:42
step 1100: loss 1.7863 cos 0.3363 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:04:55
step 1150: loss 1.7709 cos 0.3391 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:05:08
step 1200: loss 1.7498 cos 0.3484 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:05:22
step 1250: loss 1.7632 cos 0.3506 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:05:35
step 1300: loss 1.7489 cos 0.3534 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:05:49
step 1350: loss 1.7342 cos 0.3599 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:06:02
step 1400: loss 1.7241 cos 0.3641 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:06:15
step 1450: loss 1.7179 cos 0.3670 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:06:29
step 1500: loss 1.7658 cos 0.3354 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:06:42
step 1550: loss 1.7256 cos 0.3628 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:06:55
step 1600: loss 1.7523 cos 0.3481 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:07:09
step 1650: loss 1.7116 cos 0.3763 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.267 s/step | spent 0:07:22
step 1700: loss 1.7448 cos 0.3584 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.266 s/step | spent 0:07:36
step 1750: loss 1.7391 cos 0.3582 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.267 s/step | spent 0:07:49
step 1800: loss 1.7268 cos 0.3597 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:08:02
step 1850: loss 1.7170 cos 0.3681 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:08:16
step 1900: loss 1.7304 cos 0.3645 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:08:29
step 1950: loss 1.7449 cos 0.3496 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.268 s/step | spent 0:08:42
step 2000: loss 1.7245 cos 0.3695 | grad norm 0.2 before clipping | no gradient 244 / 573 | non-finite 0 | 0.266 s/step | spent 0:08:56
TARGET-LEAK READ: loss with the true clean latent in the head 1.7298, with another sample's 1.7667, difference 0.0369 (se 0.0023) over 512 held-out pairs