| [2026-05-05 16:42:55] starting temp=0.98 lambda grid on GPUs: 0,1,2,3,4,5,6,7 |
| [2026-05-05 16:42:55] out_root=/home/sunhc/diffusion/sad/eval/mauve_temp098_lambda_grid_8gpu |
| [2026-05-05 16:42:55] num_samples=1024 sample_batch_size=256 mauve_batch_size=16 |
| [2026-05-05 16:42:55] pps_list=1,2,4 lambda_list=0.2,0.4,0.6,0.8 |
| [2026-05-05 16:42:55] models=top1,top2,top3,h2 |
| [2026-05-05 16:42:55] launched top1_pps1_lam0.2_temp0.98 on gpu0 pid=4127605 |
| [2026-05-05 16:42:55] launched top2_pps1_lam0.2_temp0.98 on gpu1 pid=4127611 |
| [2026-05-05 16:42:55] launched top3_pps1_lam0.2_temp0.98 on gpu2 pid=4127617 |
| [2026-05-05 16:42:55] launched h2_pps1_lam0.2_temp0.98 on gpu3 pid=4127623 |
| [2026-05-05 16:42:55] launched top1_pps1_lam0.4_temp0.98 on gpu4 pid=4127629 |
| [2026-05-05 16:42:55] launched top2_pps1_lam0.4_temp0.98 on gpu5 pid=4127635 |
| [2026-05-05 16:42:55] launched top3_pps1_lam0.4_temp0.98 on gpu6 pid=4127641 |
| [2026-05-05 16:42:55] launched h2_pps1_lam0.4_temp0.98 on gpu7 pid=4127647 |
| [2026-05-05 16:49:05] nonzero exit (-1) for top2_pps1_lam0.2_temp0.98 on gpu1 pid=4127611 but output exists |
| [2026-05-05 16:49:05] launched top1_pps1_lam0.6_temp0.98 on gpu1 pid=4137853 |
| [2026-05-05 16:49:20] nonzero exit (-1) for top1_pps1_lam0.2_temp0.98 on gpu0 pid=4127605 but output exists |
| [2026-05-05 16:49:20] launched top2_pps1_lam0.6_temp0.98 on gpu0 pid=4138318 |
| [2026-05-05 16:49:20] finished top2_pps1_lam0.4_temp0.98 on gpu5 pid=4127635 |
| [2026-05-05 16:49:20] launched top3_pps1_lam0.6_temp0.98 on gpu5 pid=4138327 |
| [2026-05-05 16:49:35] nonzero exit (-1) for top1_pps1_lam0.4_temp0.98 on gpu4 pid=4127629 but output exists |
| [2026-05-05 16:49:35] launched h2_pps1_lam0.6_temp0.98 on gpu4 pid=4139184 |
| [2026-05-05 16:49:55] nonzero exit (-1) for top3_pps1_lam0.2_temp0.98 on gpu2 pid=4127617 but output exists |
| [2026-05-05 16:49:55] launched top1_pps1_lam0.8_temp0.98 on gpu2 pid=4139866 |
| [2026-05-05 16:50:10] nonzero exit (-1) for top3_pps1_lam0.4_temp0.98 on gpu6 pid=4127641 but output exists |
| [2026-05-05 16:50:10] launched top2_pps1_lam0.8_temp0.98 on gpu6 pid=4140268 |
| [2026-05-05 16:50:30] nonzero exit (-1) for h2_pps1_lam0.2_temp0.98 on gpu3 pid=4127623 but output exists |
| [2026-05-05 16:50:30] launched top3_pps1_lam0.8_temp0.98 on gpu3 pid=4140691 |
| [2026-05-05 16:51:55] nonzero exit (-1) for h2_pps1_lam0.4_temp0.98 on gpu7 pid=4127647 but output exists |
| [2026-05-05 16:51:55] launched h2_pps1_lam0.8_temp0.98 on gpu7 pid=4141511 |
| [2026-05-05 16:55:15] nonzero exit (-1) for top2_pps1_lam0.6_temp0.98 on gpu0 pid=4138318 but output exists |
| [2026-05-05 16:55:15] launched top1_pps2_lam0.2_temp0.98 on gpu0 pid=4146512 |
| [2026-05-05 16:55:35] nonzero exit (-1) for top1_pps1_lam0.6_temp0.98 on gpu1 pid=4137853 but output exists |
| [2026-05-05 16:55:35] launched top2_pps2_lam0.2_temp0.98 on gpu1 pid=4146866 |
| [2026-05-05 16:56:10] nonzero exit (-1) for top2_pps1_lam0.8_temp0.98 on gpu6 pid=4140268 but output exists |
| [2026-05-05 16:56:10] launched top3_pps2_lam0.2_temp0.98 on gpu6 pid=4148194 |
| [2026-05-05 16:56:45] nonzero exit (-1) for top3_pps1_lam0.6_temp0.98 on gpu5 pid=4138327 but output exists |
| [2026-05-05 16:56:45] launched h2_pps2_lam0.2_temp0.98 on gpu5 pid=4149393 |
| [2026-05-05 16:56:50] nonzero exit (-1) for top1_pps1_lam0.8_temp0.98 on gpu2 pid=4139866 but output exists |
| [2026-05-05 16:56:50] launched top1_pps2_lam0.4_temp0.98 on gpu2 pid=4149545 |
| [2026-05-05 16:58:01] nonzero exit (-1) for top3_pps1_lam0.8_temp0.98 on gpu3 pid=4140691 but output exists |
| [2026-05-05 16:58:01] launched top2_pps2_lam0.4_temp0.98 on gpu3 pid=4151103 |
| [2026-05-05 16:59:01] nonzero exit (-1) for h2_pps1_lam0.6_temp0.98 on gpu4 pid=4139184 but output exists |
| [2026-05-05 16:59:01] launched top3_pps2_lam0.4_temp0.98 on gpu4 pid=4153465 |
| [2026-05-05 16:59:51] nonzero exit (-1) for top1_pps2_lam0.2_temp0.98 on gpu0 pid=4146512 but output exists |
| [2026-05-05 16:59:51] launched h2_pps2_lam0.4_temp0.98 on gpu0 pid=4155515 |
| [2026-05-05 17:00:11] nonzero exit (-1) for top2_pps2_lam0.2_temp0.98 on gpu1 pid=4146866 but output exists |
| [2026-05-05 17:00:11] launched top1_pps2_lam0.6_temp0.98 on gpu1 pid=4155869 |
| [2026-05-05 17:01:21] nonzero exit (-1) for top3_pps2_lam0.2_temp0.98 on gpu6 pid=4148194 but output exists |
| [2026-05-05 17:01:21] launched top2_pps2_lam0.6_temp0.98 on gpu6 pid=4157477 |
| [2026-05-05 17:01:36] nonzero exit (-1) for h2_pps1_lam0.8_temp0.98 on gpu7 pid=4141511 but output exists |
| [2026-05-05 17:01:36] launched top3_pps2_lam0.6_temp0.98 on gpu7 pid=4157824 |
| [2026-05-05 17:01:41] nonzero exit (-1) for top1_pps2_lam0.4_temp0.98 on gpu2 pid=4149545 but output exists |
| [2026-05-05 17:01:41] launched h2_pps2_lam0.6_temp0.98 on gpu2 pid=4157972 |
| [2026-05-05 17:02:26] nonzero exit (-1) for h2_pps2_lam0.2_temp0.98 on gpu5 pid=4149393 but output exists |
| [2026-05-05 17:02:26] launched top1_pps2_lam0.8_temp0.98 on gpu5 pid=4159490 |
| [2026-05-05 17:02:41] nonzero exit (-1) for top2_pps2_lam0.4_temp0.98 on gpu3 pid=4151103 but output exists |
| [2026-05-05 17:02:41] launched top2_pps2_lam0.8_temp0.98 on gpu3 pid=4159837 |
| [2026-05-05 17:04:26] nonzero exit (-1) for top3_pps2_lam0.4_temp0.98 on gpu4 pid=4153465 but output exists |
| [2026-05-05 17:04:26] launched top3_pps2_lam0.8_temp0.98 on gpu4 pid=4163088 |
| [2026-05-05 17:05:06] nonzero exit (-1) for top1_pps2_lam0.6_temp0.98 on gpu1 pid=4155869 but output exists |
| [2026-05-05 17:05:06] launched h2_pps2_lam0.8_temp0.98 on gpu1 pid=4164833 |
| [2026-05-05 17:06:01] nonzero exit (-1) for h2_pps2_lam0.4_temp0.98 on gpu0 pid=4155515 but output exists |
| [2026-05-05 17:06:01] launched top1_pps4_lam0.2_temp0.98 on gpu0 pid=4165861 |
| [2026-05-05 17:06:06] nonzero exit (-1) for top2_pps2_lam0.6_temp0.98 on gpu6 pid=4157477 but output exists |
| [2026-05-05 17:06:06] launched top2_pps4_lam0.2_temp0.98 on gpu6 pid=4166041 |
| [2026-05-05 17:07:01] nonzero exit (-1) for top3_pps2_lam0.6_temp0.98 on gpu7 pid=4157824 but output exists |
| [2026-05-05 17:07:01] launched top3_pps4_lam0.2_temp0.98 on gpu7 pid=4168423 |
| [2026-05-05 17:07:46] nonzero exit (-1) for top2_pps2_lam0.8_temp0.98 on gpu3 pid=4159837 but output exists |
| [2026-05-05 17:07:46] launched h2_pps4_lam0.2_temp0.98 on gpu3 pid=4169168 |
| [2026-05-05 17:07:51] nonzero exit (-1) for top1_pps2_lam0.8_temp0.98 on gpu5 pid=4159490 but output exists |
| [2026-05-05 17:07:51] launched top1_pps4_lam0.4_temp0.98 on gpu5 pid=4169315 |
| [2026-05-05 17:08:16] nonzero exit (-1) for h2_pps2_lam0.6_temp0.98 on gpu2 pid=4157972 but output exists |
| [2026-05-05 17:08:16] launched top2_pps4_lam0.4_temp0.98 on gpu2 pid=4170840 |
| [2026-05-05 17:10:31] nonzero exit (-1) for top3_pps2_lam0.8_temp0.98 on gpu4 pid=4163088 but output exists |
| [2026-05-05 17:10:31] launched top3_pps4_lam0.4_temp0.98 on gpu4 pid=4174602 |
| [2026-05-05 17:10:36] nonzero exit (-1) for top1_pps4_lam0.2_temp0.98 on gpu0 pid=4165861 but output exists |
| [2026-05-05 17:10:36] launched h2_pps4_lam0.4_temp0.98 on gpu0 pid=4174750 |
| [2026-05-05 17:10:46] nonzero exit (-1) for top2_pps4_lam0.2_temp0.98 on gpu6 pid=4166041 but output exists |
| [2026-05-05 17:10:46] launched top1_pps4_lam0.6_temp0.98 on gpu6 pid=4175415 |
| [2026-05-05 17:11:21] nonzero exit (-1) for top3_pps4_lam0.2_temp0.98 on gpu7 pid=4168423 but output exists |
| [2026-05-05 17:11:21] launched top2_pps4_lam0.6_temp0.98 on gpu7 pid=4175900 |
| [2026-05-05 17:11:41] nonzero exit (-1) for h2_pps2_lam0.8_temp0.98 on gpu1 pid=4164833 but output exists |
| [2026-05-05 17:11:41] launched top3_pps4_lam0.6_temp0.98 on gpu1 pid=4176376 |
| [2026-05-05 17:12:06] nonzero exit (-1) for top1_pps4_lam0.4_temp0.98 on gpu5 pid=4169315 but output exists |
| [2026-05-05 17:12:06] launched h2_pps4_lam0.6_temp0.98 on gpu5 pid=4178130 |
| [2026-05-05 17:12:41] nonzero exit (-1) for top2_pps4_lam0.4_temp0.98 on gpu2 pid=4170840 but output exists |
| [2026-05-05 17:12:41] launched top1_pps4_lam0.8_temp0.98 on gpu2 pid=4179636 |
| [2026-05-05 17:12:51] nonzero exit (-1) for h2_pps4_lam0.2_temp0.98 on gpu3 pid=4169168 but output exists |
| [2026-05-05 17:12:51] launched top2_pps4_lam0.8_temp0.98 on gpu3 pid=4179980 |
| [2026-05-05 17:15:21] nonzero exit (-1) for top3_pps4_lam0.4_temp0.98 on gpu4 pid=4174602 but output exists |
| [2026-05-05 17:15:21] launched top3_pps4_lam0.8_temp0.98 on gpu4 pid=4184281 |
| [2026-05-05 17:15:36] nonzero exit (-1) for top1_pps4_lam0.6_temp0.98 on gpu6 pid=4175415 but output exists |
| [2026-05-05 17:15:36] launched h2_pps4_lam0.8_temp0.98 on gpu6 pid=4184754 |
| [2026-05-05 17:16:01] finished h2_pps4_lam0.4_temp0.98 on gpu0 pid=4174750 |
| [2026-05-05 17:16:17] finished top3_pps4_lam0.6_temp0.98 on gpu1 pid=4176376 |
| [2026-05-05 17:17:25] finished top1_pps4_lam0.8_temp0.98 on gpu2 pid=4179636 |
| [2026-05-05 17:17:30] finished top2_pps4_lam0.8_temp0.98 on gpu3 pid=4179980 |
| [2026-05-05 17:19:58] finished top3_pps4_lam0.8_temp0.98 on gpu4 pid=4184281 |
| [2026-05-05 17:19:58] finished h2_pps4_lam0.6_temp0.98 on gpu5 pid=4178130 |
| [2026-05-05 17:20:35] finished h2_pps4_lam0.8_temp0.98 on gpu6 pid=4184754 |
| [2026-05-05 17:20:35] finished top2_pps4_lam0.6_temp0.98 on gpu7 pid=4175900 |
| [2026-05-05 17:20:35] all temp=0.98 lambda grid jobs completed |
|
|
