[05-20 10:18:10][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 10:18:10][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 10:18:10][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 10:18:10][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 10:18:10][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 10:18:10][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:18:10][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:18:10][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 10:18:11][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 10:18:11][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 10:18:12][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 10:18:12][flashy.solver][INFO] - No EMA on the model.
[05-20 10:18:12][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 10:18:12][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 10:18:12][flashy.solver][INFO] - Restoring weights and history.
[05-20 10:18:12][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 10:18:13][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 10:18:13][flashy.solver][INFO] - Train | Epoch 1 | 1/10 | 1.94 it/sec | lr 1.00E-04 | grad_norm 4.982E-01 | ce 8.061 | ppl 3167.533
[05-20 10:18:14][flashy.solver][INFO] - Train | Epoch 1 | 2/10 | 2.36 it/sec | lr 1.00E-04 | grad_norm 4.411E-01 | ce 8.097 | ppl 3285.336
[05-20 10:18:14][flashy.solver][INFO] - Train | Epoch 1 | 3/10 | 2.75 it/sec | lr 1.00E-04 | grad_norm 4.603E-01 | ce 8.086 | ppl 3250.249
[05-20 10:18:14][flashy.solver][INFO] - Train | Epoch 1 | 4/10 | 3.04 it/sec | lr 1.00E-04 | grad_norm 4.830E-01 | ce 8.048 | ppl 3128.555
[05-20 10:18:14][flashy.solver][INFO] - Train | Epoch 1 | 5/10 | 3.25 it/sec | lr 1.00E-04 | grad_norm 3.960E-01 | ce 8.073 | ppl 3207.059
[05-20 10:18:14][flashy.solver][INFO] - Train | Epoch 1 | 6/10 | 3.38 it/sec | lr 1.00E-04 | grad_norm 4.168E-01 | ce 8.073 | ppl 3207.142
[05-20 10:18:15][flashy.solver][INFO] - Train | Epoch 1 | 7/10 | 3.52 it/sec | lr 1.00E-04 | grad_norm 4.186E-01 | ce 8.071 | ppl 3200.087
[05-20 10:18:15][flashy.solver][INFO] - Train | Epoch 1 | 8/10 | 3.63 it/sec | lr 1.00E-04 | grad_norm 4.291E-01 | ce 8.084 | ppl 3241.677
[05-20 10:18:15][flashy.solver][INFO] - Train | Epoch 1 | 9/10 | 3.75 it/sec | lr 1.00E-04 | grad_norm 5.017E-01 | ce 8.040 | ppl 3101.891
[05-20 10:18:15][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=4.543E-01 | ce=8.069 | ppl=3195.706 | duration=2.814
[05-20 10:19:21][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 10:19:21][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 10:19:21][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 10:19:21][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 10:19:21][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 10:19:21][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:19:21][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:19:21][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 10:19:38][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 10:19:38][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 10:19:38][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 10:19:38][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 10:19:38][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 10:19:38][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:19:38][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:19:38][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 10:19:39][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 10:19:39][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 10:19:51][flashy.solver][INFO] - Model hash: 17ec360e796d79ea6c5d29324f2e44425407fa47
[05-20 10:19:51][flashy.solver][INFO] - No EMA on the model.
[05-20 10:19:51][flashy.solver][INFO] - Model size: 1838.34 M params
[05-20 10:19:51][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 29.41 GB
[05-20 10:19:51][flashy.solver][INFO] - Restoring weights and history.
[05-20 10:20:00][flashy.solver][INFO] - Model hash: 17ec360e796d79ea6c5d29324f2e44425407fa47
[05-20 10:20:01][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 10:20:02][flashy.solver][INFO] - Train | Epoch 1 | 1/10 | 0.87 it/sec | lr 1.00E-04 | grad_norm 6.036E+00 | ce 7.865 | ppl 2669.385
[05-20 10:20:03][flashy.solver][INFO] - Train | Epoch 1 | 2/10 | 0.96 it/sec | lr 1.00E-04 | grad_norm 1.788E+00 | ce 7.685 | ppl 2176.189
[05-20 10:20:04][flashy.solver][INFO] - Train | Epoch 1 | 3/10 | 1.01 it/sec | lr 1.00E-04 | grad_norm 1.300E+00 | ce 7.579 | ppl 1956.641
[05-20 10:20:05][flashy.solver][INFO] - Train | Epoch 1 | 4/10 | 1.05 it/sec | lr 1.00E-04 | grad_norm 1.218E+00 | ce 7.410 | ppl 1653.122
[05-20 10:20:05][flashy.solver][INFO] - Train | Epoch 1 | 5/10 | 1.08 it/sec | lr 1.00E-04 | grad_norm 1.175E+00 | ce 7.222 | ppl 1369.862
[05-20 10:20:06][flashy.solver][INFO] - Train | Epoch 1 | 6/10 | 1.10 it/sec | lr 1.00E-04 | grad_norm 1.085E+00 | ce 7.217 | ppl 1363.011
[05-20 10:20:07][flashy.solver][INFO] - Train | Epoch 1 | 7/10 | 1.11 it/sec | lr 1.00E-04 | grad_norm 9.294E-01 | ce 7.110 | ppl 1223.996
[05-20 10:20:08][flashy.solver][INFO] - Train | Epoch 1 | 8/10 | 1.13 it/sec | lr 1.00E-04 | grad_norm 1.072E+00 | ce 6.978 | ppl 1073.040
[05-20 10:20:09][flashy.solver][INFO] - Train | Epoch 1 | 9/10 | 1.14 it/sec | lr 1.00E-04 | grad_norm 9.572E-01 | ce 6.934 | ppl 1026.170
[05-20 10:20:09][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=2.160E+00 | ce=7.387 | ppl=1718.080 | duration=8.954
[05-20 10:26:54][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 10:26:54][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 10:26:54][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 10:26:54][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 10:26:54][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 10:26:54][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:26:54][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:26:54][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 10:26:54][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 10:26:54][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 10:27:06][flashy.solver][INFO] - Model hash: 17ec360e796d79ea6c5d29324f2e44425407fa47
[05-20 10:27:06][flashy.solver][INFO] - No EMA on the model.
[05-20 10:27:06][flashy.solver][INFO] - Model size: 1838.34 M params
[05-20 10:27:06][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 29.41 GB
[05-20 10:27:06][flashy.solver][INFO] - Restoring weights and history.
[05-20 10:27:15][flashy.solver][INFO] - Model hash: 17ec360e796d79ea6c5d29324f2e44425407fa47
[05-20 10:27:15][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 10:27:17][flashy.solver][INFO] - Train | Epoch 1 | 1/10 | 0.88 it/sec | lr 1.00E-04 | grad_norm 6.036E+00 | ce 7.865 | ppl 2669.385
[05-20 10:27:18][flashy.solver][INFO] - Train | Epoch 1 | 2/10 | 0.97 it/sec | lr 1.00E-04 | grad_norm 1.788E+00 | ce 7.685 | ppl 2176.189
[05-20 10:27:19][flashy.solver][INFO] - Train | Epoch 1 | 3/10 | 1.02 it/sec | lr 1.00E-04 | grad_norm 1.300E+00 | ce 7.579 | ppl 1956.641
[05-20 10:27:19][flashy.solver][INFO] - Train | Epoch 1 | 4/10 | 1.06 it/sec | lr 1.00E-04 | grad_norm 1.218E+00 | ce 7.410 | ppl 1653.122
[05-20 10:27:20][flashy.solver][INFO] - Train | Epoch 1 | 5/10 | 1.09 it/sec | lr 1.00E-04 | grad_norm 1.175E+00 | ce 7.222 | ppl 1369.862
[05-20 10:27:21][flashy.solver][INFO] - Train | Epoch 1 | 6/10 | 1.11 it/sec | lr 1.00E-04 | grad_norm 1.085E+00 | ce 7.217 | ppl 1363.011
[05-20 10:27:22][flashy.solver][INFO] - Train | Epoch 1 | 7/10 | 1.12 it/sec | lr 1.00E-04 | grad_norm 9.294E-01 | ce 7.110 | ppl 1223.996
[05-20 10:27:23][flashy.solver][INFO] - Train | Epoch 1 | 8/10 | 1.13 it/sec | lr 1.00E-04 | grad_norm 1.072E+00 | ce 6.978 | ppl 1073.040
[05-20 10:27:23][flashy.solver][INFO] - Train | Epoch 1 | 9/10 | 1.14 it/sec | lr 1.00E-04 | grad_norm 9.572E-01 | ce 6.934 | ppl 1026.170
[05-20 10:27:24][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=2.160E+00 | ce=7.387 | ppl=1718.080 | duration=8.938
[05-20 10:27:41][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 10:27:41][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 10:27:41][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 10:27:41][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 10:27:41][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 10:27:41][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:27:41][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:27:41][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 10:27:42][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 10:27:42][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 10:27:54][flashy.solver][INFO] - Model hash: 17ec360e796d79ea6c5d29324f2e44425407fa47
[05-20 10:27:54][flashy.solver][INFO] - No EMA on the model.
[05-20 10:27:54][flashy.solver][INFO] - Model size: 1838.34 M params
[05-20 10:27:54][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 29.41 GB
[05-20 10:27:54][flashy.solver][INFO] - Restoring weights and history.
[05-20 10:28:02][flashy.solver][INFO] - Model hash: 17ec360e796d79ea6c5d29324f2e44425407fa47
[05-20 10:28:03][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 10:28:27][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 10:28:27][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 10:28:27][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 10:28:27][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 10:28:27][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 10:28:27][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:28:27][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:28:27][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 10:28:27][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 10:28:27][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 10:28:39][flashy.solver][INFO] - Model hash: 17ec360e796d79ea6c5d29324f2e44425407fa47
[05-20 10:28:39][flashy.solver][INFO] - No EMA on the model.
[05-20 10:28:39][flashy.solver][INFO] - Model size: 1838.34 M params
[05-20 10:28:39][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 29.41 GB
[05-20 10:28:39][flashy.solver][INFO] - Restoring weights and history.
[05-20 10:28:48][flashy.solver][INFO] - Model hash: 17ec360e796d79ea6c5d29324f2e44425407fa47
[05-20 10:28:54][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 10:31:14][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 10:31:14][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 10:31:14][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 10:31:14][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 10:31:14][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 10:31:14][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:31:14][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:31:14][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 10:31:15][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 10:31:15][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 10:31:18][flashy.solver][INFO] - Model hash: c8b9170701e73c0c04243c93e1e31631ba1daf49
[05-20 10:31:18][flashy.solver][INFO] - No EMA on the model.
[05-20 10:31:18][flashy.solver][INFO] - Model size: 420.11 M params
[05-20 10:31:18][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 6.72 GB
[05-20 10:31:18][flashy.solver][INFO] - Restoring weights and history.
[05-20 10:31:20][flashy.solver][INFO] - Model hash: c8b9170701e73c0c04243c93e1e31631ba1daf49
[05-20 10:31:26][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 10:31:51][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 10:31:51][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 10:31:51][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 10:31:51][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 10:31:51][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 10:31:51][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:31:51][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 10:31:51][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 10:31:52][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 10:31:52][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 10:31:53][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 10:31:53][flashy.solver][INFO] - No EMA on the model.
[05-20 10:31:53][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 10:31:53][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 10:31:53][flashy.solver][INFO] - Restoring weights and history.
[05-20 10:31:53][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 10:32:00][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 10:51:10][flashy.solver][INFO] - Train | Epoch 1 | 200/2000 | 0.17 it/sec | lr 1.00E-04 | grad_norm 1.913E-01 | ce 7.440 | ppl 1804.100
[05-20 11:10:28][flashy.solver][INFO] - Train | Epoch 1 | 400/2000 | 0.17 it/sec | lr 1.00E-04 | grad_norm 1.073E-01 | ce 6.707 | ppl 823.787
[05-20 11:29:43][flashy.solver][INFO] - Train | Epoch 1 | 600/2000 | 0.17 it/sec | lr 1.00E-04 | grad_norm 6.780E-02 | ce 6.442 | ppl 628.869
[05-20 11:49:00][flashy.solver][INFO] - Train | Epoch 1 | 800/2000 | 0.17 it/sec | lr 1.00E-04 | grad_norm 8.025E-02 | ce 6.301 | ppl 546.179
[05-20 12:43:51][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 12:43:51][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 12:43:51][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 12:43:51][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 12:43:51][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 12:43:51][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:43:51][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:43:51][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 12:43:52][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 12:43:52][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 12:43:53][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:43:53][flashy.solver][INFO] - No EMA on the model.
[05-20 12:43:53][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 12:43:53][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 12:43:53][flashy.solver][INFO] - Restoring weights and history.
[05-20 12:43:53][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-4-3.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-4-3.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-2-17.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-2-17.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-36.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-36.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-18-11-11.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-18-11-11.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-09-5-2.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-09-5-2.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-23-6-24.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-23-6-24.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-29-4-20.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-29-4-20.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-3-12.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-3-12.wav')
[05-20 12:43:53][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav')
[05-20 12:45:01][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 12:45:01][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 12:45:01][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 12:45:01][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 12:45:01][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 12:45:01][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:45:01][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:45:01][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 12:45:01][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 12:45:01][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 12:45:03][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:45:03][flashy.solver][INFO] - No EMA on the model.
[05-20 12:45:03][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 12:45:03][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 12:45:03][flashy.solver][INFO] - Restoring weights and history.
[05-20 12:45:03][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-4-3.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-4-3.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-2-17.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-2-17.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-36.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-36.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-18-11-11.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-18-11-11.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-09-5-2.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-09-5-2.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-23-6-24.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-23-6-24.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-29-4-20.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-29-4-20.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-3-12.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-3-12.wav')
[05-20 12:45:03][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav')
[05-20 12:45:39][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 12:45:39][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 12:45:39][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 12:45:39][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 12:45:39][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 12:45:39][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:45:39][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:45:39][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 12:45:40][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 12:45:40][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 12:45:41][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:45:41][flashy.solver][INFO] - No EMA on the model.
[05-20 12:45:41][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 12:45:41][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 12:45:41][flashy.solver][INFO] - Restoring weights and history.
[05-20 12:45:41][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-4-3.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-4-3.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-2-17.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-2-17.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-36.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-36.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-18-11-11.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-18-11-11.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-09-5-2.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-09-5-2.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-23-6-24.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-23-6-24.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-29-4-20.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-29-4-20.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-3-12.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-11-3-12.wav')
[05-20 12:45:41][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-9.wav')
[05-20 12:46:23][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 12:46:23][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 12:46:23][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 12:46:23][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 12:46:23][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 12:46:23][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:46:23][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:46:23][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 12:46:24][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 12:46:24][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 12:46:25][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:46:25][flashy.solver][INFO] - No EMA on the model.
[05-20 12:46:25][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 12:46:25][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 12:46:25][flashy.solver][INFO] - Restoring weights and history.
[05-20 12:46:25][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:54:24][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 12:54:24][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 12:54:24][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 12:54:24][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 12:54:24][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 12:54:24][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:54:24][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:54:24][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 12:54:25][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 12:54:25][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 12:54:26][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:54:26][flashy.solver][INFO] - No EMA on the model.
[05-20 12:54:26][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 12:54:26][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 12:54:26][flashy.solver][INFO] - Restoring weights and history.
[05-20 12:54:26][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:55:16][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 12:55:16][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 12:55:16][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 12:55:16][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 12:55:16][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 12:55:16][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:55:16][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:55:16][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 12:55:16][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 12:55:16][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 12:55:17][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:55:17][flashy.solver][INFO] - No EMA on the model.
[05-20 12:55:17][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 12:55:17][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 12:55:17][flashy.solver][INFO] - Restoring weights and history.
[05-20 12:55:17][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:55:18][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:55:21][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:55:21][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:55:23][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 12:55:32][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:55:38][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:55:39][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:55:44][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:55:55][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:56:08][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:56:08][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-21.wav')
[05-20 12:56:17][dora.distrib][INFO] - world_size is 1, skipping init.
[05-20 12:56:17][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-20 12:56:17][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-20 12:56:17][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-20 12:56:17][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-20 12:56:17][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:56:17][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-20 12:56:17][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-20 12:56:18][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-20 12:56:18][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-20 12:56:19][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:56:19][flashy.solver][INFO] - No EMA on the model.
[05-20 12:56:19][flashy.solver][INFO] - Model size: 1.21 M params
[05-20 12:56:19][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-20 12:56:19][flashy.solver][INFO] - Restoring weights and history.
[05-20 12:56:19][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-20 12:56:24][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-20 13:12:30][flashy.solver][INFO] - Train | Epoch 1 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 2.620E+00 | ce 5.489 | ppl 528.680
[05-20 13:28:32][flashy.solver][INFO] - Train | Epoch 1 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.896E+00 | ce 3.386 | ppl 32.351
[05-20 13:44:37][flashy.solver][INFO] - Train | Epoch 1 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.919E-01 | ce 2.385 | ppl 11.154
[05-20 14:00:42][flashy.solver][INFO] - Train | Epoch 1 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 2.674E-01 | ce 1.978 | ppl 7.314
[05-20 14:16:47][flashy.solver][INFO] - Train | Epoch 1 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.848E-01 | ce 1.758 | ppl 5.862
[05-20 14:32:55][flashy.solver][INFO] - Train | Epoch 1 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.847E-01 | ce 1.588 | ppl 4.930
[05-20 14:49:02][flashy.solver][INFO] - Train | Epoch 1 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 2.033E-01 | ce 1.461 | ppl 4.340
[05-20 15:06:22][flashy.solver][INFO] - Train | Epoch 1 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 2.552E-01 | ce 1.345 | ppl 3.863
[05-20 15:24:35][flashy.solver][INFO] - Train | Epoch 1 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 2.984E-01 | ce 1.208 | ppl 3.373
[05-20 15:42:49][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[05-20 15:43:31][flashy.solver][INFO] - Valid | Epoch 1 | 7/79 | 0.19 it/sec | ce 1.049 | ppl 2.860
[05-20 15:44:07][flashy.solver][INFO] - Valid | Epoch 1 | 14/79 | 0.19 it/sec | ce 1.011 | ppl 2.756
[05-20 15:44:43][flashy.solver][INFO] - Valid | Epoch 1 | 21/79 | 0.19 it/sec | ce 1.075 | ppl 2.940
[05-20 15:45:19][flashy.solver][INFO] - Valid | Epoch 1 | 28/79 | 0.19 it/sec | ce 1.010 | ppl 2.767
[05-20 15:45:56][flashy.solver][INFO] - Valid | Epoch 1 | 35/79 | 0.19 it/sec | ce 1.061 | ppl 2.898
[05-20 15:46:32][flashy.solver][INFO] - Valid | Epoch 1 | 42/79 | 0.19 it/sec | ce 1.001 | ppl 2.733
[05-20 15:47:09][flashy.solver][INFO] - Valid | Epoch 1 | 49/79 | 0.19 it/sec | ce 1.156 | ppl 3.193
[05-20 15:47:45][flashy.solver][INFO] - Valid | Epoch 1 | 56/79 | 0.19 it/sec | ce 0.998 | ppl 2.722
[05-20 15:48:22][flashy.solver][INFO] - Valid | Epoch 1 | 63/79 | 0.19 it/sec | ce 1.118 | ppl 3.063
[05-20 15:48:58][flashy.solver][INFO] - Valid | Epoch 1 | 70/79 | 0.19 it/sec | ce 1.056 | ppl 2.885
[05-20 15:49:35][flashy.solver][INFO] - Valid | Epoch 1 | 77/79 | 0.19 it/sec | ce 1.107 | ppl 3.034
[05-20 15:49:36][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[05-20 15:49:36][flashy.solver][INFO] - New best state with ce=1.058 (was inf)
[05-20 15:49:36][flashy.solver][INFO] - Model hash: 86e3de79d62bf9ad49b0fd7c60e558675cebfebe
[05-20 15:49:36][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-20 16:06:45][flashy.solver][INFO] - Train | Epoch 2 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 4.056E-01 | ce 1.032 | ppl 2.821
[05-20 16:22:50][flashy.solver][INFO] - Train | Epoch 2 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 5.198E-01 | ce 0.948 | ppl 2.596
[05-20 16:38:55][flashy.solver][INFO] - Train | Epoch 2 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.279E-01 | ce 0.878 | ppl 2.417
[05-20 16:55:01][flashy.solver][INFO] - Train | Epoch 2 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.918E-01 | ce 0.839 | ppl 2.326
[05-20 17:11:03][flashy.solver][INFO] - Train | Epoch 2 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.742E-01 | ce 0.756 | ppl 2.141
[05-20 17:27:05][flashy.solver][INFO] - Train | Epoch 2 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.569E-01 | ce 0.700 | ppl 2.023
[05-20 17:43:10][flashy.solver][INFO] - Train | Epoch 2 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.032E+00 | ce 0.680 | ppl 1.986
[05-20 17:59:15][flashy.solver][INFO] - Train | Epoch 2 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.140E+00 | ce 0.626 | ppl 1.879
[05-20 18:15:22][flashy.solver][INFO] - Train | Epoch 2 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.297E+00 | ce 0.601 | ppl 1.831
[05-20 18:31:25][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[05-20 18:32:02][flashy.solver][INFO] - Valid | Epoch 2 | 7/79 | 0.21 it/sec | ce 0.510 | ppl 1.670
[05-20 18:32:35][flashy.solver][INFO] - Valid | Epoch 2 | 14/79 | 0.22 it/sec | ce 0.484 | ppl 1.629
[05-20 18:33:08][flashy.solver][INFO] - Valid | Epoch 2 | 21/79 | 0.21 it/sec | ce 0.538 | ppl 1.718
[05-20 18:33:40][flashy.solver][INFO] - Valid | Epoch 2 | 28/79 | 0.22 it/sec | ce 0.487 | ppl 1.637
[05-20 18:34:12][flashy.solver][INFO] - Valid | Epoch 2 | 35/79 | 0.22 it/sec | ce 0.537 | ppl 1.715
[05-20 18:34:45][flashy.solver][INFO] - Valid | Epoch 2 | 42/79 | 0.22 it/sec | ce 0.486 | ppl 1.632
[05-20 18:35:18][flashy.solver][INFO] - Valid | Epoch 2 | 49/79 | 0.21 it/sec | ce 0.607 | ppl 1.845
[05-20 18:35:51][flashy.solver][INFO] - Valid | Epoch 2 | 56/79 | 0.21 it/sec | ce 0.479 | ppl 1.619
[05-20 18:36:23][flashy.solver][INFO] - Valid | Epoch 2 | 63/79 | 0.21 it/sec | ce 0.579 | ppl 1.787
[05-20 18:36:56][flashy.solver][INFO] - Valid | Epoch 2 | 70/79 | 0.21 it/sec | ce 0.536 | ppl 1.715
[05-20 18:37:29][flashy.solver][INFO] - Valid | Epoch 2 | 77/79 | 0.21 it/sec | ce 0.593 | ppl 1.813
[05-20 18:37:29][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[05-20 18:37:29][flashy.solver][INFO] - New best state with ce=0.530 (was 1.058)
[05-20 18:37:29][flashy.solver][INFO] - Model hash: 4cb540fd02b4e993b18fd192777ee196af8399e0
[05-20 18:37:29][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-20 18:53:40][flashy.solver][INFO] - Train | Epoch 3 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.686E+00 | ce 0.538 | ppl 1.721
[05-20 19:09:44][flashy.solver][INFO] - Train | Epoch 3 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.429E+00 | ce 0.516 | ppl 1.685
[05-20 19:25:49][flashy.solver][INFO] - Train | Epoch 3 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.660E+00 | ce 0.497 | ppl 1.652
[05-20 19:41:52][flashy.solver][INFO] - Train | Epoch 3 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.687E+00 | ce 0.473 | ppl 1.614
[05-20 19:58:00][flashy.solver][INFO] - Train | Epoch 3 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.717E+00 | ce 0.474 | ppl 1.615
[05-20 20:14:04][flashy.solver][INFO] - Train | Epoch 3 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.509E+00 | ce 0.435 | ppl 1.552
[05-20 20:30:03][flashy.solver][INFO] - Train | Epoch 3 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.866E+00 | ce 0.430 | ppl 1.543
[05-20 20:46:10][flashy.solver][INFO] - Train | Epoch 3 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.888E+00 | ce 0.420 | ppl 1.528
[05-20 21:02:11][flashy.solver][INFO] - Train | Epoch 3 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.892E+00 | ce 0.409 | ppl 1.512
[05-20 21:18:16][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[05-20 21:18:53][flashy.solver][INFO] - Valid | Epoch 3 | 7/79 | 0.21 it/sec | ce 0.367 | ppl 1.447
[05-20 21:19:25][flashy.solver][INFO] - Valid | Epoch 3 | 14/79 | 0.22 it/sec | ce 0.343 | ppl 1.415
[05-20 21:19:58][flashy.solver][INFO] - Valid | Epoch 3 | 21/79 | 0.21 it/sec | ce 0.395 | ppl 1.487
[05-20 21:20:30][flashy.solver][INFO] - Valid | Epoch 3 | 28/79 | 0.22 it/sec | ce 0.348 | ppl 1.424
[05-20 21:21:03][flashy.solver][INFO] - Valid | Epoch 3 | 35/79 | 0.22 it/sec | ce 0.397 | ppl 1.491
[05-20 21:21:35][flashy.solver][INFO] - Valid | Epoch 3 | 42/79 | 0.22 it/sec | ce 0.349 | ppl 1.422
[05-20 21:22:08][flashy.solver][INFO] - Valid | Epoch 3 | 49/79 | 0.21 it/sec | ce 0.456 | ppl 1.586
[05-20 21:22:41][flashy.solver][INFO] - Valid | Epoch 3 | 56/79 | 0.21 it/sec | ce 0.342 | ppl 1.410
[05-20 21:23:14][flashy.solver][INFO] - Valid | Epoch 3 | 63/79 | 0.21 it/sec | ce 0.432 | ppl 1.542
[05-20 21:23:46][flashy.solver][INFO] - Valid | Epoch 3 | 70/79 | 0.21 it/sec | ce 0.397 | ppl 1.491
[05-20 21:24:19][flashy.solver][INFO] - Valid | Epoch 3 | 77/79 | 0.21 it/sec | ce 0.451 | ppl 1.574
[05-20 21:24:20][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[05-20 21:24:20][flashy.solver][INFO] - New best state with ce=0.388 (was 0.530)
[05-20 21:24:20][flashy.solver][INFO] - Model hash: d7f624dad711222790a15a7a0a37c4a58d688b71
[05-20 21:24:20][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-20 21:40:28][flashy.solver][INFO] - Train | Epoch 4 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.485E+00 | ce 0.389 | ppl 1.481
[05-20 21:56:35][flashy.solver][INFO] - Train | Epoch 4 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.633E+00 | ce 0.405 | ppl 1.507
[05-20 22:12:39][flashy.solver][INFO] - Train | Epoch 4 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.571E+00 | ce 0.392 | ppl 1.486
[05-20 22:28:41][flashy.solver][INFO] - Train | Epoch 4 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.307E+00 | ce 0.383 | ppl 1.473
[05-20 22:44:47][flashy.solver][INFO] - Train | Epoch 4 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.052E+00 | ce 0.381 | ppl 1.471
[05-20 23:00:52][flashy.solver][INFO] - Train | Epoch 4 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.041E+00 | ce 0.371 | ppl 1.454
[05-20 23:16:56][flashy.solver][INFO] - Train | Epoch 4 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.167E+00 | ce 0.361 | ppl 1.440
[05-20 23:33:01][flashy.solver][INFO] - Train | Epoch 4 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.359E+00 | ce 0.357 | ppl 1.434
[05-20 23:49:03][flashy.solver][INFO] - Train | Epoch 4 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.227E+00 | ce 0.371 | ppl 1.456
[05-21 00:05:08][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[05-21 00:05:46][flashy.solver][INFO] - Valid | Epoch 4 | 7/79 | 0.21 it/sec | ce 0.326 | ppl 1.389
[05-21 00:06:18][flashy.solver][INFO] - Valid | Epoch 4 | 14/79 | 0.22 it/sec | ce 0.304 | ppl 1.360
[05-21 00:06:50][flashy.solver][INFO] - Valid | Epoch 4 | 21/79 | 0.22 it/sec | ce 0.354 | ppl 1.428
[05-21 00:07:23][flashy.solver][INFO] - Valid | Epoch 4 | 28/79 | 0.22 it/sec | ce 0.309 | ppl 1.369
[05-21 00:07:55][flashy.solver][INFO] - Valid | Epoch 4 | 35/79 | 0.22 it/sec | ce 0.356 | ppl 1.430
[05-21 00:08:28][flashy.solver][INFO] - Valid | Epoch 4 | 42/79 | 0.22 it/sec | ce 0.310 | ppl 1.367
[05-21 00:09:01][flashy.solver][INFO] - Valid | Epoch 4 | 49/79 | 0.21 it/sec | ce 0.411 | ppl 1.515
[05-21 00:09:33][flashy.solver][INFO] - Valid | Epoch 4 | 56/79 | 0.21 it/sec | ce 0.304 | ppl 1.359
[05-21 00:10:06][flashy.solver][INFO] - Valid | Epoch 4 | 63/79 | 0.21 it/sec | ce 0.388 | ppl 1.476
[05-21 00:10:39][flashy.solver][INFO] - Valid | Epoch 4 | 70/79 | 0.21 it/sec | ce 0.357 | ppl 1.432
[05-21 00:11:12][flashy.solver][INFO] - Valid | Epoch 4 | 77/79 | 0.21 it/sec | ce 0.408 | ppl 1.507
[05-21 00:11:12][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[05-21 00:11:12][flashy.solver][INFO] - New best state with ce=0.347 (was 0.388)
[05-21 00:11:12][flashy.solver][INFO] - Model hash: b597a9b0a2fa40852115db71f9d49b999c98118f
[05-21 00:11:12][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 00:27:20][flashy.solver][INFO] - Train | Epoch 5 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.255E+00 | ce 0.364 | ppl 1.445
[05-21 00:43:24][flashy.solver][INFO] - Train | Epoch 5 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.045E+00 | ce 0.355 | ppl 1.433
[05-21 00:59:28][flashy.solver][INFO] - Train | Epoch 5 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.128E+00 | ce 0.365 | ppl 1.447
[05-21 01:15:34][flashy.solver][INFO] - Train | Epoch 5 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.055E+00 | ce 0.354 | ppl 1.431
[05-21 01:31:40][flashy.solver][INFO] - Train | Epoch 5 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.032E+00 | ce 0.365 | ppl 1.446
[05-21 01:47:46][flashy.solver][INFO] - Train | Epoch 5 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.116E+00 | ce 0.342 | ppl 1.413
[05-21 02:03:47][flashy.solver][INFO] - Train | Epoch 5 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.169E+00 | ce 0.341 | ppl 1.412
[05-21 02:19:54][flashy.solver][INFO] - Train | Epoch 5 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.142E-01 | ce 0.351 | ppl 1.426
[05-21 02:35:58][flashy.solver][INFO] - Train | Epoch 5 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.068E+00 | ce 0.330 | ppl 1.396
[05-21 02:52:02][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[05-21 02:52:39][flashy.solver][INFO] - Valid | Epoch 5 | 7/79 | 0.21 it/sec | ce 0.307 | ppl 1.362
[05-21 02:53:12][flashy.solver][INFO] - Valid | Epoch 5 | 14/79 | 0.22 it/sec | ce 0.286 | ppl 1.335
[05-21 02:53:45][flashy.solver][INFO] - Valid | Epoch 5 | 21/79 | 0.21 it/sec | ce 0.335 | ppl 1.400
[05-21 02:54:17][flashy.solver][INFO] - Valid | Epoch 5 | 28/79 | 0.22 it/sec | ce 0.291 | ppl 1.344
[05-21 02:54:49][flashy.solver][INFO] - Valid | Epoch 5 | 35/79 | 0.22 it/sec | ce 0.335 | ppl 1.401
[05-21 02:55:22][flashy.solver][INFO] - Valid | Epoch 5 | 42/79 | 0.22 it/sec | ce 0.291 | ppl 1.342
[05-21 02:55:55][flashy.solver][INFO] - Valid | Epoch 5 | 49/79 | 0.21 it/sec | ce 0.388 | ppl 1.481
[05-21 02:56:27][flashy.solver][INFO] - Valid | Epoch 5 | 56/79 | 0.22 it/sec | ce 0.287 | ppl 1.335
[05-21 02:57:00][flashy.solver][INFO] - Valid | Epoch 5 | 63/79 | 0.21 it/sec | ce 0.366 | ppl 1.444
[05-21 02:57:33][flashy.solver][INFO] - Valid | Epoch 5 | 70/79 | 0.21 it/sec | ce 0.337 | ppl 1.403
[05-21 02:58:06][flashy.solver][INFO] - Valid | Epoch 5 | 77/79 | 0.21 it/sec | ce 0.386 | ppl 1.475
[05-21 02:58:06][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[05-21 02:58:06][flashy.solver][INFO] - New best state with ce=0.327 (was 0.347)
[05-21 02:58:06][flashy.solver][INFO] - Model hash: d71c9592f2994c7327f00a0b5662ba57f1ecfb9f
[05-21 02:58:07][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 03:14:18][flashy.solver][INFO] - Train | Epoch 6 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.078E-01 | ce 0.340 | ppl 1.410
[05-21 03:30:22][flashy.solver][INFO] - Train | Epoch 6 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.173E+00 | ce 0.338 | ppl 1.407
[05-21 03:46:25][flashy.solver][INFO] - Train | Epoch 6 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.597E-01 | ce 0.337 | ppl 1.405
[05-21 04:02:27][flashy.solver][INFO] - Train | Epoch 6 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.513E-01 | ce 0.335 | ppl 1.405
[05-21 04:18:31][flashy.solver][INFO] - Train | Epoch 6 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.502E-01 | ce 0.333 | ppl 1.400
[05-21 04:34:37][flashy.solver][INFO] - Train | Epoch 6 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.503E-01 | ce 0.329 | ppl 1.395
[05-21 04:50:41][flashy.solver][INFO] - Train | Epoch 6 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.723E-01 | ce 0.341 | ppl 1.410
[05-21 05:06:44][flashy.solver][INFO] - Train | Epoch 6 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.495E-01 | ce 0.319 | ppl 1.380
[05-21 05:22:51][flashy.solver][INFO] - Train | Epoch 6 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.341E-01 | ce 0.328 | ppl 1.393
[05-21 05:38:55][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[05-21 05:39:32][flashy.solver][INFO] - Valid | Epoch 6 | 7/79 | 0.21 it/sec | ce 0.294 | ppl 1.345
[05-21 05:40:04][flashy.solver][INFO] - Valid | Epoch 6 | 14/79 | 0.21 it/sec | ce 0.274 | ppl 1.320
[05-21 05:40:37][flashy.solver][INFO] - Valid | Epoch 6 | 21/79 | 0.21 it/sec | ce 0.323 | ppl 1.383
[05-21 05:41:09][flashy.solver][INFO] - Valid | Epoch 6 | 28/79 | 0.22 it/sec | ce 0.280 | ppl 1.329
[05-21 05:41:42][flashy.solver][INFO] - Valid | Epoch 6 | 35/79 | 0.22 it/sec | ce 0.322 | ppl 1.383
[05-21 05:42:15][flashy.solver][INFO] - Valid | Epoch 6 | 42/79 | 0.22 it/sec | ce 0.280 | ppl 1.326
[05-21 05:42:48][flashy.solver][INFO] - Valid | Epoch 6 | 49/79 | 0.21 it/sec | ce 0.373 | ppl 1.458
[05-21 05:43:20][flashy.solver][INFO] - Valid | Epoch 6 | 56/79 | 0.21 it/sec | ce 0.277 | ppl 1.321
[05-21 05:43:53][flashy.solver][INFO] - Valid | Epoch 6 | 63/79 | 0.21 it/sec | ce 0.352 | ppl 1.424
[05-21 05:44:26][flashy.solver][INFO] - Valid | Epoch 6 | 70/79 | 0.21 it/sec | ce 0.325 | ppl 1.386
[05-21 05:44:59][flashy.solver][INFO] - Valid | Epoch 6 | 77/79 | 0.21 it/sec | ce 0.372 | ppl 1.454
[05-21 05:44:59][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[05-21 05:44:59][flashy.solver][INFO] - New best state with ce=0.315 (was 0.327)
[05-21 05:44:59][flashy.solver][INFO] - Model hash: 61e0e21f805fd4145c9d258067ba2306b8c11cde
[05-21 05:44:59][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 06:01:08][flashy.solver][INFO] - Train | Epoch 7 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.754E-01 | ce 0.323 | ppl 1.385
[05-21 06:17:13][flashy.solver][INFO] - Train | Epoch 7 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.524E-01 | ce 0.326 | ppl 1.390
[05-21 06:33:18][flashy.solver][INFO] - Train | Epoch 7 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.035E+00 | ce 0.331 | ppl 1.397
[05-21 06:49:22][flashy.solver][INFO] - Train | Epoch 7 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.787E-01 | ce 0.329 | ppl 1.395
[05-21 07:05:26][flashy.solver][INFO] - Train | Epoch 7 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.140E-01 | ce 0.319 | ppl 1.379
[05-21 07:21:31][flashy.solver][INFO] - Train | Epoch 7 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.761E-01 | ce 0.323 | ppl 1.387
[05-21 07:37:36][flashy.solver][INFO] - Train | Epoch 7 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.099E-01 | ce 0.319 | ppl 1.380
[05-21 07:53:40][flashy.solver][INFO] - Train | Epoch 7 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.774E-01 | ce 0.315 | ppl 1.375
[05-21 08:09:46][flashy.solver][INFO] - Train | Epoch 7 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.199E-01 | ce 0.307 | ppl 1.364
[05-21 08:25:52][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=7.891E-01 | ce=0.322 | ppl=1.385 | duration=9652.954
[05-21 08:26:30][flashy.solver][INFO] - Valid | Epoch 7 | 7/79 | 0.21 it/sec | ce 0.284 | ppl 1.332
[05-21 08:27:02][flashy.solver][INFO] - Valid | Epoch 7 | 14/79 | 0.22 it/sec | ce 0.265 | ppl 1.307
[05-21 08:27:35][flashy.solver][INFO] - Valid | Epoch 7 | 21/79 | 0.22 it/sec | ce 0.312 | ppl 1.369
[05-21 08:28:07][flashy.solver][INFO] - Valid | Epoch 7 | 28/79 | 0.22 it/sec | ce 0.270 | ppl 1.316
[05-21 08:28:47][dora.distrib][INFO] - world_size is 1, skipping init.
[05-21 08:28:47][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-21 08:28:47][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-21 08:28:47][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-21 08:28:47][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-21 08:28:47][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-21 08:28:47][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-21 08:28:47][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-21 08:28:47][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-21 08:28:47][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-21 08:28:49][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-21 08:28:49][flashy.solver][INFO] - No EMA on the model.
[05-21 08:28:49][flashy.solver][INFO] - Model size: 1.21 M params
[05-21 08:28:49][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-21 08:28:49][flashy.solver][INFO] - Restoring weights and history.
[05-21 08:28:49][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 08:28:49][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 08:28:49][flashy.solver][INFO] - Model hash: 61e0e21f805fd4145c9d258067ba2306b8c11cde
[05-21 08:28:49][flashy.solver][INFO] - Replaying past metrics...
[05-21 08:28:49][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[05-21 08:28:49][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[05-21 08:28:49][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[05-21 08:28:49][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[05-21 08:28:49][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[05-21 08:28:49][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[05-21 08:28:49][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[05-21 08:28:49][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[05-21 08:28:49][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[05-21 08:28:49][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[05-21 08:28:49][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[05-21 08:28:49][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[05-21 08:28:54][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-21 08:45:03][flashy.solver][INFO] - Train | Epoch 7 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.814E-01 | ce 0.320 | ppl 1.382
[05-21 09:01:10][flashy.solver][INFO] - Train | Epoch 7 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.331E-01 | ce 0.326 | ppl 1.390
[05-21 09:17:18][flashy.solver][INFO] - Train | Epoch 7 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.351E-01 | ce 0.328 | ppl 1.393
[05-21 09:33:28][flashy.solver][INFO] - Train | Epoch 7 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.451E-01 | ce 0.325 | ppl 1.388
[05-21 09:49:34][flashy.solver][INFO] - Train | Epoch 7 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.291E-01 | ce 0.326 | ppl 1.390
[05-21 10:05:39][flashy.solver][INFO] - Train | Epoch 7 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.280E-01 | ce 0.314 | ppl 1.372
[05-21 10:21:49][flashy.solver][INFO] - Train | Epoch 7 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.425E-01 | ce 0.319 | ppl 1.380
[05-21 10:37:56][flashy.solver][INFO] - Train | Epoch 7 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.844E-01 | ce 0.329 | ppl 1.394
[05-21 10:54:09][flashy.solver][INFO] - Train | Epoch 7 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.620E-01 | ce 0.327 | ppl 1.391
[05-21 11:10:21][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[05-21 11:10:59][flashy.solver][INFO] - Valid | Epoch 7 | 7/79 | 0.21 it/sec | ce 0.284 | ppl 1.331
[05-21 11:11:32][flashy.solver][INFO] - Valid | Epoch 7 | 14/79 | 0.21 it/sec | ce 0.265 | ppl 1.307
[05-21 11:12:05][flashy.solver][INFO] - Valid | Epoch 7 | 21/79 | 0.21 it/sec | ce 0.312 | ppl 1.369
[05-21 11:12:38][flashy.solver][INFO] - Valid | Epoch 7 | 28/79 | 0.21 it/sec | ce 0.270 | ppl 1.316
[05-21 11:13:11][flashy.solver][INFO] - Valid | Epoch 7 | 35/79 | 0.21 it/sec | ce 0.312 | ppl 1.368
[05-21 11:13:44][flashy.solver][INFO] - Valid | Epoch 7 | 42/79 | 0.21 it/sec | ce 0.270 | ppl 1.313
[05-21 11:14:17][flashy.solver][INFO] - Valid | Epoch 7 | 49/79 | 0.21 it/sec | ce 0.361 | ppl 1.440
[05-21 11:14:50][flashy.solver][INFO] - Valid | Epoch 7 | 56/79 | 0.21 it/sec | ce 0.268 | ppl 1.309
[05-21 11:15:23][flashy.solver][INFO] - Valid | Epoch 7 | 63/79 | 0.21 it/sec | ce 0.341 | ppl 1.408
[05-21 11:15:56][flashy.solver][INFO] - Valid | Epoch 7 | 70/79 | 0.21 it/sec | ce 0.314 | ppl 1.371
[05-21 11:16:30][flashy.solver][INFO] - Valid | Epoch 7 | 77/79 | 0.21 it/sec | ce 0.360 | ppl 1.436
[05-21 11:16:30][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[05-21 11:16:30][flashy.solver][INFO] - New best state with ce=0.304 (was 0.315)
[05-21 11:16:30][flashy.solver][INFO] - Model hash: 00820a04196ce0b5827ec70106749b69e59ee6d4
[05-21 11:16:30][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 11:32:54][flashy.solver][INFO] - Train | Epoch 8 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.677E-01 | ce 0.318 | ppl 1.378
[05-21 11:49:21][flashy.solver][INFO] - Train | Epoch 8 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.054E-01 | ce 0.321 | ppl 1.383
[05-21 12:05:45][flashy.solver][INFO] - Train | Epoch 8 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.535E-01 | ce 0.324 | ppl 1.386
[05-21 12:22:03][flashy.solver][INFO] - Train | Epoch 8 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.471E-01 | ce 0.318 | ppl 1.380
[05-21 12:38:13][flashy.solver][INFO] - Train | Epoch 8 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.420E-01 | ce 0.309 | ppl 1.366
[05-21 12:54:25][flashy.solver][INFO] - Train | Epoch 8 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.808E-01 | ce 0.307 | ppl 1.363
[05-21 13:10:39][flashy.solver][INFO] - Train | Epoch 8 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.593E-01 | ce 0.312 | ppl 1.370
[05-21 13:26:51][flashy.solver][INFO] - Train | Epoch 8 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.783E-01 | ce 0.328 | ppl 1.394
[05-21 13:43:03][flashy.solver][INFO] - Train | Epoch 8 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.021E-01 | ce 0.318 | ppl 1.379
[05-21 13:59:13][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[05-21 13:59:51][flashy.solver][INFO] - Valid | Epoch 8 | 7/79 | 0.21 it/sec | ce 0.277 | ppl 1.322
[05-21 14:00:24][flashy.solver][INFO] - Valid | Epoch 8 | 14/79 | 0.21 it/sec | ce 0.258 | ppl 1.299
[05-21 14:00:57][flashy.solver][INFO] - Valid | Epoch 8 | 21/79 | 0.21 it/sec | ce 0.305 | ppl 1.359
[05-21 14:01:29][flashy.solver][INFO] - Valid | Epoch 8 | 28/79 | 0.21 it/sec | ce 0.264 | ppl 1.307
[05-21 14:02:02][flashy.solver][INFO] - Valid | Epoch 8 | 35/79 | 0.21 it/sec | ce 0.304 | ppl 1.357
[05-21 14:02:35][flashy.solver][INFO] - Valid | Epoch 8 | 42/79 | 0.21 it/sec | ce 0.264 | ppl 1.304
[05-21 14:03:08][flashy.solver][INFO] - Valid | Epoch 8 | 49/79 | 0.21 it/sec | ce 0.352 | ppl 1.428
[05-21 14:03:41][flashy.solver][INFO] - Valid | Epoch 8 | 56/79 | 0.21 it/sec | ce 0.262 | ppl 1.301
[05-21 14:04:14][flashy.solver][INFO] - Valid | Epoch 8 | 63/79 | 0.21 it/sec | ce 0.333 | ppl 1.397
[05-21 14:04:47][flashy.solver][INFO] - Valid | Epoch 8 | 70/79 | 0.21 it/sec | ce 0.307 | ppl 1.361
[05-21 14:05:20][flashy.solver][INFO] - Valid | Epoch 8 | 77/79 | 0.21 it/sec | ce 0.352 | ppl 1.425
[05-21 14:05:21][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[05-21 14:05:21][flashy.solver][INFO] - New best state with ce=0.297 (was 0.304)
[05-21 14:05:21][flashy.solver][INFO] - Model hash: ca30fbdcd78996014fbc21d3a1ea8f1dc82507e6
[05-21 14:05:21][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 14:21:36][flashy.solver][INFO] - Train | Epoch 9 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.787E-01 | ce 0.304 | ppl 1.359
[05-21 14:37:49][flashy.solver][INFO] - Train | Epoch 9 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.528E-01 | ce 0.309 | ppl 1.366
[05-21 14:54:01][flashy.solver][INFO] - Train | Epoch 9 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.652E-01 | ce 0.307 | ppl 1.363
[05-21 15:10:11][flashy.solver][INFO] - Train | Epoch 9 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.049E-01 | ce 0.297 | ppl 1.349
[05-21 15:26:23][flashy.solver][INFO] - Train | Epoch 9 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.791E-01 | ce 0.306 | ppl 1.362
[05-21 15:42:35][flashy.solver][INFO] - Train | Epoch 9 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.344E-01 | ce 0.302 | ppl 1.356
[05-21 15:58:46][flashy.solver][INFO] - Train | Epoch 9 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.516E-01 | ce 0.310 | ppl 1.368
[05-21 16:14:55][flashy.solver][INFO] - Train | Epoch 9 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.680E-01 | ce 0.300 | ppl 1.354
[05-21 16:31:05][flashy.solver][INFO] - Train | Epoch 9 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.721E-01 | ce 0.311 | ppl 1.369
[05-21 16:47:16][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[05-21 16:47:53][flashy.solver][INFO] - Valid | Epoch 9 | 7/79 | 0.21 it/sec | ce 0.271 | ppl 1.314
[05-21 16:48:26][flashy.solver][INFO] - Valid | Epoch 9 | 14/79 | 0.21 it/sec | ce 0.252 | ppl 1.291
[05-21 16:48:59][flashy.solver][INFO] - Valid | Epoch 9 | 21/79 | 0.21 it/sec | ce 0.299 | ppl 1.350
[05-21 16:49:32][flashy.solver][INFO] - Valid | Epoch 9 | 28/79 | 0.21 it/sec | ce 0.258 | ppl 1.299
[05-21 16:50:05][flashy.solver][INFO] - Valid | Epoch 9 | 35/79 | 0.21 it/sec | ce 0.297 | ppl 1.348
[05-21 16:50:38][flashy.solver][INFO] - Valid | Epoch 9 | 42/79 | 0.21 it/sec | ce 0.258 | ppl 1.297
[05-21 16:51:11][flashy.solver][INFO] - Valid | Epoch 9 | 49/79 | 0.21 it/sec | ce 0.345 | ppl 1.417
[05-21 16:51:44][flashy.solver][INFO] - Valid | Epoch 9 | 56/79 | 0.21 it/sec | ce 0.256 | ppl 1.294
[05-21 16:52:17][flashy.solver][INFO] - Valid | Epoch 9 | 63/79 | 0.21 it/sec | ce 0.326 | ppl 1.387
[05-21 16:52:50][flashy.solver][INFO] - Valid | Epoch 9 | 70/79 | 0.21 it/sec | ce 0.301 | ppl 1.352
[05-21 16:53:23][flashy.solver][INFO] - Valid | Epoch 9 | 77/79 | 0.21 it/sec | ce 0.345 | ppl 1.414
[05-21 16:53:23][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[05-21 16:53:23][flashy.solver][INFO] - New best state with ce=0.291 (was 0.297)
[05-21 16:53:24][flashy.solver][INFO] - Model hash: 88c900d9d98ae17e07fa544c3c533099a8609a18
[05-21 16:53:24][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 17:09:39][flashy.solver][INFO] - Train | Epoch 10 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.715E-01 | ce 0.301 | ppl 1.355
[05-21 17:25:51][flashy.solver][INFO] - Train | Epoch 10 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.295E-01 | ce 0.302 | ppl 1.357
[05-21 17:42:03][flashy.solver][INFO] - Train | Epoch 10 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.886E-01 | ce 0.304 | ppl 1.359
[05-21 17:58:17][flashy.solver][INFO] - Train | Epoch 10 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.435E-01 | ce 0.295 | ppl 1.347
[05-21 18:14:27][flashy.solver][INFO] - Train | Epoch 10 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.372E-01 | ce 0.300 | ppl 1.354
[05-21 18:30:38][flashy.solver][INFO] - Train | Epoch 10 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.626E-01 | ce 0.303 | ppl 1.357
[05-21 18:46:49][flashy.solver][INFO] - Train | Epoch 10 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.308E-01 | ce 0.292 | ppl 1.343
[05-21 19:03:03][flashy.solver][INFO] - Train | Epoch 10 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.222E-01 | ce 0.297 | ppl 1.349
[05-21 19:19:16][flashy.solver][INFO] - Train | Epoch 10 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.568E-01 | ce 0.303 | ppl 1.358
[05-21 19:35:24][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[05-21 19:36:02][flashy.solver][INFO] - Valid | Epoch 10 | 7/79 | 0.21 it/sec | ce 0.266 | ppl 1.308
[05-21 19:36:34][flashy.solver][INFO] - Valid | Epoch 10 | 14/79 | 0.21 it/sec | ce 0.248 | ppl 1.285
[05-21 19:37:07][flashy.solver][INFO] - Valid | Epoch 10 | 21/79 | 0.21 it/sec | ce 0.294 | ppl 1.344
[05-21 19:37:40][flashy.solver][INFO] - Valid | Epoch 10 | 28/79 | 0.21 it/sec | ce 0.254 | ppl 1.293
[05-21 19:38:13][flashy.solver][INFO] - Valid | Epoch 10 | 35/79 | 0.21 it/sec | ce 0.292 | ppl 1.341
[05-21 19:38:46][flashy.solver][INFO] - Valid | Epoch 10 | 42/79 | 0.21 it/sec | ce 0.253 | ppl 1.291
[05-21 19:39:19][flashy.solver][INFO] - Valid | Epoch 10 | 49/79 | 0.21 it/sec | ce 0.338 | ppl 1.408
[05-21 19:39:52][flashy.solver][INFO] - Valid | Epoch 10 | 56/79 | 0.21 it/sec | ce 0.252 | ppl 1.289
[05-21 19:40:25][flashy.solver][INFO] - Valid | Epoch 10 | 63/79 | 0.21 it/sec | ce 0.320 | ppl 1.379
[05-21 19:40:58][flashy.solver][INFO] - Valid | Epoch 10 | 70/79 | 0.21 it/sec | ce 0.295 | ppl 1.345
[05-21 19:41:31][flashy.solver][INFO] - Valid | Epoch 10 | 77/79 | 0.21 it/sec | ce 0.339 | ppl 1.405
[05-21 19:41:32][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[05-21 19:41:32][flashy.solver][INFO] - New best state with ce=0.286 (was 0.291)
[05-21 19:41:32][flashy.solver][INFO] - Model hash: 2c50b106773e472464cf2eb137222387b121fdfc
[05-21 19:41:32][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 19:57:44][flashy.solver][INFO] - Train | Epoch 11 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.468E-01 | ce 0.301 | ppl 1.355
[05-21 20:13:55][flashy.solver][INFO] - Train | Epoch 11 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.762E-01 | ce 0.292 | ppl 1.343
[05-21 20:30:06][flashy.solver][INFO] - Train | Epoch 11 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.107E-01 | ce 0.297 | ppl 1.350
[05-21 20:46:14][flashy.solver][INFO] - Train | Epoch 11 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.233E-01 | ce 0.286 | ppl 1.334
[05-21 21:02:25][flashy.solver][INFO] - Train | Epoch 11 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.115E-01 | ce 0.296 | ppl 1.347
[05-21 21:18:33][flashy.solver][INFO] - Train | Epoch 11 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.110E-01 | ce 0.287 | ppl 1.337
[05-21 21:34:40][flashy.solver][INFO] - Train | Epoch 11 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.952E-01 | ce 0.292 | ppl 1.343
[05-21 21:50:48][flashy.solver][INFO] - Train | Epoch 11 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.706E-01 | ce 0.283 | ppl 1.330
[05-21 22:07:00][flashy.solver][INFO] - Train | Epoch 11 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.678E-01 | ce 0.292 | ppl 1.343
[05-21 22:23:11][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[05-21 22:23:48][flashy.solver][INFO] - Valid | Epoch 11 | 7/79 | 0.21 it/sec | ce 0.262 | ppl 1.302
[05-21 22:24:21][flashy.solver][INFO] - Valid | Epoch 11 | 14/79 | 0.21 it/sec | ce 0.244 | ppl 1.280
[05-21 22:24:54][flashy.solver][INFO] - Valid | Epoch 11 | 21/79 | 0.21 it/sec | ce 0.289 | ppl 1.337
[05-21 22:25:26][flashy.solver][INFO] - Valid | Epoch 11 | 28/79 | 0.21 it/sec | ce 0.249 | ppl 1.288
[05-21 22:25:59][flashy.solver][INFO] - Valid | Epoch 11 | 35/79 | 0.21 it/sec | ce 0.287 | ppl 1.335
[05-21 22:26:32][flashy.solver][INFO] - Valid | Epoch 11 | 42/79 | 0.21 it/sec | ce 0.249 | ppl 1.285
[05-21 22:27:05][flashy.solver][INFO] - Valid | Epoch 11 | 49/79 | 0.21 it/sec | ce 0.333 | ppl 1.400
[05-21 22:27:38][flashy.solver][INFO] - Valid | Epoch 11 | 56/79 | 0.21 it/sec | ce 0.248 | ppl 1.284
[05-21 22:28:11][flashy.solver][INFO] - Valid | Epoch 11 | 63/79 | 0.21 it/sec | ce 0.315 | ppl 1.372
[05-21 22:28:44][flashy.solver][INFO] - Valid | Epoch 11 | 70/79 | 0.21 it/sec | ce 0.291 | ppl 1.339
[05-21 22:29:17][flashy.solver][INFO] - Valid | Epoch 11 | 77/79 | 0.21 it/sec | ce 0.334 | ppl 1.398
[05-21 22:29:18][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[05-21 22:29:18][flashy.solver][INFO] - New best state with ce=0.281 (was 0.286)
[05-21 22:29:18][flashy.solver][INFO] - Model hash: 97b4d8142da4e52ef60746b5d2db3c14aa8bc7e4
[05-21 22:29:18][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-21 22:45:34][flashy.solver][INFO] - Train | Epoch 12 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.101E-01 | ce 0.287 | ppl 1.336
[05-21 23:01:42][flashy.solver][INFO] - Train | Epoch 12 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.541E-01 | ce 0.290 | ppl 1.340
[05-21 23:17:52][flashy.solver][INFO] - Train | Epoch 12 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.043E-01 | ce 0.284 | ppl 1.332
[05-21 23:34:03][flashy.solver][INFO] - Train | Epoch 12 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.158E-01 | ce 0.291 | ppl 1.340
[05-21 23:50:16][flashy.solver][INFO] - Train | Epoch 12 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.221E-01 | ce 0.276 | ppl 1.322
[05-22 00:06:33][flashy.solver][INFO] - Train | Epoch 12 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.672E-01 | ce 0.287 | ppl 1.336
[05-22 00:22:45][flashy.solver][INFO] - Train | Epoch 12 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.929E-01 | ce 0.295 | ppl 1.346
[05-22 00:38:56][flashy.solver][INFO] - Train | Epoch 12 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.199E-01 | ce 0.287 | ppl 1.336
[05-22 00:55:06][flashy.solver][INFO] - Train | Epoch 12 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.070E-01 | ce 0.298 | ppl 1.351
[05-22 01:11:16][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[05-22 01:11:54][flashy.solver][INFO] - Valid | Epoch 12 | 7/79 | 0.21 it/sec | ce 0.260 | ppl 1.299
[05-22 01:12:27][flashy.solver][INFO] - Valid | Epoch 12 | 14/79 | 0.21 it/sec | ce 0.242 | ppl 1.277
[05-22 01:13:00][flashy.solver][INFO] - Valid | Epoch 12 | 21/79 | 0.21 it/sec | ce 0.287 | ppl 1.334
[05-22 01:13:33][flashy.solver][INFO] - Valid | Epoch 12 | 28/79 | 0.21 it/sec | ce 0.247 | ppl 1.285
[05-22 01:14:06][flashy.solver][INFO] - Valid | Epoch 12 | 35/79 | 0.21 it/sec | ce 0.284 | ppl 1.331
[05-22 01:14:39][flashy.solver][INFO] - Valid | Epoch 12 | 42/79 | 0.21 it/sec | ce 0.247 | ppl 1.283
[05-22 01:15:13][flashy.solver][INFO] - Valid | Epoch 12 | 49/79 | 0.21 it/sec | ce 0.330 | ppl 1.396
[05-22 01:15:46][flashy.solver][INFO] - Valid | Epoch 12 | 56/79 | 0.21 it/sec | ce 0.246 | ppl 1.281
[05-22 01:16:19][flashy.solver][INFO] - Valid | Epoch 12 | 63/79 | 0.21 it/sec | ce 0.313 | ppl 1.368
[05-22 01:16:52][flashy.solver][INFO] - Valid | Epoch 12 | 70/79 | 0.21 it/sec | ce 0.288 | ppl 1.336
[05-22 01:17:26][flashy.solver][INFO] - Valid | Epoch 12 | 77/79 | 0.21 it/sec | ce 0.331 | ppl 1.394
[05-22 01:17:26][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[05-22 01:17:26][flashy.solver][INFO] - New best state with ce=0.279 (was 0.281)
[05-22 01:17:26][flashy.solver][INFO] - Model hash: 8e859b5663731da044049357fc59c9fbe8478383
[05-22 01:17:26][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 01:33:51][flashy.solver][INFO] - Train | Epoch 13 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 5.614E-01 | ce 0.300 | ppl 1.353
[05-22 01:50:04][flashy.solver][INFO] - Train | Epoch 13 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.685E-01 | ce 0.289 | ppl 1.339
[05-22 02:06:14][flashy.solver][INFO] - Train | Epoch 13 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.159E-01 | ce 0.285 | ppl 1.334
[05-22 02:22:27][flashy.solver][INFO] - Train | Epoch 13 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.942E-01 | ce 0.283 | ppl 1.331
[05-22 02:38:38][flashy.solver][INFO] - Train | Epoch 13 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.070E-01 | ce 0.284 | ppl 1.332
[05-22 02:54:56][flashy.solver][INFO] - Train | Epoch 13 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.400E-01 | ce 0.281 | ppl 1.327
[05-22 03:11:06][flashy.solver][INFO] - Train | Epoch 13 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.686E-01 | ce 0.288 | ppl 1.337
[05-22 03:27:19][flashy.solver][INFO] - Train | Epoch 13 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.754E-01 | ce 0.281 | ppl 1.328
[05-22 03:43:29][flashy.solver][INFO] - Train | Epoch 13 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.210E-01 | ce 0.283 | ppl 1.330
[05-22 03:59:41][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[05-22 04:00:18][flashy.solver][INFO] - Valid | Epoch 13 | 7/79 | 0.21 it/sec | ce 0.255 | ppl 1.293
[05-22 04:00:51][flashy.solver][INFO] - Valid | Epoch 13 | 14/79 | 0.21 it/sec | ce 0.237 | ppl 1.271
[05-22 04:01:24][flashy.solver][INFO] - Valid | Epoch 13 | 21/79 | 0.21 it/sec | ce 0.282 | ppl 1.328
[05-22 04:01:57][flashy.solver][INFO] - Valid | Epoch 13 | 28/79 | 0.21 it/sec | ce 0.243 | ppl 1.279
[05-22 04:02:30][flashy.solver][INFO] - Valid | Epoch 13 | 35/79 | 0.21 it/sec | ce 0.280 | ppl 1.324
[05-22 04:03:02][flashy.solver][INFO] - Valid | Epoch 13 | 42/79 | 0.21 it/sec | ce 0.242 | ppl 1.277
[05-22 04:03:36][flashy.solver][INFO] - Valid | Epoch 13 | 49/79 | 0.21 it/sec | ce 0.325 | ppl 1.388
[05-22 04:04:09][flashy.solver][INFO] - Valid | Epoch 13 | 56/79 | 0.21 it/sec | ce 0.242 | ppl 1.276
[05-22 04:04:42][flashy.solver][INFO] - Valid | Epoch 13 | 63/79 | 0.21 it/sec | ce 0.307 | ppl 1.361
[05-22 04:05:15][flashy.solver][INFO] - Valid | Epoch 13 | 70/79 | 0.21 it/sec | ce 0.283 | ppl 1.329
[05-22 04:05:48][flashy.solver][INFO] - Valid | Epoch 13 | 77/79 | 0.21 it/sec | ce 0.325 | ppl 1.386
[05-22 04:05:49][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[05-22 04:05:49][flashy.solver][INFO] - New best state with ce=0.274 (was 0.279)
[05-22 04:05:49][flashy.solver][INFO] - Model hash: 3a16e72fdeb5cd8a6a5eb5072b9d7565ebd66fd4
[05-22 04:05:49][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 04:22:04][flashy.solver][INFO] - Train | Epoch 14 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.586E-01 | ce 0.286 | ppl 1.334
[05-22 04:38:15][flashy.solver][INFO] - Train | Epoch 14 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.789E-01 | ce 0.277 | ppl 1.322
[05-22 04:54:27][flashy.solver][INFO] - Train | Epoch 14 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.153E-01 | ce 0.289 | ppl 1.339
[05-22 05:10:39][flashy.solver][INFO] - Train | Epoch 14 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.577E-01 | ce 0.288 | ppl 1.337
[05-22 05:26:48][flashy.solver][INFO] - Train | Epoch 14 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.580E-01 | ce 0.265 | ppl 1.307
[05-22 05:43:01][flashy.solver][INFO] - Train | Epoch 14 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.657E-01 | ce 0.277 | ppl 1.323
[05-22 05:59:14][flashy.solver][INFO] - Train | Epoch 14 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.662E-01 | ce 0.284 | ppl 1.332
[05-22 06:15:26][flashy.solver][INFO] - Train | Epoch 14 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.806E-01 | ce 0.282 | ppl 1.330
[05-22 06:31:43][flashy.solver][INFO] - Train | Epoch 14 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.182E-01 | ce 0.284 | ppl 1.332
[05-22 06:47:56][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[05-22 06:48:34][flashy.solver][INFO] - Valid | Epoch 14 | 7/79 | 0.21 it/sec | ce 0.252 | ppl 1.289
[05-22 06:49:06][flashy.solver][INFO] - Valid | Epoch 14 | 14/79 | 0.21 it/sec | ce 0.235 | ppl 1.268
[05-22 06:49:40][flashy.solver][INFO] - Valid | Epoch 14 | 21/79 | 0.21 it/sec | ce 0.279 | ppl 1.324
[05-22 06:50:12][flashy.solver][INFO] - Valid | Epoch 14 | 28/79 | 0.21 it/sec | ce 0.240 | ppl 1.276
[05-22 06:50:45][flashy.solver][INFO] - Valid | Epoch 14 | 35/79 | 0.21 it/sec | ce 0.276 | ppl 1.320
[05-22 06:51:18][flashy.solver][INFO] - Valid | Epoch 14 | 42/79 | 0.21 it/sec | ce 0.240 | ppl 1.273
[05-22 06:51:52][flashy.solver][INFO] - Valid | Epoch 14 | 49/79 | 0.21 it/sec | ce 0.321 | ppl 1.383
[05-22 06:52:25][flashy.solver][INFO] - Valid | Epoch 14 | 56/79 | 0.21 it/sec | ce 0.240 | ppl 1.273
[05-22 06:52:58][flashy.solver][INFO] - Valid | Epoch 14 | 63/79 | 0.21 it/sec | ce 0.304 | ppl 1.357
[05-22 06:53:31][flashy.solver][INFO] - Valid | Epoch 14 | 70/79 | 0.21 it/sec | ce 0.280 | ppl 1.325
[05-22 06:54:05][flashy.solver][INFO] - Valid | Epoch 14 | 77/79 | 0.21 it/sec | ce 0.322 | ppl 1.382
[05-22 06:54:05][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[05-22 06:54:05][flashy.solver][INFO] - New best state with ce=0.271 (was 0.274)
[05-22 06:54:05][flashy.solver][INFO] - Model hash: bf4b97ba7fce945ab0290e77cf94536485b04f37
[05-22 06:54:05][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 07:10:28][flashy.solver][INFO] - Train | Epoch 15 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 5.338E-01 | ce 0.281 | ppl 1.327
[05-22 07:26:44][flashy.solver][INFO] - Train | Epoch 15 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.345E-01 | ce 0.277 | ppl 1.323
[05-22 07:40:31][dora.distrib][INFO] - world_size is 1, skipping init.
[05-22 07:40:31][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-22 07:40:31][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-22 07:40:31][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-22 07:40:31][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-22 07:40:31][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-22 07:40:31][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-22 07:40:31][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-22 07:40:32][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-22 07:40:32][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-22 07:40:33][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-22 07:40:33][flashy.solver][INFO] - No EMA on the model.
[05-22 07:40:33][flashy.solver][INFO] - Model size: 1.21 M params
[05-22 07:40:33][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-22 07:40:33][flashy.solver][INFO] - Restoring weights and history.
[05-22 07:40:33][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 07:40:33][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 07:40:33][flashy.solver][INFO] - Model hash: bf4b97ba7fce945ab0290e77cf94536485b04f37
[05-22 07:40:33][flashy.solver][INFO] - Replaying past metrics...
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[05-22 07:40:33][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[05-22 07:40:33][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[05-22 07:40:39][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-22 07:56:39][flashy.solver][INFO] - Train | Epoch 15 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.883E-01 | ce 0.277 | ppl 1.322
[05-22 08:12:37][flashy.solver][INFO] - Train | Epoch 15 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.129E-01 | ce 0.277 | ppl 1.322
[05-22 08:28:35][flashy.solver][INFO] - Train | Epoch 15 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.421E-01 | ce 0.284 | ppl 1.332
[05-22 08:44:33][flashy.solver][INFO] - Train | Epoch 15 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.684E-01 | ce 0.278 | ppl 1.324
[05-22 09:00:32][flashy.solver][INFO] - Train | Epoch 15 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.643E-01 | ce 0.280 | ppl 1.326
[05-22 09:16:32][flashy.solver][INFO] - Train | Epoch 15 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.420E-01 | ce 0.284 | ppl 1.332
[05-22 09:32:31][flashy.solver][INFO] - Train | Epoch 15 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.525E-01 | ce 0.274 | ppl 1.318
[05-22 09:48:30][flashy.solver][INFO] - Train | Epoch 15 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.923E-01 | ce 0.280 | ppl 1.325
[05-22 10:04:26][flashy.solver][INFO] - Train | Epoch 15 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.478E-01 | ce 0.275 | ppl 1.319
[05-22 10:20:23][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[05-22 10:21:00][flashy.solver][INFO] - Valid | Epoch 15 | 7/79 | 0.22 it/sec | ce 0.249 | ppl 1.285
[05-22 10:21:32][flashy.solver][INFO] - Valid | Epoch 15 | 14/79 | 0.22 it/sec | ce 0.232 | ppl 1.264
[05-22 10:22:05][flashy.solver][INFO] - Valid | Epoch 15 | 21/79 | 0.22 it/sec | ce 0.275 | ppl 1.319
[05-22 10:22:37][flashy.solver][INFO] - Valid | Epoch 15 | 28/79 | 0.22 it/sec | ce 0.237 | ppl 1.272
[05-22 10:23:09][flashy.solver][INFO] - Valid | Epoch 15 | 35/79 | 0.22 it/sec | ce 0.273 | ppl 1.316
[05-22 10:23:42][flashy.solver][INFO] - Valid | Epoch 15 | 42/79 | 0.22 it/sec | ce 0.237 | ppl 1.270
[05-22 10:24:15][flashy.solver][INFO] - Valid | Epoch 15 | 49/79 | 0.22 it/sec | ce 0.317 | ppl 1.378
[05-22 10:24:47][flashy.solver][INFO] - Valid | Epoch 15 | 56/79 | 0.22 it/sec | ce 0.237 | ppl 1.269
[05-22 10:25:20][flashy.solver][INFO] - Valid | Epoch 15 | 63/79 | 0.22 it/sec | ce 0.301 | ppl 1.352
[05-22 10:25:52][flashy.solver][INFO] - Valid | Epoch 15 | 70/79 | 0.22 it/sec | ce 0.277 | ppl 1.321
[05-22 10:26:25][flashy.solver][INFO] - Valid | Epoch 15 | 77/79 | 0.22 it/sec | ce 0.318 | ppl 1.376
[05-22 10:26:26][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[05-22 10:26:26][flashy.solver][INFO] - New best state with ce=0.268 (was 0.271)
[05-22 10:26:26][flashy.solver][INFO] - Model hash: 3f79780266b8620cc1da852510d72dbf4e0ce72c
[05-22 10:26:26][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 10:42:33][flashy.solver][INFO] - Train | Epoch 16 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.341E-01 | ce 0.269 | ppl 1.312
[05-22 10:58:34][flashy.solver][INFO] - Train | Epoch 16 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.332E-01 | ce 0.270 | ppl 1.312
[05-22 11:14:33][flashy.solver][INFO] - Train | Epoch 16 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.470E-01 | ce 0.279 | ppl 1.325
[05-22 11:30:29][flashy.solver][INFO] - Train | Epoch 16 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.297E-01 | ce 0.282 | ppl 1.329
[05-22 11:46:28][flashy.solver][INFO] - Train | Epoch 16 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.582E-01 | ce 0.275 | ppl 1.320
[05-22 12:02:28][flashy.solver][INFO] - Train | Epoch 16 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.747E-01 | ce 0.274 | ppl 1.318
[05-22 12:18:27][flashy.solver][INFO] - Train | Epoch 16 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.418E-01 | ce 0.273 | ppl 1.316
[05-22 12:34:27][flashy.solver][INFO] - Train | Epoch 16 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.132E-01 | ce 0.278 | ppl 1.323
[05-22 12:50:29][flashy.solver][INFO] - Train | Epoch 16 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.757E-01 | ce 0.275 | ppl 1.321
[05-22 13:03:20][dora.distrib][INFO] - world_size is 1, skipping init.
[05-22 13:03:20][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-22 13:03:20][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-22 13:03:20][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-22 13:03:20][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-22 13:03:20][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-22 13:03:20][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-22 13:03:20][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-22 13:03:21][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-22 13:03:21][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-22 13:03:22][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-22 13:03:22][flashy.solver][INFO] - No EMA on the model.
[05-22 13:03:22][flashy.solver][INFO] - Model size: 1.21 M params
[05-22 13:03:22][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-22 13:03:22][flashy.solver][INFO] - Restoring weights and history.
[05-22 13:03:22][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 13:03:22][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 13:03:22][flashy.solver][INFO] - Model hash: 3f79780266b8620cc1da852510d72dbf4e0ce72c
[05-22 13:03:22][flashy.solver][INFO] - Replaying past metrics...
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[05-22 13:03:22][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[05-22 13:03:22][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[05-22 13:03:27][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-22 13:19:22][flashy.solver][INFO] - Train | Epoch 16 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.357E-01 | ce 0.279 | ppl 1.325
[05-22 13:35:17][flashy.solver][INFO] - Train | Epoch 16 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.085E-01 | ce 0.275 | ppl 1.319
[05-22 13:51:08][flashy.solver][INFO] - Train | Epoch 16 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.266E-01 | ce 0.280 | ppl 1.326
[05-22 14:07:02][flashy.solver][INFO] - Train | Epoch 16 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.854E-01 | ce 0.268 | ppl 1.311
[05-22 14:22:59][flashy.solver][INFO] - Train | Epoch 16 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.835E-01 | ce 0.273 | ppl 1.316
[05-22 14:38:57][flashy.solver][INFO] - Train | Epoch 16 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.069E-01 | ce 0.278 | ppl 1.324
[05-22 14:54:49][flashy.solver][INFO] - Train | Epoch 16 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.443E-01 | ce 0.271 | ppl 1.315
[05-22 15:10:43][flashy.solver][INFO] - Train | Epoch 16 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.886E-01 | ce 0.272 | ppl 1.316
[05-22 15:26:38][flashy.solver][INFO] - Train | Epoch 16 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.349E-01 | ce 0.271 | ppl 1.315
[05-22 15:42:35][flashy.solver][INFO] - Train Summary | Epoch 16 | lr=1.00E-04 | grad_norm=5.373E-01 | ce=0.274 | ppl=1.318 | duration=9552.291
[05-22 15:43:12][flashy.solver][INFO] - Valid | Epoch 16 | 7/79 | 0.22 it/sec | ce 0.247 | ppl 1.282
[05-22 15:43:44][flashy.solver][INFO] - Valid | Epoch 16 | 14/79 | 0.22 it/sec | ce 0.230 | ppl 1.261
[05-22 15:44:16][flashy.solver][INFO] - Valid | Epoch 16 | 21/79 | 0.22 it/sec | ce 0.273 | ppl 1.316
[05-22 15:44:48][flashy.solver][INFO] - Valid | Epoch 16 | 28/79 | 0.22 it/sec | ce 0.235 | ppl 1.269
[05-22 15:45:21][flashy.solver][INFO] - Valid | Epoch 16 | 35/79 | 0.22 it/sec | ce 0.271 | ppl 1.312
[05-22 15:45:53][flashy.solver][INFO] - Valid | Epoch 16 | 42/79 | 0.22 it/sec | ce 0.235 | ppl 1.267
[05-22 15:46:26][flashy.solver][INFO] - Valid | Epoch 16 | 49/79 | 0.22 it/sec | ce 0.314 | ppl 1.374
[05-22 15:46:58][flashy.solver][INFO] - Valid | Epoch 16 | 56/79 | 0.22 it/sec | ce 0.235 | ppl 1.266
[05-22 15:47:31][flashy.solver][INFO] - Valid | Epoch 16 | 63/79 | 0.22 it/sec | ce 0.298 | ppl 1.348
[05-22 15:48:03][flashy.solver][INFO] - Valid | Epoch 16 | 70/79 | 0.22 it/sec | ce 0.274 | ppl 1.317
[05-22 15:48:36][flashy.solver][INFO] - Valid | Epoch 16 | 77/79 | 0.22 it/sec | ce 0.315 | ppl 1.372
[05-22 15:48:37][flashy.solver][INFO] - Valid Summary | Epoch 16 | ce=0.265 | ppl=1.306 | duration=362.290
[05-22 15:48:37][flashy.solver][INFO] - New best state with ce=0.265 (was 0.268)
[05-22 15:48:37][flashy.solver][INFO] - Model hash: 639ed31df5e3c257cca9e00f15cee3271fefc016
[05-22 15:48:37][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 16:04:41][flashy.solver][INFO] - Train | Epoch 17 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.276E-01 | ce 0.272 | ppl 1.316
[05-22 16:20:43][flashy.solver][INFO] - Train | Epoch 17 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.633E-01 | ce 0.278 | ppl 1.324
[05-22 16:36:41][flashy.solver][INFO] - Train | Epoch 17 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.241E-01 | ce 0.279 | ppl 1.324
[05-22 16:52:41][flashy.solver][INFO] - Train | Epoch 17 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.097E-01 | ce 0.277 | ppl 1.323
[05-22 17:08:41][flashy.solver][INFO] - Train | Epoch 17 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.662E-01 | ce 0.276 | ppl 1.321
[05-22 17:24:40][flashy.solver][INFO] - Train | Epoch 17 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.625E-01 | ce 0.270 | ppl 1.313
[05-22 17:40:40][flashy.solver][INFO] - Train | Epoch 17 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.087E-01 | ce 0.272 | ppl 1.317
[05-22 17:56:37][flashy.solver][INFO] - Train | Epoch 17 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.635E-01 | ce 0.277 | ppl 1.323
[05-22 18:12:35][flashy.solver][INFO] - Train | Epoch 17 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.819E-01 | ce 0.264 | ppl 1.306
[05-22 18:28:30][flashy.solver][INFO] - Train Summary | Epoch 17 | lr=1.00E-04 | grad_norm=5.495E-01 | ce=0.274 | ppl=1.318 | duration=9593.252
[05-22 18:29:07][flashy.solver][INFO] - Valid | Epoch 17 | 7/79 | 0.22 it/sec | ce 0.245 | ppl 1.280
[05-22 18:29:39][flashy.solver][INFO] - Valid | Epoch 17 | 14/79 | 0.22 it/sec | ce 0.228 | ppl 1.259
[05-22 18:30:12][flashy.solver][INFO] - Valid | Epoch 17 | 21/79 | 0.22 it/sec | ce 0.271 | ppl 1.313
[05-22 18:30:44][flashy.solver][INFO] - Valid | Epoch 17 | 28/79 | 0.22 it/sec | ce 0.233 | ppl 1.266
[05-22 18:31:16][flashy.solver][INFO] - Valid | Epoch 17 | 35/79 | 0.22 it/sec | ce 0.268 | ppl 1.309
[05-22 18:31:48][flashy.solver][INFO] - Valid | Epoch 17 | 42/79 | 0.22 it/sec | ce 0.232 | ppl 1.264
[05-22 18:32:21][flashy.solver][INFO] - Valid | Epoch 17 | 49/79 | 0.22 it/sec | ce 0.312 | ppl 1.370
[05-22 18:32:53][flashy.solver][INFO] - Valid | Epoch 17 | 56/79 | 0.22 it/sec | ce 0.233 | ppl 1.264
[05-22 18:33:26][flashy.solver][INFO] - Valid | Epoch 17 | 63/79 | 0.22 it/sec | ce 0.295 | ppl 1.344
[05-22 18:33:58][flashy.solver][INFO] - Valid | Epoch 17 | 70/79 | 0.22 it/sec | ce 0.272 | ppl 1.315
[05-22 18:34:31][flashy.solver][INFO] - Valid | Epoch 17 | 77/79 | 0.22 it/sec | ce 0.313 | ppl 1.369
[05-22 18:34:32][flashy.solver][INFO] - Valid Summary | Epoch 17 | ce=0.263 | ppl=1.303 | duration=361.603
[05-22 18:34:32][flashy.solver][INFO] - New best state with ce=0.263 (was 0.265)
[05-22 18:34:32][flashy.solver][INFO] - Model hash: 708f28b61cb78f4b2b9a9b5139fb6a3c79a7ea23
[05-22 18:34:32][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 18:50:34][flashy.solver][INFO] - Train | Epoch 18 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.135E-01 | ce 0.273 | ppl 1.317
[05-22 19:06:32][flashy.solver][INFO] - Train | Epoch 18 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.591E-01 | ce 0.273 | ppl 1.317
[05-22 19:22:34][flashy.solver][INFO] - Train | Epoch 18 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.601E-01 | ce 0.277 | ppl 1.323
[05-22 19:38:32][flashy.solver][INFO] - Train | Epoch 18 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.587E-01 | ce 0.263 | ppl 1.304
[05-22 19:54:33][flashy.solver][INFO] - Train | Epoch 18 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.813E-01 | ce 0.264 | ppl 1.305
[05-22 20:10:34][flashy.solver][INFO] - Train | Epoch 18 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.083E-01 | ce 0.271 | ppl 1.315
[05-22 20:26:32][flashy.solver][INFO] - Train | Epoch 18 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.004E-01 | ce 0.269 | ppl 1.311
[05-22 20:42:32][flashy.solver][INFO] - Train | Epoch 18 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.998E-01 | ce 0.275 | ppl 1.320
[05-22 20:58:32][flashy.solver][INFO] - Train | Epoch 18 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.307E-01 | ce 0.269 | ppl 1.311
[05-22 21:14:34][flashy.solver][INFO] - Train Summary | Epoch 18 | lr=1.00E-04 | grad_norm=5.303E-01 | ce=0.270 | ppl=1.314 | duration=9601.733
[05-22 21:15:11][flashy.solver][INFO] - Valid | Epoch 18 | 7/79 | 0.22 it/sec | ce 0.243 | ppl 1.277
[05-22 21:15:43][flashy.solver][INFO] - Valid | Epoch 18 | 14/79 | 0.22 it/sec | ce 0.226 | ppl 1.257
[05-22 21:16:16][flashy.solver][INFO] - Valid | Epoch 18 | 21/79 | 0.22 it/sec | ce 0.269 | ppl 1.310
[05-22 21:16:48][flashy.solver][INFO] - Valid | Epoch 18 | 28/79 | 0.22 it/sec | ce 0.231 | ppl 1.264
[05-22 21:17:20][flashy.solver][INFO] - Valid | Epoch 18 | 35/79 | 0.22 it/sec | ce 0.266 | ppl 1.306
[05-22 21:17:52][flashy.solver][INFO] - Valid | Epoch 18 | 42/79 | 0.22 it/sec | ce 0.231 | ppl 1.262
[05-22 21:18:25][flashy.solver][INFO] - Valid | Epoch 18 | 49/79 | 0.22 it/sec | ce 0.309 | ppl 1.367
[05-22 21:18:58][flashy.solver][INFO] - Valid | Epoch 18 | 56/79 | 0.22 it/sec | ce 0.231 | ppl 1.262
[05-22 21:19:31][flashy.solver][INFO] - Valid | Epoch 18 | 63/79 | 0.22 it/sec | ce 0.293 | ppl 1.341
[05-22 21:20:03][flashy.solver][INFO] - Valid | Epoch 18 | 70/79 | 0.22 it/sec | ce 0.270 | ppl 1.311
[05-22 21:20:36][flashy.solver][INFO] - Valid | Epoch 18 | 77/79 | 0.22 it/sec | ce 0.310 | ppl 1.365
[05-22 21:20:36][flashy.solver][INFO] - Valid Summary | Epoch 18 | ce=0.261 | ppl=1.301 | duration=362.833
[05-22 21:20:36][flashy.solver][INFO] - New best state with ce=0.261 (was 0.263)
[05-22 21:20:36][flashy.solver][INFO] - Model hash: 7bec5ee8ef97ef9dc39fb7eeb592925edb9d992a
[05-22 21:20:36][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-22 21:36:41][flashy.solver][INFO] - Train | Epoch 19 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.491E-01 | ce 0.269 | ppl 1.311
[05-22 21:52:38][flashy.solver][INFO] - Train | Epoch 19 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.226E-01 | ce 0.276 | ppl 1.321
[05-22 22:08:37][flashy.solver][INFO] - Train | Epoch 19 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.389E-01 | ce 0.264 | ppl 1.305
[05-22 22:24:36][flashy.solver][INFO] - Train | Epoch 19 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.898E-01 | ce 0.270 | ppl 1.313
[05-22 22:40:35][flashy.solver][INFO] - Train | Epoch 19 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.907E-01 | ce 0.271 | ppl 1.314
[05-22 22:56:37][flashy.solver][INFO] - Train | Epoch 19 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.946E-01 | ce 0.267 | ppl 1.309
[05-22 23:12:37][flashy.solver][INFO] - Train | Epoch 19 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.817E-01 | ce 0.275 | ppl 1.319
[05-22 23:28:37][flashy.solver][INFO] - Train | Epoch 19 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.886E-01 | ce 0.269 | ppl 1.311
[05-22 23:44:37][flashy.solver][INFO] - Train | Epoch 19 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.483E-01 | ce 0.273 | ppl 1.317
[05-23 00:00:40][flashy.solver][INFO] - Train Summary | Epoch 19 | lr=1.00E-04 | grad_norm=5.436E-01 | ce=0.271 | ppl=1.314 | duration=9603.793
[05-23 00:01:17][flashy.solver][INFO] - Valid | Epoch 19 | 7/79 | 0.22 it/sec | ce 0.241 | ppl 1.275
[05-23 00:01:49][flashy.solver][INFO] - Valid | Epoch 19 | 14/79 | 0.22 it/sec | ce 0.224 | ppl 1.254
[05-23 00:02:22][flashy.solver][INFO] - Valid | Epoch 19 | 21/79 | 0.22 it/sec | ce 0.266 | ppl 1.307
[05-23 00:02:54][flashy.solver][INFO] - Valid | Epoch 19 | 28/79 | 0.22 it/sec | ce 0.229 | ppl 1.261
[05-23 00:03:27][flashy.solver][INFO] - Valid | Epoch 19 | 35/79 | 0.22 it/sec | ce 0.264 | ppl 1.304
[05-23 00:03:59][flashy.solver][INFO] - Valid | Epoch 19 | 42/79 | 0.22 it/sec | ce 0.229 | ppl 1.259
[05-23 00:04:32][flashy.solver][INFO] - Valid | Epoch 19 | 49/79 | 0.22 it/sec | ce 0.307 | ppl 1.363
[05-23 00:05:05][flashy.solver][INFO] - Valid | Epoch 19 | 56/79 | 0.22 it/sec | ce 0.229 | ppl 1.259
[05-23 00:05:37][flashy.solver][INFO] - Valid | Epoch 19 | 63/79 | 0.22 it/sec | ce 0.290 | ppl 1.338
[05-23 00:06:10][flashy.solver][INFO] - Valid | Epoch 19 | 70/79 | 0.22 it/sec | ce 0.268 | ppl 1.309
[05-23 00:06:42][flashy.solver][INFO] - Valid | Epoch 19 | 77/79 | 0.22 it/sec | ce 0.308 | ppl 1.362
[05-23 00:06:43][flashy.solver][INFO] - Valid Summary | Epoch 19 | ce=0.259 | ppl=1.298 | duration=362.807
[05-23 00:06:43][flashy.solver][INFO] - New best state with ce=0.259 (was 0.261)
[05-23 00:06:43][flashy.solver][INFO] - Model hash: 20af2e80541621139b1ab9715c10eea1d3ed16d8
[05-23 00:06:43][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 00:22:51][flashy.solver][INFO] - Train | Epoch 20 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.489E-01 | ce 0.276 | ppl 1.321
[05-23 00:38:52][flashy.solver][INFO] - Train | Epoch 20 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.862E-01 | ce 0.263 | ppl 1.303
[05-23 00:54:51][flashy.solver][INFO] - Train | Epoch 20 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.351E-01 | ce 0.263 | ppl 1.304
[05-23 01:10:47][flashy.solver][INFO] - Train | Epoch 20 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.339E-01 | ce 0.262 | ppl 1.302
[05-23 01:26:49][flashy.solver][INFO] - Train | Epoch 20 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.956E-01 | ce 0.264 | ppl 1.305
[05-23 01:28:49][dora.distrib][INFO] - world_size is 1, skipping init.
[05-23 01:28:49][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-23 01:28:49][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-23 01:28:49][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-23 01:28:49][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-23 01:28:49][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-23 01:28:49][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-23 01:28:49][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-23 01:28:50][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-23 01:28:50][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-23 01:28:51][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-23 01:28:51][flashy.solver][INFO] - No EMA on the model.
[05-23 01:28:51][flashy.solver][INFO] - Model size: 1.21 M params
[05-23 01:28:51][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-23 01:28:51][flashy.solver][INFO] - Restoring weights and history.
[05-23 01:28:51][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 01:28:51][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 01:28:51][flashy.solver][INFO] - Model hash: 20af2e80541621139b1ab9715c10eea1d3ed16d8
[05-23 01:28:51][flashy.solver][INFO] - Replaying past metrics...
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 16 | lr=1.00E-04 | grad_norm=5.373E-01 | ce=0.274 | ppl=1.318 | duration=9552.291
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 16 | ce=0.265 | ppl=1.306 | duration=362.290
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 17 | lr=1.00E-04 | grad_norm=5.495E-01 | ce=0.274 | ppl=1.318 | duration=9593.252
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 17 | ce=0.263 | ppl=1.303 | duration=361.603
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 18 | lr=1.00E-04 | grad_norm=5.303E-01 | ce=0.270 | ppl=1.314 | duration=9601.733
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 18 | ce=0.261 | ppl=1.301 | duration=362.833
[05-23 01:28:51][flashy.solver][INFO] - Train Summary | Epoch 19 | lr=1.00E-04 | grad_norm=5.436E-01 | ce=0.271 | ppl=1.314 | duration=9603.793
[05-23 01:28:51][flashy.solver][INFO] - Valid Summary | Epoch 19 | ce=0.259 | ppl=1.298 | duration=362.807
[05-23 01:28:56][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-23 01:44:49][flashy.solver][INFO] - Train | Epoch 20 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.556E-01 | ce 0.272 | ppl 1.315
[05-23 02:00:43][flashy.solver][INFO] - Train | Epoch 20 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.949E-01 | ce 0.276 | ppl 1.321
[05-23 02:16:32][flashy.solver][INFO] - Train | Epoch 20 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.526E-01 | ce 0.267 | ppl 1.310
[05-23 02:32:23][flashy.solver][INFO] - Train | Epoch 20 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.722E-01 | ce 0.269 | ppl 1.312
[05-23 02:48:13][flashy.solver][INFO] - Train | Epoch 20 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.967E-01 | ce 0.261 | ppl 1.301
[05-23 03:04:03][flashy.solver][INFO] - Train | Epoch 20 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.716E-01 | ce 0.267 | ppl 1.309
[05-23 03:19:51][flashy.solver][INFO] - Train | Epoch 20 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.737E-01 | ce 0.259 | ppl 1.299
[05-23 03:35:43][flashy.solver][INFO] - Train | Epoch 20 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.274E-01 | ce 0.270 | ppl 1.313
[05-23 03:51:34][flashy.solver][INFO] - Train | Epoch 20 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.992E-01 | ce 0.262 | ppl 1.302
[05-23 04:07:24][flashy.solver][INFO] - Train Summary | Epoch 20 | lr=1.00E-04 | grad_norm=5.256E-01 | ce=0.267 | ppl=1.309 | duration=9513.591
[05-23 04:08:01][flashy.solver][INFO] - Valid | Epoch 20 | 7/79 | 0.22 it/sec | ce 0.239 | ppl 1.272
[05-23 04:08:33][flashy.solver][INFO] - Valid | Epoch 20 | 14/79 | 0.22 it/sec | ce 0.222 | ppl 1.252
[05-23 04:09:05][flashy.solver][INFO] - Valid | Epoch 20 | 21/79 | 0.22 it/sec | ce 0.264 | ppl 1.304
[05-23 04:09:37][flashy.solver][INFO] - Valid | Epoch 20 | 28/79 | 0.22 it/sec | ce 0.227 | ppl 1.259
[05-23 04:10:09][flashy.solver][INFO] - Valid | Epoch 20 | 35/79 | 0.22 it/sec | ce 0.262 | ppl 1.301
[05-23 04:10:41][flashy.solver][INFO] - Valid | Epoch 20 | 42/79 | 0.22 it/sec | ce 0.227 | ppl 1.256
[05-23 04:11:14][flashy.solver][INFO] - Valid | Epoch 20 | 49/79 | 0.22 it/sec | ce 0.304 | ppl 1.360
[05-23 04:11:46][flashy.solver][INFO] - Valid | Epoch 20 | 56/79 | 0.22 it/sec | ce 0.227 | ppl 1.257
[05-23 04:12:18][flashy.solver][INFO] - Valid | Epoch 20 | 63/79 | 0.22 it/sec | ce 0.288 | ppl 1.335
[05-23 04:12:50][flashy.solver][INFO] - Valid | Epoch 20 | 70/79 | 0.22 it/sec | ce 0.266 | ppl 1.306
[05-23 04:13:23][flashy.solver][INFO] - Valid | Epoch 20 | 77/79 | 0.22 it/sec | ce 0.305 | ppl 1.358
[05-23 04:13:24][flashy.solver][INFO] - Valid Summary | Epoch 20 | ce=0.256 | ppl=1.295 | duration=359.164
[05-23 04:13:24][flashy.solver][INFO] - New best state with ce=0.256 (was 0.259)
[05-23 04:13:24][flashy.solver][INFO] - Model hash: ce4ce7ab1097ac9760e0a7c9bdea9df4dfb50e19
[05-23 04:13:24][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 04:29:22][flashy.solver][INFO] - Train | Epoch 21 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.866E-01 | ce 0.265 | ppl 1.306
[05-23 04:45:15][flashy.solver][INFO] - Train | Epoch 21 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.347E-01 | ce 0.270 | ppl 1.313
[05-23 05:01:05][flashy.solver][INFO] - Train | Epoch 21 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.483E-01 | ce 0.265 | ppl 1.306
[05-23 05:16:56][flashy.solver][INFO] - Train | Epoch 21 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.555E-01 | ce 0.264 | ppl 1.305
[05-23 05:32:45][flashy.solver][INFO] - Train | Epoch 21 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.327E-01 | ce 0.263 | ppl 1.304
[05-23 05:48:37][flashy.solver][INFO] - Train | Epoch 21 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.599E-01 | ce 0.274 | ppl 1.319
[05-23 06:04:30][flashy.solver][INFO] - Train | Epoch 21 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.217E-01 | ce 0.265 | ppl 1.307
[05-23 06:20:22][flashy.solver][INFO] - Train | Epoch 21 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.644E-01 | ce 0.268 | ppl 1.311
[05-23 06:36:15][flashy.solver][INFO] - Train | Epoch 21 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.402E-01 | ce 0.270 | ppl 1.314
[05-23 06:52:05][flashy.solver][INFO] - Train Summary | Epoch 21 | lr=1.00E-04 | grad_norm=5.398E-01 | ce=0.267 | ppl=1.309 | duration=9521.433
[05-23 06:52:42][flashy.solver][INFO] - Valid | Epoch 21 | 7/79 | 0.22 it/sec | ce 0.238 | ppl 1.270
[05-23 06:53:14][flashy.solver][INFO] - Valid | Epoch 21 | 14/79 | 0.22 it/sec | ce 0.221 | ppl 1.250
[05-23 06:53:46][flashy.solver][INFO] - Valid | Epoch 21 | 21/79 | 0.22 it/sec | ce 0.263 | ppl 1.303
[05-23 06:54:18][flashy.solver][INFO] - Valid | Epoch 21 | 28/79 | 0.22 it/sec | ce 0.226 | ppl 1.257
[05-23 06:54:50][flashy.solver][INFO] - Valid | Epoch 21 | 35/79 | 0.22 it/sec | ce 0.260 | ppl 1.299
[05-23 06:55:22][flashy.solver][INFO] - Valid | Epoch 21 | 42/79 | 0.22 it/sec | ce 0.225 | ppl 1.255
[05-23 06:55:55][flashy.solver][INFO] - Valid | Epoch 21 | 49/79 | 0.22 it/sec | ce 0.302 | ppl 1.357
[05-23 06:56:27][flashy.solver][INFO] - Valid | Epoch 21 | 56/79 | 0.22 it/sec | ce 0.226 | ppl 1.256
[05-23 06:57:00][flashy.solver][INFO] - Valid | Epoch 21 | 63/79 | 0.22 it/sec | ce 0.286 | ppl 1.333
[05-23 06:57:32][flashy.solver][INFO] - Valid | Epoch 21 | 70/79 | 0.22 it/sec | ce 0.264 | ppl 1.304
[05-23 06:58:04][flashy.solver][INFO] - Valid | Epoch 21 | 77/79 | 0.22 it/sec | ce 0.303 | ppl 1.356
[05-23 06:58:05][flashy.solver][INFO] - Valid Summary | Epoch 21 | ce=0.255 | ppl=1.293 | duration=359.736
[05-23 06:58:05][flashy.solver][INFO] - New best state with ce=0.255 (was 0.256)
[05-23 06:58:05][flashy.solver][INFO] - Model hash: 46e92f064cdc8fcbd64d0a0e606067262b363047
[05-23 06:58:05][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 07:13:59][flashy.solver][INFO] - Train | Epoch 22 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.535E-01 | ce 0.253 | ppl 1.290
[05-23 07:29:52][flashy.solver][INFO] - Train | Epoch 22 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.851E-01 | ce 0.265 | ppl 1.306
[05-23 07:45:43][flashy.solver][INFO] - Train | Epoch 22 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.980E-01 | ce 0.262 | ppl 1.302
[05-23 08:01:35][flashy.solver][INFO] - Train | Epoch 22 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.900E-01 | ce 0.265 | ppl 1.307
[05-23 08:17:27][flashy.solver][INFO] - Train | Epoch 22 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.156E-01 | ce 0.266 | ppl 1.308
[05-23 08:33:22][flashy.solver][INFO] - Train | Epoch 22 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.164E-01 | ce 0.265 | ppl 1.306
[05-23 08:49:13][flashy.solver][INFO] - Train | Epoch 22 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.380E-01 | ce 0.263 | ppl 1.303
[05-23 09:05:03][flashy.solver][INFO] - Train | Epoch 22 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.281E-01 | ce 0.264 | ppl 1.304
[05-23 09:20:56][flashy.solver][INFO] - Train | Epoch 22 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.487E-01 | ce 0.268 | ppl 1.310
[05-23 09:36:48][flashy.solver][INFO] - Train Summary | Epoch 22 | lr=1.00E-04 | grad_norm=5.273E-01 | ce=0.264 | ppl=1.305 | duration=9522.932
[05-23 09:37:25][flashy.solver][INFO] - Valid | Epoch 22 | 7/79 | 0.22 it/sec | ce 0.236 | ppl 1.269
[05-23 09:37:57][flashy.solver][INFO] - Valid | Epoch 22 | 14/79 | 0.22 it/sec | ce 0.219 | ppl 1.248
[05-23 09:38:29][flashy.solver][INFO] - Valid | Epoch 22 | 21/79 | 0.22 it/sec | ce 0.261 | ppl 1.300
[05-23 09:39:01][flashy.solver][INFO] - Valid | Epoch 22 | 28/79 | 0.22 it/sec | ce 0.225 | ppl 1.256
[05-23 09:39:33][flashy.solver][INFO] - Valid | Epoch 22 | 35/79 | 0.22 it/sec | ce 0.259 | ppl 1.297
[05-23 09:40:05][flashy.solver][INFO] - Valid | Epoch 22 | 42/79 | 0.22 it/sec | ce 0.224 | ppl 1.253
[05-23 09:40:38][flashy.solver][INFO] - Valid | Epoch 22 | 49/79 | 0.22 it/sec | ce 0.300 | ppl 1.354
[05-23 09:41:10][flashy.solver][INFO] - Valid | Epoch 22 | 56/79 | 0.22 it/sec | ce 0.225 | ppl 1.254
[05-23 09:41:43][flashy.solver][INFO] - Valid | Epoch 22 | 63/79 | 0.22 it/sec | ce 0.285 | ppl 1.330
[05-23 09:42:15][flashy.solver][INFO] - Valid | Epoch 22 | 70/79 | 0.22 it/sec | ce 0.263 | ppl 1.302
[05-23 09:42:47][flashy.solver][INFO] - Valid | Epoch 22 | 77/79 | 0.22 it/sec | ce 0.302 | ppl 1.353
[05-23 09:42:48][flashy.solver][INFO] - Valid Summary | Epoch 22 | ce=0.254 | ppl=1.291 | duration=359.952
[05-23 09:42:48][flashy.solver][INFO] - New best state with ce=0.254 (was 0.255)
[05-23 09:42:48][flashy.solver][INFO] - Model hash: 47909d4196cf5b0bf3fa5785c45158a4b332d107
[05-23 09:42:48][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 09:58:45][flashy.solver][INFO] - Train | Epoch 23 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.937E-01 | ce 0.274 | ppl 1.318
[05-23 10:14:34][flashy.solver][INFO] - Train | Epoch 23 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.608E-01 | ce 0.261 | ppl 1.301
[05-23 10:30:26][flashy.solver][INFO] - Train | Epoch 23 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.193E-01 | ce 0.266 | ppl 1.307
[05-23 10:46:19][flashy.solver][INFO] - Train | Epoch 23 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.236E-01 | ce 0.259 | ppl 1.298
[05-23 11:02:09][flashy.solver][INFO] - Train | Epoch 23 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.042E-01 | ce 0.255 | ppl 1.294
[05-23 11:18:04][flashy.solver][INFO] - Train | Epoch 23 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.111E-01 | ce 0.262 | ppl 1.302
[05-23 11:33:55][flashy.solver][INFO] - Train | Epoch 23 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.296E-01 | ce 0.259 | ppl 1.298
[05-23 11:49:46][flashy.solver][INFO] - Train | Epoch 23 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.264E-01 | ce 0.266 | ppl 1.308
[05-23 12:05:36][flashy.solver][INFO] - Train | Epoch 23 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.499E-01 | ce 0.262 | ppl 1.302
[05-23 12:21:29][flashy.solver][INFO] - Train Summary | Epoch 23 | lr=1.00E-04 | grad_norm=5.359E-01 | ce=0.263 | ppl=1.304 | duration=9520.848
[05-23 12:22:06][flashy.solver][INFO] - Valid | Epoch 23 | 7/79 | 0.22 it/sec | ce 0.236 | ppl 1.268
[05-23 12:22:37][flashy.solver][INFO] - Valid | Epoch 23 | 14/79 | 0.22 it/sec | ce 0.219 | ppl 1.247
[05-23 12:23:10][flashy.solver][INFO] - Valid | Epoch 23 | 21/79 | 0.22 it/sec | ce 0.260 | ppl 1.299
[05-23 12:23:42][flashy.solver][INFO] - Valid | Epoch 23 | 28/79 | 0.22 it/sec | ce 0.224 | ppl 1.254
[05-23 12:24:14][flashy.solver][INFO] - Valid | Epoch 23 | 35/79 | 0.22 it/sec | ce 0.257 | ppl 1.295
[05-23 12:24:46][flashy.solver][INFO] - Valid | Epoch 23 | 42/79 | 0.22 it/sec | ce 0.223 | ppl 1.251
[05-23 12:25:19][flashy.solver][INFO] - Valid | Epoch 23 | 49/79 | 0.22 it/sec | ce 0.299 | ppl 1.352
[05-23 12:25:51][flashy.solver][INFO] - Valid | Epoch 23 | 56/79 | 0.22 it/sec | ce 0.224 | ppl 1.253
[05-23 12:26:23][flashy.solver][INFO] - Valid | Epoch 23 | 63/79 | 0.22 it/sec | ce 0.283 | ppl 1.328
[05-23 12:26:55][flashy.solver][INFO] - Valid | Epoch 23 | 70/79 | 0.22 it/sec | ce 0.262 | ppl 1.301
[05-23 12:27:28][flashy.solver][INFO] - Valid | Epoch 23 | 77/79 | 0.22 it/sec | ce 0.300 | ppl 1.352
[05-23 12:27:29][flashy.solver][INFO] - Valid Summary | Epoch 23 | ce=0.252 | ppl=1.290 | duration=359.946
[05-23 12:27:29][flashy.solver][INFO] - New best state with ce=0.252 (was 0.254)
[05-23 12:27:29][flashy.solver][INFO] - Model hash: 476031573bcd99cea9ff8abb079832729c98e701
[05-23 12:27:29][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 12:43:25][flashy.solver][INFO] - Train | Epoch 24 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.559E-01 | ce 0.261 | ppl 1.302
[05-23 12:59:17][flashy.solver][INFO] - Train | Epoch 24 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.384E-01 | ce 0.266 | ppl 1.308
[05-23 13:15:08][flashy.solver][INFO] - Train | Epoch 24 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.306E-01 | ce 0.255 | ppl 1.293
[05-23 13:30:58][flashy.solver][INFO] - Train | Epoch 24 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.543E-01 | ce 0.255 | ppl 1.294
[05-23 13:46:52][flashy.solver][INFO] - Train | Epoch 24 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.830E-01 | ce 0.264 | ppl 1.304
[05-23 14:02:44][flashy.solver][INFO] - Train | Epoch 24 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.288E-01 | ce 0.266 | ppl 1.308
[05-23 14:18:34][flashy.solver][INFO] - Train | Epoch 24 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.550E-01 | ce 0.261 | ppl 1.301
[05-23 14:34:25][flashy.solver][INFO] - Train | Epoch 24 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.317E-01 | ce 0.261 | ppl 1.302
[05-23 14:50:16][flashy.solver][INFO] - Train | Epoch 24 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.673E-01 | ce 0.257 | ppl 1.296
[05-23 15:06:06][flashy.solver][INFO] - Train Summary | Epoch 24 | lr=1.00E-04 | grad_norm=5.356E-01 | ce=0.260 | ppl=1.300 | duration=9517.164
[05-23 15:06:43][flashy.solver][INFO] - Valid | Epoch 24 | 7/79 | 0.22 it/sec | ce 0.233 | ppl 1.265
[05-23 15:07:15][flashy.solver][INFO] - Valid | Epoch 24 | 14/79 | 0.22 it/sec | ce 0.216 | ppl 1.244
[05-23 15:07:47][flashy.solver][INFO] - Valid | Epoch 24 | 21/79 | 0.22 it/sec | ce 0.258 | ppl 1.296
[05-23 15:08:19][flashy.solver][INFO] - Valid | Epoch 24 | 28/79 | 0.22 it/sec | ce 0.222 | ppl 1.252
[05-23 15:08:51][flashy.solver][INFO] - Valid | Epoch 24 | 35/79 | 0.22 it/sec | ce 0.255 | ppl 1.292
[05-23 15:09:23][flashy.solver][INFO] - Valid | Epoch 24 | 42/79 | 0.22 it/sec | ce 0.221 | ppl 1.249
[05-23 15:09:56][flashy.solver][INFO] - Valid | Epoch 24 | 49/79 | 0.22 it/sec | ce 0.297 | ppl 1.349
[05-23 15:10:28][flashy.solver][INFO] - Valid | Epoch 24 | 56/79 | 0.22 it/sec | ce 0.222 | ppl 1.250
[05-23 15:11:01][flashy.solver][INFO] - Valid | Epoch 24 | 63/79 | 0.22 it/sec | ce 0.281 | ppl 1.325
[05-23 15:11:33][flashy.solver][INFO] - Valid | Epoch 24 | 70/79 | 0.22 it/sec | ce 0.260 | ppl 1.298
[05-23 15:12:06][flashy.solver][INFO] - Valid | Epoch 24 | 77/79 | 0.22 it/sec | ce 0.298 | ppl 1.348
[05-23 15:12:06][flashy.solver][INFO] - Valid Summary | Epoch 24 | ce=0.250 | ppl=1.287 | duration=360.223
[05-23 15:12:06][flashy.solver][INFO] - New best state with ce=0.250 (was 0.252)
[05-23 15:12:06][flashy.solver][INFO] - Model hash: 0dc199ded83dcc2a3819eab3dd5115626ba12503
[05-23 15:12:06][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 15:28:06][flashy.solver][INFO] - Train | Epoch 25 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.971E-01 | ce 0.258 | ppl 1.296
[05-23 15:43:57][flashy.solver][INFO] - Train | Epoch 25 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.715E-01 | ce 0.258 | ppl 1.298
[05-23 15:59:47][flashy.solver][INFO] - Train | Epoch 25 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.141E-01 | ce 0.259 | ppl 1.298
[05-23 16:15:40][flashy.solver][INFO] - Train | Epoch 25 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.456E-01 | ce 0.262 | ppl 1.303
[05-23 16:31:31][flashy.solver][INFO] - Train | Epoch 25 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.779E-01 | ce 0.253 | ppl 1.291
[05-23 16:47:25][flashy.solver][INFO] - Train | Epoch 25 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.205E-01 | ce 0.259 | ppl 1.298
[05-23 17:03:16][flashy.solver][INFO] - Train | Epoch 25 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.473E-01 | ce 0.257 | ppl 1.295
[05-23 17:19:07][flashy.solver][INFO] - Train | Epoch 25 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.975E-01 | ce 0.257 | ppl 1.296
[05-23 17:35:00][flashy.solver][INFO] - Train | Epoch 25 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.467E-01 | ce 0.262 | ppl 1.302
[05-23 17:50:53][flashy.solver][INFO] - Train Summary | Epoch 25 | lr=1.00E-04 | grad_norm=5.350E-01 | ce=0.258 | ppl=1.297 | duration=9526.316
[05-23 17:51:29][flashy.solver][INFO] - Valid | Epoch 25 | 7/79 | 0.22 it/sec | ce 0.233 | ppl 1.264
[05-23 17:52:01][flashy.solver][INFO] - Valid | Epoch 25 | 14/79 | 0.22 it/sec | ce 0.216 | ppl 1.244
[05-23 17:52:34][flashy.solver][INFO] - Valid | Epoch 25 | 21/79 | 0.22 it/sec | ce 0.257 | ppl 1.295
[05-23 17:53:05][flashy.solver][INFO] - Valid | Epoch 25 | 28/79 | 0.22 it/sec | ce 0.221 | ppl 1.251
[05-23 17:53:38][flashy.solver][INFO] - Valid | Epoch 25 | 35/79 | 0.22 it/sec | ce 0.254 | ppl 1.291
[05-23 17:54:10][flashy.solver][INFO] - Valid | Epoch 25 | 42/79 | 0.22 it/sec | ce 0.220 | ppl 1.248
[05-23 17:54:43][flashy.solver][INFO] - Valid | Epoch 25 | 49/79 | 0.22 it/sec | ce 0.296 | ppl 1.348
[05-23 17:55:15][flashy.solver][INFO] - Valid | Epoch 25 | 56/79 | 0.22 it/sec | ce 0.221 | ppl 1.249
[05-23 17:55:47][flashy.solver][INFO] - Valid | Epoch 25 | 63/79 | 0.22 it/sec | ce 0.280 | ppl 1.324
[05-23 17:56:19][flashy.solver][INFO] - Valid | Epoch 25 | 70/79 | 0.22 it/sec | ce 0.259 | ppl 1.297
[05-23 17:56:52][flashy.solver][INFO] - Valid | Epoch 25 | 77/79 | 0.22 it/sec | ce 0.297 | ppl 1.347
[05-23 17:56:52][flashy.solver][INFO] - Valid Summary | Epoch 25 | ce=0.249 | ppl=1.286 | duration=359.858
[05-23 17:56:52][flashy.solver][INFO] - New best state with ce=0.249 (was 0.250)
[05-23 17:56:52][flashy.solver][INFO] - Evaluate Summary | Epoch 25 | duration=0.000
[05-23 17:56:52][flashy.solver][INFO] - Generating samples in /tmp/audiocraft_root/xps/a4911d1c/samples
[05-23 17:57:00][flashy.solver][INFO] - Generate Summary | Epoch 25 | rtf=0.143 | duration=7.398
[05-23 17:57:00][flashy.solver][INFO] - Model hash: 82c2fcbc8bd1496521b2818b13c15d8789a80548
[05-23 17:57:00][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 18:12:58][flashy.solver][INFO] - Train | Epoch 26 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.386E-01 | ce 0.262 | ppl 1.302
[05-23 18:28:49][flashy.solver][INFO] - Train | Epoch 26 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.030E-01 | ce 0.252 | ppl 1.289
[05-23 18:44:37][flashy.solver][INFO] - Train | Epoch 26 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.877E-01 | ce 0.261 | ppl 1.301
[05-23 19:00:27][flashy.solver][INFO] - Train | Epoch 26 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.051E-01 | ce 0.255 | ppl 1.293
[05-23 19:16:16][flashy.solver][INFO] - Train | Epoch 26 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.895E-01 | ce 0.254 | ppl 1.292
[05-23 19:32:04][flashy.solver][INFO] - Train | Epoch 26 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.937E-01 | ce 0.259 | ppl 1.299
[05-23 19:47:52][flashy.solver][INFO] - Train | Epoch 26 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.375E-01 | ce 0.253 | ppl 1.290
[05-23 20:03:42][flashy.solver][INFO] - Train | Epoch 26 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.000E-01 | ce 0.256 | ppl 1.295
[05-23 20:19:33][flashy.solver][INFO] - Train | Epoch 26 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.955E-01 | ce 0.252 | ppl 1.290
[05-23 20:35:25][flashy.solver][INFO] - Train Summary | Epoch 26 | lr=1.00E-04 | grad_norm=5.344E-01 | ce=0.255 | ppl=1.294 | duration=9505.304
[05-23 20:36:02][flashy.solver][INFO] - Valid | Epoch 26 | 7/79 | 0.22 it/sec | ce 0.231 | ppl 1.262
[05-23 20:36:34][flashy.solver][INFO] - Valid | Epoch 26 | 14/79 | 0.22 it/sec | ce 0.214 | ppl 1.242
[05-23 20:37:06][flashy.solver][INFO] - Valid | Epoch 26 | 21/79 | 0.22 it/sec | ce 0.256 | ppl 1.293
[05-23 20:37:38][flashy.solver][INFO] - Valid | Epoch 26 | 28/79 | 0.22 it/sec | ce 0.219 | ppl 1.249
[05-23 20:38:10][flashy.solver][INFO] - Valid | Epoch 26 | 35/79 | 0.22 it/sec | ce 0.252 | ppl 1.289
[05-23 20:38:43][flashy.solver][INFO] - Valid | Epoch 26 | 42/79 | 0.22 it/sec | ce 0.218 | ppl 1.246
[05-23 20:39:15][flashy.solver][INFO] - Valid | Epoch 26 | 49/79 | 0.22 it/sec | ce 0.294 | ppl 1.345
[05-23 20:39:47][flashy.solver][INFO] - Valid | Epoch 26 | 56/79 | 0.22 it/sec | ce 0.220 | ppl 1.247
[05-23 20:40:20][flashy.solver][INFO] - Valid | Epoch 26 | 63/79 | 0.22 it/sec | ce 0.278 | ppl 1.321
[05-23 20:40:52][flashy.solver][INFO] - Valid | Epoch 26 | 70/79 | 0.22 it/sec | ce 0.257 | ppl 1.294
[05-23 20:41:24][flashy.solver][INFO] - Valid | Epoch 26 | 77/79 | 0.22 it/sec | ce 0.295 | ppl 1.344
[05-23 20:41:25][flashy.solver][INFO] - Valid Summary | Epoch 26 | ce=0.248 | ppl=1.284 | duration=359.841
[05-23 20:41:25][flashy.solver][INFO] - New best state with ce=0.248 (was 0.249)
[05-23 20:41:25][flashy.solver][INFO] - Model hash: 089095e59713faaf19b02e819c2f6dc8ef1e9dab
[05-23 20:41:25][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 20:57:22][flashy.solver][INFO] - Train | Epoch 27 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.654E-01 | ce 0.260 | ppl 1.300
[05-23 21:13:13][flashy.solver][INFO] - Train | Epoch 27 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.251E-01 | ce 0.254 | ppl 1.291
[05-23 21:29:05][flashy.solver][INFO] - Train | Epoch 27 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.386E-01 | ce 0.256 | ppl 1.294
[05-23 21:44:58][flashy.solver][INFO] - Train | Epoch 27 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.366E-01 | ce 0.256 | ppl 1.295
[05-23 22:00:47][flashy.solver][INFO] - Train | Epoch 27 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.061E-01 | ce 0.254 | ppl 1.291
[05-23 22:16:37][flashy.solver][INFO] - Train | Epoch 27 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.201E-01 | ce 0.256 | ppl 1.294
[05-23 22:32:26][flashy.solver][INFO] - Train | Epoch 27 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.026E-01 | ce 0.253 | ppl 1.291
[05-23 22:48:16][flashy.solver][INFO] - Train | Epoch 27 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.352E-01 | ce 0.261 | ppl 1.301
[05-23 23:04:11][flashy.solver][INFO] - Train | Epoch 27 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.283E-01 | ce 0.255 | ppl 1.294
[05-23 23:20:01][flashy.solver][INFO] - Train Summary | Epoch 27 | lr=1.00E-04 | grad_norm=5.476E-01 | ce=0.256 | ppl=1.294 | duration=9515.969
[05-23 23:20:38][flashy.solver][INFO] - Valid | Epoch 27 | 7/79 | 0.22 it/sec | ce 0.230 | ppl 1.261
[05-23 23:21:10][flashy.solver][INFO] - Valid | Epoch 27 | 14/79 | 0.22 it/sec | ce 0.214 | ppl 1.241
[05-23 23:21:42][flashy.solver][INFO] - Valid | Epoch 27 | 21/79 | 0.22 it/sec | ce 0.255 | ppl 1.292
[05-23 23:22:14][flashy.solver][INFO] - Valid | Epoch 27 | 28/79 | 0.22 it/sec | ce 0.219 | ppl 1.248
[05-23 23:22:46][flashy.solver][INFO] - Valid | Epoch 27 | 35/79 | 0.22 it/sec | ce 0.251 | ppl 1.287
[05-23 23:23:18][flashy.solver][INFO] - Valid | Epoch 27 | 42/79 | 0.22 it/sec | ce 0.218 | ppl 1.245
[05-23 23:23:51][flashy.solver][INFO] - Valid | Epoch 27 | 49/79 | 0.22 it/sec | ce 0.293 | ppl 1.344
[05-23 23:24:23][flashy.solver][INFO] - Valid | Epoch 27 | 56/79 | 0.22 it/sec | ce 0.219 | ppl 1.246
[05-23 23:24:55][flashy.solver][INFO] - Valid | Epoch 27 | 63/79 | 0.22 it/sec | ce 0.277 | ppl 1.320
[05-23 23:25:27][flashy.solver][INFO] - Valid | Epoch 27 | 70/79 | 0.22 it/sec | ce 0.256 | ppl 1.294
[05-23 23:26:00][flashy.solver][INFO] - Valid | Epoch 27 | 77/79 | 0.22 it/sec | ce 0.294 | ppl 1.343
[05-23 23:26:01][flashy.solver][INFO] - Valid Summary | Epoch 27 | ce=0.247 | ppl=1.282 | duration=359.513
[05-23 23:26:01][flashy.solver][INFO] - New best state with ce=0.247 (was 0.248)
[05-23 23:26:01][flashy.solver][INFO] - Model hash: ce34cd299e956d4309b90649c62e11fc580efaff
[05-23 23:26:01][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-23 23:41:57][flashy.solver][INFO] - Train | Epoch 28 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.395E-01 | ce 0.255 | ppl 1.293
[05-23 23:57:46][flashy.solver][INFO] - Train | Epoch 28 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.770E-01 | ce 0.255 | ppl 1.293
[05-24 00:13:38][flashy.solver][INFO] - Train | Epoch 28 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.924E-01 | ce 0.260 | ppl 1.299
[05-24 00:29:30][flashy.solver][INFO] - Train | Epoch 28 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.739E-01 | ce 0.257 | ppl 1.296
[05-24 00:45:22][flashy.solver][INFO] - Train | Epoch 28 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 4.952E-01 | ce 0.252 | ppl 1.289
[05-24 01:01:15][flashy.solver][INFO] - Train | Epoch 28 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.369E-01 | ce 0.258 | ppl 1.296
[05-24 01:17:08][flashy.solver][INFO] - Train | Epoch 28 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.589E-01 | ce 0.257 | ppl 1.295
[05-24 01:33:00][flashy.solver][INFO] - Train | Epoch 28 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.766E-01 | ce 0.244 | ppl 1.279
[05-24 01:48:50][flashy.solver][INFO] - Train | Epoch 28 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.690E-01 | ce 0.252 | ppl 1.289
[05-24 02:04:43][flashy.solver][INFO] - Train Summary | Epoch 28 | lr=1.00E-04 | grad_norm=5.444E-01 | ce=0.254 | ppl=1.292 | duration=9521.965
[05-24 02:05:19][flashy.solver][INFO] - Valid | Epoch 28 | 7/79 | 0.22 it/sec | ce 0.229 | ppl 1.259
[05-24 02:05:51][flashy.solver][INFO] - Valid | Epoch 28 | 14/79 | 0.22 it/sec | ce 0.212 | ppl 1.239
[05-24 02:06:24][flashy.solver][INFO] - Valid | Epoch 28 | 21/79 | 0.22 it/sec | ce 0.253 | ppl 1.290
[05-24 02:06:55][flashy.solver][INFO] - Valid | Epoch 28 | 28/79 | 0.22 it/sec | ce 0.217 | ppl 1.246
[05-24 02:07:28][flashy.solver][INFO] - Valid | Epoch 28 | 35/79 | 0.22 it/sec | ce 0.250 | ppl 1.286
[05-24 02:08:00][flashy.solver][INFO] - Valid | Epoch 28 | 42/79 | 0.22 it/sec | ce 0.216 | ppl 1.243
[05-24 02:08:33][flashy.solver][INFO] - Valid | Epoch 28 | 49/79 | 0.22 it/sec | ce 0.291 | ppl 1.341
[05-24 02:09:05][flashy.solver][INFO] - Valid | Epoch 28 | 56/79 | 0.22 it/sec | ce 0.218 | ppl 1.245
[05-24 02:09:37][flashy.solver][INFO] - Valid | Epoch 28 | 63/79 | 0.22 it/sec | ce 0.275 | ppl 1.318
[05-24 02:10:09][flashy.solver][INFO] - Valid | Epoch 28 | 70/79 | 0.22 it/sec | ce 0.255 | ppl 1.291
[05-24 02:10:42][flashy.solver][INFO] - Valid | Epoch 28 | 77/79 | 0.22 it/sec | ce 0.292 | ppl 1.341
[05-24 02:10:43][flashy.solver][INFO] - Valid Summary | Epoch 28 | ce=0.245 | ppl=1.281 | duration=360.116
[05-24 02:10:43][flashy.solver][INFO] - New best state with ce=0.245 (was 0.247)
[05-24 02:10:43][flashy.solver][INFO] - Model hash: 567a27cbb456d75b4be504917a87c47b4d578054
[05-24 02:10:43][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-24 02:26:41][flashy.solver][INFO] - Train | Epoch 29 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.435E-01 | ce 0.254 | ppl 1.292
[05-24 02:42:34][flashy.solver][INFO] - Train | Epoch 29 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.189E-01 | ce 0.255 | ppl 1.293
[05-24 02:58:27][flashy.solver][INFO] - Train | Epoch 29 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.327E-01 | ce 0.249 | ppl 1.285
[05-24 03:14:21][flashy.solver][INFO] - Train | Epoch 29 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.991E-01 | ce 0.255 | ppl 1.293
[05-24 03:30:12][flashy.solver][INFO] - Train | Epoch 29 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.458E-01 | ce 0.251 | ppl 1.287
[05-24 03:46:05][flashy.solver][INFO] - Train | Epoch 29 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.023E-01 | ce 0.259 | ppl 1.298
[05-24 04:01:58][flashy.solver][INFO] - Train | Epoch 29 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.590E-01 | ce 0.256 | ppl 1.294
[05-24 04:17:53][flashy.solver][INFO] - Train | Epoch 29 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.849E-01 | ce 0.260 | ppl 1.299
[05-24 04:33:46][flashy.solver][INFO] - Train | Epoch 29 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.561E-01 | ce 0.250 | ppl 1.287
[05-24 04:49:39][flashy.solver][INFO] - Train Summary | Epoch 29 | lr=1.00E-04 | grad_norm=5.705E-01 | ce=0.255 | ppl=1.293 | duration=9536.583
[05-24 04:50:16][flashy.solver][INFO] - Valid | Epoch 29 | 7/79 | 0.22 it/sec | ce 0.229 | ppl 1.259
[05-24 04:50:49][flashy.solver][INFO] - Valid | Epoch 29 | 14/79 | 0.22 it/sec | ce 0.212 | ppl 1.239
[05-24 04:51:21][flashy.solver][INFO] - Valid | Epoch 29 | 21/79 | 0.22 it/sec | ce 0.253 | ppl 1.290
[05-24 04:51:53][flashy.solver][INFO] - Valid | Epoch 29 | 28/79 | 0.22 it/sec | ce 0.217 | ppl 1.246
[05-24 04:52:25][flashy.solver][INFO] - Valid | Epoch 29 | 35/79 | 0.22 it/sec | ce 0.250 | ppl 1.285
[05-24 04:52:58][flashy.solver][INFO] - Valid | Epoch 29 | 42/79 | 0.22 it/sec | ce 0.216 | ppl 1.243
[05-24 04:53:30][flashy.solver][INFO] - Valid | Epoch 29 | 49/79 | 0.22 it/sec | ce 0.291 | ppl 1.341
[05-24 04:54:03][flashy.solver][INFO] - Valid | Epoch 29 | 56/79 | 0.22 it/sec | ce 0.218 | ppl 1.245
[05-24 04:54:35][flashy.solver][INFO] - Valid | Epoch 29 | 63/79 | 0.22 it/sec | ce 0.275 | ppl 1.318
[05-24 04:55:07][flashy.solver][INFO] - Valid | Epoch 29 | 70/79 | 0.22 it/sec | ce 0.255 | ppl 1.291
[05-24 04:55:40][flashy.solver][INFO] - Valid | Epoch 29 | 77/79 | 0.22 it/sec | ce 0.291 | ppl 1.340
[05-24 04:55:40][flashy.solver][INFO] - Valid Summary | Epoch 29 | ce=0.245 | ppl=1.280 | duration=361.077
[05-24 04:55:40][flashy.solver][INFO] - New best state with ce=0.245 (was 0.245)
[05-24 04:55:40][flashy.solver][INFO] - Model hash: a73caf2914df9412a113d65df262fdabca51c93b
[05-24 04:55:40][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-24 05:11:39][flashy.solver][INFO] - Train | Epoch 30 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.017E-01 | ce 0.254 | ppl 1.292
[05-24 05:27:30][flashy.solver][INFO] - Train | Epoch 30 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.751E-01 | ce 0.251 | ppl 1.288
[05-24 05:43:21][flashy.solver][INFO] - Train | Epoch 30 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.390E-01 | ce 0.254 | ppl 1.292
[05-24 05:59:11][flashy.solver][INFO] - Train | Epoch 30 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.182E-01 | ce 0.254 | ppl 1.291
[05-24 06:15:01][flashy.solver][INFO] - Train | Epoch 30 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.205E-01 | ce 0.252 | ppl 1.290
[05-24 06:30:51][flashy.solver][INFO] - Train | Epoch 30 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.301E-01 | ce 0.257 | ppl 1.295
[05-24 06:46:43][flashy.solver][INFO] - Train | Epoch 30 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.656E-01 | ce 0.252 | ppl 1.289
[05-24 07:02:36][flashy.solver][INFO] - Train | Epoch 30 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.067E-01 | ce 0.260 | ppl 1.300
[05-24 07:18:27][flashy.solver][INFO] - Train | Epoch 30 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.246E-01 | ce 0.245 | ppl 1.280
[05-24 07:34:18][flashy.solver][INFO] - Train Summary | Epoch 30 | lr=1.00E-04 | grad_norm=5.520E-01 | ce=0.253 | ppl=1.291 | duration=9517.157
[05-24 07:34:55][flashy.solver][INFO] - Valid | Epoch 30 | 7/79 | 0.22 it/sec | ce 0.227 | ppl 1.257
[05-24 07:35:27][flashy.solver][INFO] - Valid | Epoch 30 | 14/79 | 0.22 it/sec | ce 0.211 | ppl 1.237
[05-24 07:35:59][flashy.solver][INFO] - Valid | Epoch 30 | 21/79 | 0.22 it/sec | ce 0.251 | ppl 1.287
[05-24 07:36:31][flashy.solver][INFO] - Valid | Epoch 30 | 28/79 | 0.22 it/sec | ce 0.215 | ppl 1.244
[05-24 07:37:03][flashy.solver][INFO] - Valid | Epoch 30 | 35/79 | 0.22 it/sec | ce 0.248 | ppl 1.283
[05-24 07:37:35][flashy.solver][INFO] - Valid | Epoch 30 | 42/79 | 0.22 it/sec | ce 0.214 | ppl 1.241
[05-24 07:38:08][flashy.solver][INFO] - Valid | Epoch 30 | 49/79 | 0.22 it/sec | ce 0.288 | ppl 1.338
[05-24 07:38:40][flashy.solver][INFO] - Valid | Epoch 30 | 56/79 | 0.22 it/sec | ce 0.216 | ppl 1.243
[05-24 07:39:12][flashy.solver][INFO] - Valid | Epoch 30 | 63/79 | 0.22 it/sec | ce 0.273 | ppl 1.315
[05-24 07:39:44][flashy.solver][INFO] - Valid | Epoch 30 | 70/79 | 0.22 it/sec | ce 0.253 | ppl 1.289
[05-24 07:40:17][flashy.solver][INFO] - Valid | Epoch 30 | 77/79 | 0.22 it/sec | ce 0.289 | ppl 1.337
[05-24 07:40:18][flashy.solver][INFO] - Valid Summary | Epoch 30 | ce=0.243 | ppl=1.278 | duration=360.175
[05-24 07:40:18][flashy.solver][INFO] - New best state with ce=0.243 (was 0.245)
[05-24 07:40:18][flashy.solver][INFO] - Model hash: af5ad3d51af03adc908af24eb6edb8df054b14b5
[05-24 07:40:18][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-24 07:56:15][flashy.solver][INFO] - Train | Epoch 31 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.549E-01 | ce 0.255 | ppl 1.293
[05-24 08:12:06][flashy.solver][INFO] - Train | Epoch 31 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.101E-01 | ce 0.248 | ppl 1.284
[05-24 08:27:56][flashy.solver][INFO] - Train | Epoch 31 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.389E-01 | ce 0.253 | ppl 1.290
[05-24 08:43:47][flashy.solver][INFO] - Train | Epoch 31 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.224E-01 | ce 0.245 | ppl 1.280
[05-24 08:59:37][flashy.solver][INFO] - Train | Epoch 31 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.551E-01 | ce 0.257 | ppl 1.296
[05-24 09:15:30][flashy.solver][INFO] - Train | Epoch 31 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.719E-01 | ce 0.253 | ppl 1.291
[05-24 09:31:22][flashy.solver][INFO] - Train | Epoch 31 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.871E-01 | ce 0.252 | ppl 1.289
[05-24 09:47:12][flashy.solver][INFO] - Train | Epoch 31 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.260E-01 | ce 0.254 | ppl 1.292
[05-24 10:03:03][flashy.solver][INFO] - Train | Epoch 31 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.704E-01 | ce 0.252 | ppl 1.290
[05-24 10:18:54][flashy.solver][INFO] - Train Summary | Epoch 31 | lr=1.00E-04 | grad_norm=5.536E-01 | ce=0.252 | ppl=1.290 | duration=9515.999
[05-24 10:19:31][flashy.solver][INFO] - Valid | Epoch 31 | 7/79 | 0.22 it/sec | ce 0.226 | ppl 1.256
[05-24 10:20:03][flashy.solver][INFO] - Valid | Epoch 31 | 14/79 | 0.22 it/sec | ce 0.210 | ppl 1.236
[05-24 10:20:35][flashy.solver][INFO] - Valid | Epoch 31 | 21/79 | 0.22 it/sec | ce 0.250 | ppl 1.286
[05-24 10:21:07][flashy.solver][INFO] - Valid | Epoch 31 | 28/79 | 0.22 it/sec | ce 0.215 | ppl 1.243
[05-24 10:21:39][flashy.solver][INFO] - Valid | Epoch 31 | 35/79 | 0.22 it/sec | ce 0.247 | ppl 1.282
[05-24 10:22:11][flashy.solver][INFO] - Valid | Epoch 31 | 42/79 | 0.22 it/sec | ce 0.214 | ppl 1.240
[05-24 10:22:44][flashy.solver][INFO] - Valid | Epoch 31 | 49/79 | 0.22 it/sec | ce 0.287 | ppl 1.336
[05-24 10:23:16][flashy.solver][INFO] - Valid | Epoch 31 | 56/79 | 0.22 it/sec | ce 0.216 | ppl 1.242
[05-24 10:23:48][flashy.solver][INFO] - Valid | Epoch 31 | 63/79 | 0.22 it/sec | ce 0.272 | ppl 1.314
[05-24 10:24:20][flashy.solver][INFO] - Valid | Epoch 31 | 70/79 | 0.22 it/sec | ce 0.252 | ppl 1.288
[05-24 10:24:53][flashy.solver][INFO] - Valid | Epoch 31 | 77/79 | 0.22 it/sec | ce 0.288 | ppl 1.336
[05-24 10:24:54][flashy.solver][INFO] - Valid Summary | Epoch 31 | ce=0.242 | ppl=1.277 | duration=359.757
[05-24 10:24:54][flashy.solver][INFO] - New best state with ce=0.242 (was 0.243)
[05-24 10:24:54][flashy.solver][INFO] - Model hash: 76b85fe16b33946521175c2c9a28e10fc2356026
[05-24 10:24:54][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-24 10:40:51][flashy.solver][INFO] - Train | Epoch 32 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.817E-01 | ce 0.252 | ppl 1.289
[05-24 10:56:43][flashy.solver][INFO] - Train | Epoch 32 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.465E-01 | ce 0.258 | ppl 1.297
[05-24 11:12:37][flashy.solver][INFO] - Train | Epoch 32 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.570E-01 | ce 0.259 | ppl 1.299
[05-24 11:28:28][flashy.solver][INFO] - Train | Epoch 32 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.353E-01 | ce 0.249 | ppl 1.285
[05-24 11:44:22][flashy.solver][INFO] - Train | Epoch 32 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.530E-01 | ce 0.245 | ppl 1.280
[05-24 12:00:12][flashy.solver][INFO] - Train | Epoch 32 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.862E-01 | ce 0.254 | ppl 1.292
[05-24 12:16:03][flashy.solver][INFO] - Train | Epoch 32 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.853E-01 | ce 0.248 | ppl 1.283
[05-24 12:31:55][flashy.solver][INFO] - Train | Epoch 32 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.626E-01 | ce 0.247 | ppl 1.283
[05-24 12:47:48][flashy.solver][INFO] - Train | Epoch 32 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.465E-01 | ce 0.249 | ppl 1.285
[05-24 13:03:39][flashy.solver][INFO] - Train Summary | Epoch 32 | lr=1.00E-04 | grad_norm=5.734E-01 | ce=0.251 | ppl=1.288 | duration=9525.455
[05-24 13:04:16][flashy.solver][INFO] - Valid | Epoch 32 | 7/79 | 0.22 it/sec | ce 0.226 | ppl 1.255
[05-24 13:04:48][flashy.solver][INFO] - Valid | Epoch 32 | 14/79 | 0.22 it/sec | ce 0.209 | ppl 1.235
[05-24 13:05:20][flashy.solver][INFO] - Valid | Epoch 32 | 21/79 | 0.22 it/sec | ce 0.249 | ppl 1.285
[05-24 13:05:52][flashy.solver][INFO] - Valid | Epoch 32 | 28/79 | 0.22 it/sec | ce 0.214 | ppl 1.242
[05-24 13:06:24][flashy.solver][INFO] - Valid | Epoch 32 | 35/79 | 0.22 it/sec | ce 0.246 | ppl 1.280
[05-24 13:06:56][flashy.solver][INFO] - Valid | Epoch 32 | 42/79 | 0.22 it/sec | ce 0.213 | ppl 1.239
[05-24 13:07:29][flashy.solver][INFO] - Valid | Epoch 32 | 49/79 | 0.22 it/sec | ce 0.286 | ppl 1.335
[05-24 13:08:01][flashy.solver][INFO] - Valid | Epoch 32 | 56/79 | 0.22 it/sec | ce 0.214 | ppl 1.241
[05-24 13:08:34][flashy.solver][INFO] - Valid | Epoch 32 | 63/79 | 0.22 it/sec | ce 0.271 | ppl 1.312
[05-24 13:09:06][flashy.solver][INFO] - Valid | Epoch 32 | 70/79 | 0.22 it/sec | ce 0.251 | ppl 1.286
[05-24 13:09:38][flashy.solver][INFO] - Valid | Epoch 32 | 77/79 | 0.22 it/sec | ce 0.287 | ppl 1.334
[05-24 13:09:39][flashy.solver][INFO] - Valid Summary | Epoch 32 | ce=0.241 | ppl=1.275 | duration=359.850
[05-24 13:09:39][flashy.solver][INFO] - New best state with ce=0.241 (was 0.242)
[05-24 13:09:39][flashy.solver][INFO] - Model hash: f7d0691657a18de0c2e851fdb09b96737a6ddc6a
[05-24 13:09:39][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-24 13:25:37][flashy.solver][INFO] - Train | Epoch 33 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.970E-01 | ce 0.249 | ppl 1.286
[05-24 13:41:28][flashy.solver][INFO] - Train | Epoch 33 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.586E-01 | ce 0.252 | ppl 1.288
[05-24 13:57:21][flashy.solver][INFO] - Train | Epoch 33 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.145E-01 | ce 0.263 | ppl 1.303
[05-24 14:13:10][flashy.solver][INFO] - Train | Epoch 33 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.589E-01 | ce 0.253 | ppl 1.291
[05-24 14:29:04][flashy.solver][INFO] - Train | Epoch 33 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.990E-01 | ce 0.252 | ppl 1.290
[05-24 14:44:55][flashy.solver][INFO] - Train | Epoch 33 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.516E-01 | ce 0.242 | ppl 1.276
[05-24 15:00:45][flashy.solver][INFO] - Train | Epoch 33 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.286E-01 | ce 0.250 | ppl 1.287
[05-24 15:16:39][flashy.solver][INFO] - Train | Epoch 33 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.518E-01 | ce 0.251 | ppl 1.289
[05-24 15:32:30][flashy.solver][INFO] - Train | Epoch 33 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.202E-01 | ce 0.251 | ppl 1.289
[05-24 15:48:21][flashy.solver][INFO] - Train Summary | Epoch 33 | lr=1.00E-04 | grad_norm=5.841E-01 | ce=0.252 | ppl=1.289 | duration=9522.251
[05-24 15:48:58][flashy.solver][INFO] - Valid | Epoch 33 | 7/79 | 0.22 it/sec | ce 0.225 | ppl 1.254
[05-24 15:49:30][flashy.solver][INFO] - Valid | Epoch 33 | 14/79 | 0.22 it/sec | ce 0.208 | ppl 1.234
[05-24 15:50:03][flashy.solver][INFO] - Valid | Epoch 33 | 21/79 | 0.22 it/sec | ce 0.248 | ppl 1.283
[05-24 15:50:35][flashy.solver][INFO] - Valid | Epoch 33 | 28/79 | 0.22 it/sec | ce 0.213 | ppl 1.240
[05-24 15:51:07][flashy.solver][INFO] - Valid | Epoch 33 | 35/79 | 0.22 it/sec | ce 0.245 | ppl 1.279
[05-24 15:51:39][flashy.solver][INFO] - Valid | Epoch 33 | 42/79 | 0.22 it/sec | ce 0.212 | ppl 1.238
[05-24 15:52:12][flashy.solver][INFO] - Valid | Epoch 33 | 49/79 | 0.22 it/sec | ce 0.285 | ppl 1.334
[05-24 15:52:44][flashy.solver][INFO] - Valid | Epoch 33 | 56/79 | 0.22 it/sec | ce 0.214 | ppl 1.239
[05-24 15:53:17][flashy.solver][INFO] - Valid | Epoch 33 | 63/79 | 0.22 it/sec | ce 0.270 | ppl 1.311
[05-24 15:53:49][flashy.solver][INFO] - Valid | Epoch 33 | 70/79 | 0.22 it/sec | ce 0.250 | ppl 1.285
[05-24 15:54:22][flashy.solver][INFO] - Valid | Epoch 33 | 77/79 | 0.22 it/sec | ce 0.286 | ppl 1.332
[05-24 15:54:22][flashy.solver][INFO] - Valid Summary | Epoch 33 | ce=0.240 | ppl=1.274 | duration=360.855
[05-24 15:54:22][flashy.solver][INFO] - New best state with ce=0.240 (was 0.241)
[05-24 15:54:22][flashy.solver][INFO] - Model hash: 44f06f95fe8780a50c35fc898fa8e4e5cd3fb863
[05-24 15:54:22][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-24 16:10:19][flashy.solver][INFO] - Train | Epoch 34 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.040E-01 | ce 0.257 | ppl 1.296
[05-24 16:26:12][flashy.solver][INFO] - Train | Epoch 34 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.585E-01 | ce 0.249 | ppl 1.286
[05-24 16:42:02][flashy.solver][INFO] - Train | Epoch 34 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.014E-01 | ce 0.248 | ppl 1.284
[05-24 16:57:55][flashy.solver][INFO] - Train | Epoch 34 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.921E-01 | ce 0.253 | ppl 1.290
[05-24 17:13:49][flashy.solver][INFO] - Train | Epoch 34 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.551E-01 | ce 0.249 | ppl 1.285
[05-24 17:29:42][flashy.solver][INFO] - Train | Epoch 34 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.247E-01 | ce 0.249 | ppl 1.286
[05-24 17:45:33][flashy.solver][INFO] - Train | Epoch 34 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.108E-01 | ce 0.246 | ppl 1.282
[05-24 18:01:25][flashy.solver][INFO] - Train | Epoch 34 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.016E-01 | ce 0.248 | ppl 1.283
[05-24 18:17:18][flashy.solver][INFO] - Train | Epoch 34 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.011E-01 | ce 0.252 | ppl 1.290
[05-24 18:33:12][flashy.solver][INFO] - Train Summary | Epoch 34 | lr=1.00E-04 | grad_norm=5.932E-01 | ce=0.250 | ppl=1.287 | duration=9530.122
[05-24 18:33:49][flashy.solver][INFO] - Valid | Epoch 34 | 7/79 | 0.22 it/sec | ce 0.224 | ppl 1.252
[05-24 18:34:22][flashy.solver][INFO] - Valid | Epoch 34 | 14/79 | 0.22 it/sec | ce 0.207 | ppl 1.233
[05-24 18:34:54][flashy.solver][INFO] - Valid | Epoch 34 | 21/79 | 0.22 it/sec | ce 0.247 | ppl 1.282
[05-24 18:35:26][flashy.solver][INFO] - Valid | Epoch 34 | 28/79 | 0.22 it/sec | ce 0.212 | ppl 1.240
[05-24 18:35:58][flashy.solver][INFO] - Valid | Epoch 34 | 35/79 | 0.22 it/sec | ce 0.244 | ppl 1.278
[05-24 18:36:30][flashy.solver][INFO] - Valid | Epoch 34 | 42/79 | 0.22 it/sec | ce 0.211 | ppl 1.237
[05-24 18:37:03][flashy.solver][INFO] - Valid | Epoch 34 | 49/79 | 0.22 it/sec | ce 0.284 | ppl 1.332
[05-24 18:37:35][flashy.solver][INFO] - Valid | Epoch 34 | 56/79 | 0.22 it/sec | ce 0.213 | ppl 1.239
[05-24 18:38:07][flashy.solver][INFO] - Valid | Epoch 34 | 63/79 | 0.22 it/sec | ce 0.269 | ppl 1.310
[05-24 18:38:39][flashy.solver][INFO] - Valid | Epoch 34 | 70/79 | 0.22 it/sec | ce 0.249 | ppl 1.284
[05-24 18:39:12][flashy.solver][INFO] - Valid | Epoch 34 | 77/79 | 0.22 it/sec | ce 0.285 | ppl 1.331
[05-24 18:39:12][flashy.solver][INFO] - Valid Summary | Epoch 34 | ce=0.240 | ppl=1.273 | duration=360.152
[05-24 18:39:12][flashy.solver][INFO] - New best state with ce=0.240 (was 0.240)
[05-24 18:39:13][flashy.solver][INFO] - Model hash: 4d02274241806560883697c1f84d588fbc677870
[05-24 18:39:13][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-24 18:55:12][flashy.solver][INFO] - Train | Epoch 35 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.694E-01 | ce 0.249 | ppl 1.284
[05-24 19:11:03][flashy.solver][INFO] - Train | Epoch 35 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.506E-01 | ce 0.250 | ppl 1.286
[05-24 19:26:57][flashy.solver][INFO] - Train | Epoch 35 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.631E-01 | ce 0.253 | ppl 1.291
[05-24 19:42:49][flashy.solver][INFO] - Train | Epoch 35 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.017E-01 | ce 0.243 | ppl 1.278
[05-24 19:58:40][flashy.solver][INFO] - Train | Epoch 35 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.467E-01 | ce 0.242 | ppl 1.276
[05-24 20:14:31][flashy.solver][INFO] - Train | Epoch 35 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.451E-01 | ce 0.242 | ppl 1.275
[05-24 20:30:25][flashy.solver][INFO] - Train | Epoch 35 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.488E-01 | ce 0.250 | ppl 1.287
[05-24 20:46:16][flashy.solver][INFO] - Train | Epoch 35 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.197E-01 | ce 0.252 | ppl 1.290
[05-24 21:02:06][flashy.solver][INFO] - Train | Epoch 35 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.820E-01 | ce 0.250 | ppl 1.287
[05-24 21:17:57][flashy.solver][INFO] - Train Summary | Epoch 35 | lr=1.00E-04 | grad_norm=5.957E-01 | ce=0.248 | ppl=1.284 | duration=9524.631
[05-24 21:18:34][flashy.solver][INFO] - Valid | Epoch 35 | 7/79 | 0.22 it/sec | ce 0.223 | ppl 1.252
[05-24 21:19:06][flashy.solver][INFO] - Valid | Epoch 35 | 14/79 | 0.22 it/sec | ce 0.207 | ppl 1.232
[05-24 21:19:38][flashy.solver][INFO] - Valid | Epoch 35 | 21/79 | 0.22 it/sec | ce 0.247 | ppl 1.281
[05-24 21:20:10][flashy.solver][INFO] - Valid | Epoch 35 | 28/79 | 0.22 it/sec | ce 0.211 | ppl 1.238
[05-24 21:20:42][flashy.solver][INFO] - Valid | Epoch 35 | 35/79 | 0.22 it/sec | ce 0.243 | ppl 1.277
[05-24 21:21:14][flashy.solver][INFO] - Valid | Epoch 35 | 42/79 | 0.22 it/sec | ce 0.210 | ppl 1.236
[05-24 21:21:47][flashy.solver][INFO] - Valid | Epoch 35 | 49/79 | 0.22 it/sec | ce 0.283 | ppl 1.331
[05-24 21:22:19][flashy.solver][INFO] - Valid | Epoch 35 | 56/79 | 0.22 it/sec | ce 0.212 | ppl 1.238
[05-24 21:22:52][flashy.solver][INFO] - Valid | Epoch 35 | 63/79 | 0.22 it/sec | ce 0.268 | ppl 1.308
[05-24 21:23:24][flashy.solver][INFO] - Valid | Epoch 35 | 70/79 | 0.22 it/sec | ce 0.248 | ppl 1.283
[05-24 21:23:56][flashy.solver][INFO] - Valid | Epoch 35 | 77/79 | 0.22 it/sec | ce 0.284 | ppl 1.330
[05-24 21:23:57][flashy.solver][INFO] - Valid Summary | Epoch 35 | ce=0.239 | ppl=1.272 | duration=359.587
[05-24 21:23:57][flashy.solver][INFO] - New best state with ce=0.239 (was 0.240)
[05-24 21:23:57][flashy.solver][INFO] - Model hash: a70b5601a97c81bb15d7f418580a48a5dea69a9b
[05-24 21:23:57][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-24 21:39:54][flashy.solver][INFO] - Train | Epoch 36 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.471E-01 | ce 0.252 | ppl 1.289
[05-24 21:55:43][flashy.solver][INFO] - Train | Epoch 36 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.130E-01 | ce 0.253 | ppl 1.290
[05-24 22:11:37][flashy.solver][INFO] - Train | Epoch 36 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.048E-01 | ce 0.247 | ppl 1.283
[05-24 22:27:29][flashy.solver][INFO] - Train | Epoch 36 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.543E-01 | ce 0.248 | ppl 1.285
[05-24 22:43:20][flashy.solver][INFO] - Train | Epoch 36 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.046E-01 | ce 0.251 | ppl 1.287
[05-24 22:59:14][flashy.solver][INFO] - Train | Epoch 36 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.131E-01 | ce 0.245 | ppl 1.280
[05-24 23:15:08][flashy.solver][INFO] - Train | Epoch 36 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.385E-01 | ce 0.251 | ppl 1.288
[05-24 23:31:01][flashy.solver][INFO] - Train | Epoch 36 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.339E-01 | ce 0.251 | ppl 1.287
[05-24 23:46:54][flashy.solver][INFO] - Train | Epoch 36 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.124E-01 | ce 0.259 | ppl 1.299
[05-25 00:02:48][flashy.solver][INFO] - Train Summary | Epoch 36 | lr=1.00E-04 | grad_norm=6.259E-01 | ce=0.251 | ppl=1.288 | duration=9531.088
[05-25 00:03:25][flashy.solver][INFO] - Valid | Epoch 36 | 7/79 | 0.22 it/sec | ce 0.223 | ppl 1.252
[05-25 00:03:57][flashy.solver][INFO] - Valid | Epoch 36 | 14/79 | 0.22 it/sec | ce 0.207 | ppl 1.232
[05-25 00:04:29][flashy.solver][INFO] - Valid | Epoch 36 | 21/79 | 0.22 it/sec | ce 0.247 | ppl 1.282
[05-25 00:05:01][flashy.solver][INFO] - Valid | Epoch 36 | 28/79 | 0.22 it/sec | ce 0.212 | ppl 1.239
[05-25 00:05:33][flashy.solver][INFO] - Valid | Epoch 36 | 35/79 | 0.22 it/sec | ce 0.243 | ppl 1.277
[05-25 00:06:05][flashy.solver][INFO] - Valid | Epoch 36 | 42/79 | 0.22 it/sec | ce 0.210 | ppl 1.236
[05-25 00:06:38][flashy.solver][INFO] - Valid | Epoch 36 | 49/79 | 0.22 it/sec | ce 0.283 | ppl 1.330
[05-25 00:07:10][flashy.solver][INFO] - Valid | Epoch 36 | 56/79 | 0.22 it/sec | ce 0.212 | ppl 1.238
[05-25 00:07:43][flashy.solver][INFO] - Valid | Epoch 36 | 63/79 | 0.22 it/sec | ce 0.268 | ppl 1.308
[05-25 00:08:15][flashy.solver][INFO] - Valid | Epoch 36 | 70/79 | 0.22 it/sec | ce 0.248 | ppl 1.283
[05-25 00:08:47][flashy.solver][INFO] - Valid | Epoch 36 | 77/79 | 0.22 it/sec | ce 0.284 | ppl 1.330
[05-25 00:08:48][flashy.solver][INFO] - Valid Summary | Epoch 36 | ce=0.239 | ppl=1.272 | duration=360.096
[05-25 00:08:48][flashy.solver][INFO] - Model hash: cb01e245d919de8796afb41a0aedaba8fec59a59
[05-25 00:08:48][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 00:24:46][flashy.solver][INFO] - Train | Epoch 37 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.874E-01 | ce 0.245 | ppl 1.280
[05-25 00:40:39][flashy.solver][INFO] - Train | Epoch 37 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.040E-01 | ce 0.246 | ppl 1.281
[05-25 00:56:33][flashy.solver][INFO] - Train | Epoch 37 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.731E-01 | ce 0.249 | ppl 1.286
[05-25 01:12:27][flashy.solver][INFO] - Train | Epoch 37 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.121E-01 | ce 0.238 | ppl 1.272
[05-25 01:28:19][flashy.solver][INFO] - Train | Epoch 37 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.164E-01 | ce 0.251 | ppl 1.287
[05-25 01:44:13][flashy.solver][INFO] - Train | Epoch 37 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.818E-01 | ce 0.250 | ppl 1.287
[05-25 02:00:04][flashy.solver][INFO] - Train | Epoch 37 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.743E-01 | ce 0.246 | ppl 1.281
[05-25 02:15:57][flashy.solver][INFO] - Train | Epoch 37 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.882E-01 | ce 0.253 | ppl 1.291
[05-25 02:31:49][flashy.solver][INFO] - Train | Epoch 37 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.363E-01 | ce 0.252 | ppl 1.288
[05-25 02:47:41][flashy.solver][INFO] - Train Summary | Epoch 37 | lr=1.00E-04 | grad_norm=6.270E-01 | ce=0.248 | ppl=1.284 | duration=9533.307
[05-25 02:48:18][flashy.solver][INFO] - Valid | Epoch 37 | 7/79 | 0.22 it/sec | ce 0.221 | ppl 1.249
[05-25 02:48:50][flashy.solver][INFO] - Valid | Epoch 37 | 14/79 | 0.22 it/sec | ce 0.205 | ppl 1.230
[05-25 02:49:23][flashy.solver][INFO] - Valid | Epoch 37 | 21/79 | 0.22 it/sec | ce 0.245 | ppl 1.280
[05-25 02:49:55][flashy.solver][INFO] - Valid | Epoch 37 | 28/79 | 0.22 it/sec | ce 0.210 | ppl 1.237
[05-25 02:50:27][flashy.solver][INFO] - Valid | Epoch 37 | 35/79 | 0.22 it/sec | ce 0.242 | ppl 1.275
[05-25 02:50:59][flashy.solver][INFO] - Valid | Epoch 37 | 42/79 | 0.22 it/sec | ce 0.209 | ppl 1.234
[05-25 02:51:32][flashy.solver][INFO] - Valid | Epoch 37 | 49/79 | 0.22 it/sec | ce 0.281 | ppl 1.328
[05-25 02:52:04][flashy.solver][INFO] - Valid | Epoch 37 | 56/79 | 0.22 it/sec | ce 0.211 | ppl 1.236
[05-25 02:52:36][flashy.solver][INFO] - Valid | Epoch 37 | 63/79 | 0.22 it/sec | ce 0.266 | ppl 1.306
[05-25 02:53:09][flashy.solver][INFO] - Valid | Epoch 37 | 70/79 | 0.22 it/sec | ce 0.246 | ppl 1.281
[05-25 02:53:41][flashy.solver][INFO] - Valid | Epoch 37 | 77/79 | 0.22 it/sec | ce 0.282 | ppl 1.327
[05-25 02:53:42][flashy.solver][INFO] - Valid Summary | Epoch 37 | ce=0.237 | ppl=1.270 | duration=360.546
[05-25 02:53:42][flashy.solver][INFO] - New best state with ce=0.237 (was 0.239)
[05-25 02:53:42][flashy.solver][INFO] - Model hash: cd3bf3b02d1001aa26ecfbc70b7a4b920e508867
[05-25 02:53:42][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 03:09:36][flashy.solver][INFO] - Train | Epoch 38 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.020E-01 | ce 0.248 | ppl 1.283
[05-25 03:25:28][flashy.solver][INFO] - Train | Epoch 38 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.813E-01 | ce 0.247 | ppl 1.282
[05-25 03:41:19][flashy.solver][INFO] - Train | Epoch 38 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.343E-01 | ce 0.253 | ppl 1.290
[05-25 03:57:11][flashy.solver][INFO] - Train | Epoch 38 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.233E-01 | ce 0.248 | ppl 1.284
[05-25 04:13:06][flashy.solver][INFO] - Train | Epoch 38 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.261E-01 | ce 0.251 | ppl 1.288
[05-25 04:29:00][flashy.solver][INFO] - Train | Epoch 38 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.650E-01 | ce 0.251 | ppl 1.288
[05-25 04:44:53][flashy.solver][INFO] - Train | Epoch 38 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.008E-01 | ce 0.247 | ppl 1.283
[05-25 05:00:45][flashy.solver][INFO] - Train | Epoch 38 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.579E-01 | ce 0.247 | ppl 1.282
[05-25 05:16:40][flashy.solver][INFO] - Train | Epoch 38 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.793E-01 | ce 0.247 | ppl 1.283
[05-25 05:32:31][flashy.solver][INFO] - Train Summary | Epoch 38 | lr=1.00E-04 | grad_norm=6.272E-01 | ce=0.248 | ppl=1.284 | duration=9529.414
[05-25 05:33:09][flashy.solver][INFO] - Valid | Epoch 38 | 7/79 | 0.22 it/sec | ce 0.221 | ppl 1.249
[05-25 05:33:41][flashy.solver][INFO] - Valid | Epoch 38 | 14/79 | 0.22 it/sec | ce 0.205 | ppl 1.230
[05-25 05:34:13][flashy.solver][INFO] - Valid | Epoch 38 | 21/79 | 0.22 it/sec | ce 0.245 | ppl 1.279
[05-25 05:34:45][flashy.solver][INFO] - Valid | Epoch 38 | 28/79 | 0.22 it/sec | ce 0.210 | ppl 1.236
[05-25 05:35:17][flashy.solver][INFO] - Valid | Epoch 38 | 35/79 | 0.22 it/sec | ce 0.242 | ppl 1.275
[05-25 05:35:50][flashy.solver][INFO] - Valid | Epoch 38 | 42/79 | 0.22 it/sec | ce 0.209 | ppl 1.234
[05-25 05:36:22][flashy.solver][INFO] - Valid | Epoch 38 | 49/79 | 0.22 it/sec | ce 0.281 | ppl 1.328
[05-25 05:36:54][flashy.solver][INFO] - Valid | Epoch 38 | 56/79 | 0.22 it/sec | ce 0.211 | ppl 1.236
[05-25 05:37:27][flashy.solver][INFO] - Valid | Epoch 38 | 63/79 | 0.22 it/sec | ce 0.266 | ppl 1.306
[05-25 05:37:59][flashy.solver][INFO] - Valid | Epoch 38 | 70/79 | 0.22 it/sec | ce 0.246 | ppl 1.280
[05-25 05:38:32][flashy.solver][INFO] - Valid | Epoch 38 | 77/79 | 0.22 it/sec | ce 0.282 | ppl 1.327
[05-25 05:38:32][flashy.solver][INFO] - Valid Summary | Epoch 38 | ce=0.237 | ppl=1.270 | duration=360.767
[05-25 05:38:32][flashy.solver][INFO] - New best state with ce=0.237 (was 0.237)
[05-25 05:38:32][flashy.solver][INFO] - Model hash: 974b1095ca97f29f9b200189baf2d55cc5aa6eaa
[05-25 05:38:32][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 05:54:31][flashy.solver][INFO] - Train | Epoch 39 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.076E-01 | ce 0.246 | ppl 1.282
[05-25 06:10:24][flashy.solver][INFO] - Train | Epoch 39 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.448E-01 | ce 0.245 | ppl 1.281
[05-25 06:26:17][flashy.solver][INFO] - Train | Epoch 39 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.985E-01 | ce 0.251 | ppl 1.287
[05-25 06:42:09][flashy.solver][INFO] - Train | Epoch 39 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.798E-01 | ce 0.248 | ppl 1.284
[05-25 06:57:59][flashy.solver][INFO] - Train | Epoch 39 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.250E-01 | ce 0.242 | ppl 1.276
[05-25 07:13:50][flashy.solver][INFO] - Train | Epoch 39 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.279E-01 | ce 0.242 | ppl 1.276
[05-25 07:29:41][flashy.solver][INFO] - Train | Epoch 39 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.988E-01 | ce 0.246 | ppl 1.281
[05-25 07:45:32][flashy.solver][INFO] - Train | Epoch 39 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.974E-01 | ce 0.246 | ppl 1.281
[05-25 08:01:24][flashy.solver][INFO] - Train | Epoch 39 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.590E-01 | ce 0.252 | ppl 1.288
[05-25 08:17:17][flashy.solver][INFO] - Train Summary | Epoch 39 | lr=1.00E-04 | grad_norm=6.386E-01 | ce=0.246 | ppl=1.281 | duration=9524.562
[05-25 08:17:54][flashy.solver][INFO] - Valid | Epoch 39 | 7/79 | 0.22 it/sec | ce 0.220 | ppl 1.248
[05-25 08:18:26][flashy.solver][INFO] - Valid | Epoch 39 | 14/79 | 0.22 it/sec | ce 0.204 | ppl 1.229
[05-25 08:18:58][flashy.solver][INFO] - Valid | Epoch 39 | 21/79 | 0.22 it/sec | ce 0.244 | ppl 1.278
[05-25 08:19:30][flashy.solver][INFO] - Valid | Epoch 39 | 28/79 | 0.22 it/sec | ce 0.208 | ppl 1.235
[05-25 08:20:02][flashy.solver][INFO] - Valid | Epoch 39 | 35/79 | 0.22 it/sec | ce 0.240 | ppl 1.273
[05-25 08:20:34][flashy.solver][INFO] - Valid | Epoch 39 | 42/79 | 0.22 it/sec | ce 0.207 | ppl 1.232
[05-25 08:21:07][flashy.solver][INFO] - Valid | Epoch 39 | 49/79 | 0.22 it/sec | ce 0.280 | ppl 1.326
[05-25 08:21:39][flashy.solver][INFO] - Valid | Epoch 39 | 56/79 | 0.22 it/sec | ce 0.209 | ppl 1.234
[05-25 08:22:12][flashy.solver][INFO] - Valid | Epoch 39 | 63/79 | 0.22 it/sec | ce 0.265 | ppl 1.304
[05-25 08:22:44][flashy.solver][INFO] - Valid | Epoch 39 | 70/79 | 0.22 it/sec | ce 0.245 | ppl 1.279
[05-25 08:23:17][flashy.solver][INFO] - Valid | Epoch 39 | 77/79 | 0.22 it/sec | ce 0.280 | ppl 1.325
[05-25 08:23:17][flashy.solver][INFO] - Valid Summary | Epoch 39 | ce=0.236 | ppl=1.268 | duration=360.385
[05-25 08:23:17][flashy.solver][INFO] - New best state with ce=0.236 (was 0.237)
[05-25 08:23:17][flashy.solver][INFO] - Model hash: a62b20ceaadc7440362b7e064d273851cd412593
[05-25 08:23:17][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 08:39:13][flashy.solver][INFO] - Train | Epoch 40 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.792E-01 | ce 0.245 | ppl 1.280
[05-25 08:55:05][flashy.solver][INFO] - Train | Epoch 40 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.248E-01 | ce 0.243 | ppl 1.277
[05-25 09:10:56][flashy.solver][INFO] - Train | Epoch 40 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.419E-01 | ce 0.246 | ppl 1.281
[05-25 09:26:51][flashy.solver][INFO] - Train | Epoch 40 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.416E-01 | ce 0.241 | ppl 1.275
[05-25 09:42:45][flashy.solver][INFO] - Train | Epoch 40 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.018E-01 | ce 0.243 | ppl 1.277
[05-25 09:58:40][flashy.solver][INFO] - Train | Epoch 40 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.495E-01 | ce 0.249 | ppl 1.285
[05-25 10:14:35][flashy.solver][INFO] - Train | Epoch 40 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.643E-01 | ce 0.247 | ppl 1.282
[05-25 10:30:25][flashy.solver][INFO] - Train | Epoch 40 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.740E-01 | ce 0.244 | ppl 1.278
[05-25 10:46:36][flashy.solver][INFO] - Train | Epoch 40 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.227E-01 | ce 0.248 | ppl 1.283
[05-25 11:04:00][flashy.solver][INFO] - Train Summary | Epoch 40 | lr=1.00E-04 | grad_norm=6.360E-01 | ce=0.244 | ppl=1.279 | duration=9642.435
[05-25 11:04:39][flashy.solver][INFO] - Valid | Epoch 40 | 7/79 | 0.20 it/sec | ce 0.220 | ppl 1.247
[05-25 11:05:14][flashy.solver][INFO] - Valid | Epoch 40 | 14/79 | 0.20 it/sec | ce 0.203 | ppl 1.228
[05-25 11:05:49][flashy.solver][INFO] - Valid | Epoch 40 | 21/79 | 0.20 it/sec | ce 0.243 | ppl 1.277
[05-25 11:06:23][flashy.solver][INFO] - Valid | Epoch 40 | 28/79 | 0.20 it/sec | ce 0.208 | ppl 1.235
[05-25 11:06:58][flashy.solver][INFO] - Valid | Epoch 40 | 35/79 | 0.20 it/sec | ce 0.240 | ppl 1.272
[05-25 11:07:32][flashy.solver][INFO] - Valid | Epoch 40 | 42/79 | 0.20 it/sec | ce 0.207 | ppl 1.232
[05-25 11:08:07][flashy.solver][INFO] - Valid | Epoch 40 | 49/79 | 0.20 it/sec | ce 0.279 | ppl 1.325
[05-25 11:08:42][flashy.solver][INFO] - Valid | Epoch 40 | 56/79 | 0.20 it/sec | ce 0.209 | ppl 1.234
[05-25 11:09:18][flashy.solver][INFO] - Valid | Epoch 40 | 63/79 | 0.20 it/sec | ce 0.264 | ppl 1.303
[05-25 11:09:52][flashy.solver][INFO] - Valid | Epoch 40 | 70/79 | 0.20 it/sec | ce 0.245 | ppl 1.278
[05-25 11:10:25][flashy.solver][INFO] - Valid | Epoch 40 | 77/79 | 0.20 it/sec | ce 0.280 | ppl 1.324
[05-25 11:10:25][flashy.solver][INFO] - Valid Summary | Epoch 40 | ce=0.235 | ppl=1.267 | duration=385.528
[05-25 11:10:25][flashy.solver][INFO] - New best state with ce=0.235 (was 0.236)
[05-25 11:10:25][flashy.solver][INFO] - Model hash: fa08650ff740a38784bb792d05c3aa88c3285f7d
[05-25 11:10:25][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 11:26:21][flashy.solver][INFO] - Train | Epoch 41 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.025E-01 | ce 0.241 | ppl 1.275
[05-25 11:42:16][flashy.solver][INFO] - Train | Epoch 41 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.107E-01 | ce 0.243 | ppl 1.277
[05-25 11:58:12][flashy.solver][INFO] - Train | Epoch 41 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.134E-01 | ce 0.252 | ppl 1.289
[05-25 12:14:09][flashy.solver][INFO] - Train | Epoch 41 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.748E-01 | ce 0.249 | ppl 1.285
[05-25 12:30:44][flashy.solver][INFO] - Train | Epoch 41 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.731E-01 | ce 0.241 | ppl 1.275
[05-25 12:47:41][flashy.solver][INFO] - Train | Epoch 41 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.499E-01 | ce 0.246 | ppl 1.282
[05-25 13:03:50][flashy.solver][INFO] - Train | Epoch 41 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.622E-01 | ce 0.252 | ppl 1.290
[05-25 13:19:45][flashy.solver][INFO] - Train | Epoch 41 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 5.915E-01 | ce 0.244 | ppl 1.279
[05-25 13:35:41][flashy.solver][INFO] - Train | Epoch 41 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.581E-01 | ce 0.245 | ppl 1.280
[05-25 13:51:34][flashy.solver][INFO] - Train Summary | Epoch 41 | lr=1.00E-04 | grad_norm=6.410E-01 | ce=0.246 | ppl=1.281 | duration=9668.847
[05-25 13:52:12][flashy.solver][INFO] - Valid | Epoch 41 | 7/79 | 0.21 it/sec | ce 0.220 | ppl 1.248
[05-25 13:52:44][flashy.solver][INFO] - Valid | Epoch 41 | 14/79 | 0.21 it/sec | ce 0.203 | ppl 1.228
[05-25 13:53:17][flashy.solver][INFO] - Valid | Epoch 41 | 21/79 | 0.21 it/sec | ce 0.243 | ppl 1.277
[05-25 13:53:49][flashy.solver][INFO] - Valid | Epoch 41 | 28/79 | 0.22 it/sec | ce 0.208 | ppl 1.234
[05-25 13:54:21][flashy.solver][INFO] - Valid | Epoch 41 | 35/79 | 0.22 it/sec | ce 0.240 | ppl 1.272
[05-25 13:54:54][flashy.solver][INFO] - Valid | Epoch 41 | 42/79 | 0.22 it/sec | ce 0.207 | ppl 1.232
[05-25 13:55:27][flashy.solver][INFO] - Valid | Epoch 41 | 49/79 | 0.22 it/sec | ce 0.279 | ppl 1.325
[05-25 13:55:59][flashy.solver][INFO] - Valid | Epoch 41 | 56/79 | 0.22 it/sec | ce 0.209 | ppl 1.234
[05-25 13:56:32][flashy.solver][INFO] - Valid | Epoch 41 | 63/79 | 0.22 it/sec | ce 0.264 | ppl 1.303
[05-25 13:57:04][flashy.solver][INFO] - Valid | Epoch 41 | 70/79 | 0.22 it/sec | ce 0.245 | ppl 1.278
[05-25 13:57:37][flashy.solver][INFO] - Valid | Epoch 41 | 77/79 | 0.22 it/sec | ce 0.280 | ppl 1.324
[05-25 13:57:37][flashy.solver][INFO] - Valid Summary | Epoch 41 | ce=0.235 | ppl=1.267 | duration=363.285
[05-25 13:57:37][flashy.solver][INFO] - Model hash: 09e2a3abc27e6e335f3802783fbd6fe954f50d9d
[05-25 13:57:37][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 14:13:38][flashy.solver][INFO] - Train | Epoch 42 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.698E-01 | ce 0.247 | ppl 1.283
[05-25 14:29:34][flashy.solver][INFO] - Train | Epoch 42 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.624E-01 | ce 0.243 | ppl 1.278
[05-25 14:45:31][flashy.solver][INFO] - Train | Epoch 42 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.258E-01 | ce 0.240 | ppl 1.274
[05-25 15:01:28][flashy.solver][INFO] - Train | Epoch 42 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.307E-01 | ce 0.240 | ppl 1.274
[05-25 15:17:22][flashy.solver][INFO] - Train | Epoch 42 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.244E-01 | ce 0.237 | ppl 1.269
[05-25 15:33:19][flashy.solver][INFO] - Train | Epoch 42 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.679E-01 | ce 0.241 | ppl 1.275
[05-25 15:49:19][flashy.solver][INFO] - Train | Epoch 42 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.579E-01 | ce 0.247 | ppl 1.283
[05-25 16:05:16][flashy.solver][INFO] - Train | Epoch 42 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.652E-01 | ce 0.248 | ppl 1.283
[05-25 16:21:12][flashy.solver][INFO] - Train | Epoch 42 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.719E-01 | ce 0.242 | ppl 1.276
[05-25 16:37:10][flashy.solver][INFO] - Train Summary | Epoch 42 | lr=1.00E-04 | grad_norm=6.642E-01 | ce=0.244 | ppl=1.278 | duration=9572.143
[05-25 16:37:47][flashy.solver][INFO] - Valid | Epoch 42 | 7/79 | 0.22 it/sec | ce 0.218 | ppl 1.246
[05-25 16:38:19][flashy.solver][INFO] - Valid | Epoch 42 | 14/79 | 0.22 it/sec | ce 0.202 | ppl 1.226
[05-25 16:38:51][flashy.solver][INFO] - Valid | Epoch 42 | 21/79 | 0.22 it/sec | ce 0.242 | ppl 1.275
[05-25 16:39:23][flashy.solver][INFO] - Valid | Epoch 42 | 28/79 | 0.22 it/sec | ce 0.207 | ppl 1.233
[05-25 16:39:56][flashy.solver][INFO] - Valid | Epoch 42 | 35/79 | 0.22 it/sec | ce 0.238 | ppl 1.270
[05-25 16:40:28][flashy.solver][INFO] - Valid | Epoch 42 | 42/79 | 0.22 it/sec | ce 0.206 | ppl 1.230
[05-25 16:41:01][flashy.solver][INFO] - Valid | Epoch 42 | 49/79 | 0.22 it/sec | ce 0.278 | ppl 1.323
[05-25 16:41:33][flashy.solver][INFO] - Valid | Epoch 42 | 56/79 | 0.22 it/sec | ce 0.208 | ppl 1.232
[05-25 16:42:06][flashy.solver][INFO] - Valid | Epoch 42 | 63/79 | 0.22 it/sec | ce 0.263 | ppl 1.301
[05-25 16:42:38][flashy.solver][INFO] - Valid | Epoch 42 | 70/79 | 0.22 it/sec | ce 0.243 | ppl 1.277
[05-25 16:43:11][flashy.solver][INFO] - Valid | Epoch 42 | 77/79 | 0.22 it/sec | ce 0.278 | ppl 1.322
[05-25 16:43:12][flashy.solver][INFO] - Valid Summary | Epoch 42 | ce=0.234 | ppl=1.266 | duration=362.071
[05-25 16:43:12][flashy.solver][INFO] - New best state with ce=0.234 (was 0.235)
[05-25 16:43:12][flashy.solver][INFO] - Model hash: 7a5c994f138a508072ff17224f681d73fe64a7fa
[05-25 16:43:12][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 16:59:13][flashy.solver][INFO] - Train | Epoch 43 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.610E-01 | ce 0.244 | ppl 1.279
[05-25 17:15:12][flashy.solver][INFO] - Train | Epoch 43 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.619E-01 | ce 0.251 | ppl 1.287
[05-25 17:31:11][flashy.solver][INFO] - Train | Epoch 43 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.385E-01 | ce 0.244 | ppl 1.279
[05-25 17:47:08][flashy.solver][INFO] - Train | Epoch 43 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.543E-01 | ce 0.238 | ppl 1.271
[05-25 18:03:04][flashy.solver][INFO] - Train | Epoch 43 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.235E-01 | ce 0.239 | ppl 1.273
[05-25 18:18:57][flashy.solver][INFO] - Train | Epoch 43 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.888E-01 | ce 0.240 | ppl 1.273
[05-25 18:34:50][flashy.solver][INFO] - Train | Epoch 43 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.499E-01 | ce 0.247 | ppl 1.282
[05-25 18:50:42][flashy.solver][INFO] - Train | Epoch 43 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.470E-01 | ce 0.246 | ppl 1.282
[05-25 19:06:39][flashy.solver][INFO] - Train | Epoch 43 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.933E-01 | ce 0.247 | ppl 1.283
[05-25 19:22:36][flashy.solver][INFO] - Train Summary | Epoch 43 | lr=1.00E-04 | grad_norm=6.578E-01 | ce=0.244 | ppl=1.278 | duration=9564.111
[05-25 19:23:13][flashy.solver][INFO] - Valid | Epoch 43 | 7/79 | 0.22 it/sec | ce 0.218 | ppl 1.246
[05-25 19:23:45][flashy.solver][INFO] - Valid | Epoch 43 | 14/79 | 0.22 it/sec | ce 0.202 | ppl 1.226
[05-25 19:24:18][flashy.solver][INFO] - Valid | Epoch 43 | 21/79 | 0.22 it/sec | ce 0.242 | ppl 1.275
[05-25 19:24:50][flashy.solver][INFO] - Valid | Epoch 43 | 28/79 | 0.22 it/sec | ce 0.207 | ppl 1.233
[05-25 19:25:22][flashy.solver][INFO] - Valid | Epoch 43 | 35/79 | 0.22 it/sec | ce 0.238 | ppl 1.270
[05-25 19:25:55][flashy.solver][INFO] - Valid | Epoch 43 | 42/79 | 0.22 it/sec | ce 0.205 | ppl 1.230
[05-25 19:26:27][flashy.solver][INFO] - Valid | Epoch 43 | 49/79 | 0.22 it/sec | ce 0.277 | ppl 1.323
[05-25 19:27:00][flashy.solver][INFO] - Valid | Epoch 43 | 56/79 | 0.22 it/sec | ce 0.208 | ppl 1.232
[05-25 19:27:33][flashy.solver][INFO] - Valid | Epoch 43 | 63/79 | 0.22 it/sec | ce 0.262 | ppl 1.301
[05-25 19:28:05][flashy.solver][INFO] - Valid | Epoch 43 | 70/79 | 0.22 it/sec | ce 0.243 | ppl 1.276
[05-25 19:28:38][flashy.solver][INFO] - Valid | Epoch 43 | 77/79 | 0.22 it/sec | ce 0.278 | ppl 1.322
[05-25 19:28:38][flashy.solver][INFO] - Valid Summary | Epoch 43 | ce=0.234 | ppl=1.265 | duration=362.399
[05-25 19:28:38][flashy.solver][INFO] - New best state with ce=0.234 (was 0.234)
[05-25 19:28:38][flashy.solver][INFO] - Model hash: 7af89a88b4dcadafd87f443ad9f6a55db43672b1
[05-25 19:28:38][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 19:44:41][flashy.solver][INFO] - Train | Epoch 44 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.254E-01 | ce 0.242 | ppl 1.276
[05-25 20:00:37][flashy.solver][INFO] - Train | Epoch 44 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.996E-01 | ce 0.248 | ppl 1.283
[05-25 20:16:35][flashy.solver][INFO] - Train | Epoch 44 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.621E-01 | ce 0.246 | ppl 1.281
[05-25 20:32:29][flashy.solver][INFO] - Train | Epoch 44 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.448E-01 | ce 0.241 | ppl 1.276
[05-25 20:48:22][flashy.solver][INFO] - Train | Epoch 44 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.515E-01 | ce 0.234 | ppl 1.265
[05-25 21:04:13][flashy.solver][INFO] - Train | Epoch 44 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.588E-01 | ce 0.243 | ppl 1.277
[05-25 21:20:06][flashy.solver][INFO] - Train | Epoch 44 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.515E-01 | ce 0.243 | ppl 1.277
[05-25 21:35:59][flashy.solver][INFO] - Train | Epoch 44 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.543E-01 | ce 0.245 | ppl 1.280
[05-25 21:51:51][flashy.solver][INFO] - Train | Epoch 44 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.741E-01 | ce 0.247 | ppl 1.283
[05-25 22:07:44][flashy.solver][INFO] - Train Summary | Epoch 44 | lr=1.00E-04 | grad_norm=6.778E-01 | ce=0.244 | ppl=1.279 | duration=9545.633
[05-25 22:08:21][flashy.solver][INFO] - Valid | Epoch 44 | 7/79 | 0.22 it/sec | ce 0.217 | ppl 1.244
[05-25 22:08:53][flashy.solver][INFO] - Valid | Epoch 44 | 14/79 | 0.22 it/sec | ce 0.201 | ppl 1.225
[05-25 22:09:25][flashy.solver][INFO] - Valid | Epoch 44 | 21/79 | 0.22 it/sec | ce 0.241 | ppl 1.273
[05-25 22:09:57][flashy.solver][INFO] - Valid | Epoch 44 | 28/79 | 0.22 it/sec | ce 0.206 | ppl 1.232
[05-25 22:10:29][flashy.solver][INFO] - Valid | Epoch 44 | 35/79 | 0.22 it/sec | ce 0.237 | ppl 1.269
[05-25 22:11:02][flashy.solver][INFO] - Valid | Epoch 44 | 42/79 | 0.22 it/sec | ce 0.204 | ppl 1.228
[05-25 22:11:34][flashy.solver][INFO] - Valid | Epoch 44 | 49/79 | 0.22 it/sec | ce 0.276 | ppl 1.321
[05-25 22:12:06][flashy.solver][INFO] - Valid | Epoch 44 | 56/79 | 0.22 it/sec | ce 0.207 | ppl 1.231
[05-25 22:12:39][flashy.solver][INFO] - Valid | Epoch 44 | 63/79 | 0.22 it/sec | ce 0.261 | ppl 1.300
[05-25 22:13:11][flashy.solver][INFO] - Valid | Epoch 44 | 70/79 | 0.22 it/sec | ce 0.242 | ppl 1.275
[05-25 22:13:44][flashy.solver][INFO] - Valid | Epoch 44 | 77/79 | 0.22 it/sec | ce 0.277 | ppl 1.320
[05-25 22:13:44][flashy.solver][INFO] - Valid Summary | Epoch 44 | ce=0.233 | ppl=1.264 | duration=360.226
[05-25 22:13:44][flashy.solver][INFO] - New best state with ce=0.233 (was 0.234)
[05-25 22:13:44][flashy.solver][INFO] - Model hash: df5570593ba68bafd081d5cbf7e5812c86936ca3
[05-25 22:13:44][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-25 22:29:44][flashy.solver][INFO] - Train | Epoch 45 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.520E-01 | ce 0.239 | ppl 1.273
[05-25 22:45:41][flashy.solver][INFO] - Train | Epoch 45 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.714E-01 | ce 0.243 | ppl 1.277
[05-25 23:01:37][flashy.solver][INFO] - Train | Epoch 45 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.988E-01 | ce 0.245 | ppl 1.280
[05-25 23:17:34][flashy.solver][INFO] - Train | Epoch 45 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.756E-01 | ce 0.234 | ppl 1.265
[05-25 23:33:30][flashy.solver][INFO] - Train | Epoch 45 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.593E-01 | ce 0.244 | ppl 1.279
[05-25 23:49:25][flashy.solver][INFO] - Train | Epoch 45 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.775E-01 | ce 0.239 | ppl 1.273
[05-26 00:05:23][flashy.solver][INFO] - Train | Epoch 45 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.746E-01 | ce 0.236 | ppl 1.269
[05-26 00:21:17][flashy.solver][INFO] - Train | Epoch 45 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.752E-01 | ce 0.242 | ppl 1.277
[05-26 00:37:08][flashy.solver][INFO] - Train | Epoch 45 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.779E-01 | ce 0.239 | ppl 1.273
[05-26 00:53:03][flashy.solver][INFO] - Train Summary | Epoch 45 | lr=1.00E-04 | grad_norm=6.707E-01 | ce=0.240 | ppl=1.273 | duration=9558.510
[05-26 00:53:40][flashy.solver][INFO] - Valid | Epoch 45 | 7/79 | 0.22 it/sec | ce 0.217 | ppl 1.244
[05-26 00:54:12][flashy.solver][INFO] - Valid | Epoch 45 | 14/79 | 0.22 it/sec | ce 0.200 | ppl 1.224
[05-26 00:54:44][flashy.solver][INFO] - Valid | Epoch 45 | 21/79 | 0.22 it/sec | ce 0.240 | ppl 1.273
[05-26 00:55:16][flashy.solver][INFO] - Valid | Epoch 45 | 28/79 | 0.22 it/sec | ce 0.206 | ppl 1.231
[05-26 00:55:49][flashy.solver][INFO] - Valid | Epoch 45 | 35/79 | 0.22 it/sec | ce 0.237 | ppl 1.268
[05-26 00:56:21][flashy.solver][INFO] - Valid | Epoch 45 | 42/79 | 0.22 it/sec | ce 0.204 | ppl 1.228
[05-26 00:56:54][flashy.solver][INFO] - Valid | Epoch 45 | 49/79 | 0.22 it/sec | ce 0.275 | ppl 1.320
[05-26 00:57:26][flashy.solver][INFO] - Valid | Epoch 45 | 56/79 | 0.22 it/sec | ce 0.206 | ppl 1.230
[05-26 00:57:58][flashy.solver][INFO] - Valid | Epoch 45 | 63/79 | 0.22 it/sec | ce 0.260 | ppl 1.298
[05-26 00:58:31][flashy.solver][INFO] - Valid | Epoch 45 | 70/79 | 0.22 it/sec | ce 0.242 | ppl 1.274
[05-26 00:59:03][flashy.solver][INFO] - Valid | Epoch 45 | 77/79 | 0.22 it/sec | ce 0.276 | ppl 1.319
[05-26 00:59:04][flashy.solver][INFO] - Valid Summary | Epoch 45 | ce=0.232 | ppl=1.263 | duration=361.292
[05-26 00:59:04][flashy.solver][INFO] - New best state with ce=0.232 (was 0.233)
[05-26 00:59:04][flashy.solver][INFO] - Model hash: be674586388d33fbfb69c7058abff8f68c8a76f7
[05-26 00:59:04][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 01:15:04][flashy.solver][INFO] - Train | Epoch 46 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.353E-01 | ce 0.246 | ppl 1.281
[05-26 01:31:00][flashy.solver][INFO] - Train | Epoch 46 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.262E-01 | ce 0.248 | ppl 1.284
[05-26 01:46:51][flashy.solver][INFO] - Train | Epoch 46 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.580E-01 | ce 0.238 | ppl 1.271
[05-26 02:02:42][flashy.solver][INFO] - Train | Epoch 46 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.037E-01 | ce 0.242 | ppl 1.276
[05-26 02:18:33][flashy.solver][INFO] - Train | Epoch 46 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.056E-01 | ce 0.244 | ppl 1.279
[05-26 02:34:28][flashy.solver][INFO] - Train | Epoch 46 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.534E-01 | ce 0.237 | ppl 1.270
[05-26 02:50:24][flashy.solver][INFO] - Train | Epoch 46 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.121E-01 | ce 0.237 | ppl 1.269
[05-26 03:06:22][flashy.solver][INFO] - Train | Epoch 46 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.266E-01 | ce 0.249 | ppl 1.285
[05-26 03:22:19][flashy.solver][INFO] - Train | Epoch 46 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.998E-01 | ce 0.243 | ppl 1.278
[05-26 03:38:14][flashy.solver][INFO] - Train Summary | Epoch 46 | lr=1.00E-04 | grad_norm=7.081E-01 | ce=0.243 | ppl=1.277 | duration=9549.454
[05-26 03:38:51][flashy.solver][INFO] - Valid | Epoch 46 | 7/79 | 0.22 it/sec | ce 0.217 | ppl 1.244
[05-26 03:39:23][flashy.solver][INFO] - Valid | Epoch 46 | 14/79 | 0.22 it/sec | ce 0.201 | ppl 1.225
[05-26 03:39:55][flashy.solver][INFO] - Valid | Epoch 46 | 21/79 | 0.22 it/sec | ce 0.240 | ppl 1.273
[05-26 03:40:27][flashy.solver][INFO] - Valid | Epoch 46 | 28/79 | 0.22 it/sec | ce 0.205 | ppl 1.231
[05-26 03:40:59][flashy.solver][INFO] - Valid | Epoch 46 | 35/79 | 0.22 it/sec | ce 0.236 | ppl 1.268
[05-26 03:41:32][flashy.solver][INFO] - Valid | Epoch 46 | 42/79 | 0.22 it/sec | ce 0.204 | ppl 1.228
[05-26 03:42:04][flashy.solver][INFO] - Valid | Epoch 46 | 49/79 | 0.22 it/sec | ce 0.275 | ppl 1.320
[05-26 03:42:37][flashy.solver][INFO] - Valid | Epoch 46 | 56/79 | 0.22 it/sec | ce 0.207 | ppl 1.231
[05-26 03:43:10][flashy.solver][INFO] - Valid | Epoch 46 | 63/79 | 0.22 it/sec | ce 0.261 | ppl 1.299
[05-26 03:43:42][flashy.solver][INFO] - Valid | Epoch 46 | 70/79 | 0.22 it/sec | ce 0.242 | ppl 1.274
[05-26 03:44:14][flashy.solver][INFO] - Valid | Epoch 46 | 77/79 | 0.22 it/sec | ce 0.276 | ppl 1.319
[05-26 03:44:15][flashy.solver][INFO] - Valid Summary | Epoch 46 | ce=0.232 | ppl=1.263 | duration=361.476
[05-26 03:44:15][flashy.solver][INFO] - Model hash: ce6f20c5d45c353e19f7d3ac9b1583a24b93f30f
[05-26 03:44:15][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 04:00:12][flashy.solver][INFO] - Train | Epoch 47 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.031E-01 | ce 0.237 | ppl 1.269
[05-26 04:16:06][flashy.solver][INFO] - Train | Epoch 47 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.487E-01 | ce 0.232 | ppl 1.264
[05-26 04:32:44][flashy.solver][INFO] - Train | Epoch 47 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.003E-01 | ce 0.247 | ppl 1.283
[05-26 04:49:35][flashy.solver][INFO] - Train | Epoch 47 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.085E-01 | ce 0.239 | ppl 1.272
[05-26 05:06:31][flashy.solver][INFO] - Train | Epoch 47 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.744E-01 | ce 0.238 | ppl 1.271
[05-26 05:23:23][flashy.solver][INFO] - Train | Epoch 47 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.294E-01 | ce 0.242 | ppl 1.277
[05-26 05:40:24][flashy.solver][INFO] - Train | Epoch 47 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.547E-01 | ce 0.239 | ppl 1.272
[05-26 05:57:16][flashy.solver][INFO] - Train | Epoch 47 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.255E-01 | ce 0.241 | ppl 1.275
[05-26 06:14:02][flashy.solver][INFO] - Train | Epoch 47 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.599E-01 | ce 0.247 | ppl 1.283
[05-26 06:30:50][flashy.solver][INFO] - Train Summary | Epoch 47 | lr=1.00E-04 | grad_norm=6.983E-01 | ce=0.241 | ppl=1.275 | duration=9994.908
[05-26 06:31:29][flashy.solver][INFO] - Valid | Epoch 47 | 7/79 | 0.21 it/sec | ce 0.216 | ppl 1.242
[05-26 06:32:03][flashy.solver][INFO] - Valid | Epoch 47 | 14/79 | 0.21 it/sec | ce 0.200 | ppl 1.224
[05-26 06:32:37][flashy.solver][INFO] - Valid | Epoch 47 | 21/79 | 0.21 it/sec | ce 0.239 | ppl 1.272
[05-26 06:33:11][flashy.solver][INFO] - Valid | Epoch 47 | 28/79 | 0.21 it/sec | ce 0.205 | ppl 1.230
[05-26 06:33:45][flashy.solver][INFO] - Valid | Epoch 47 | 35/79 | 0.21 it/sec | ce 0.235 | ppl 1.267
[05-26 06:34:19][flashy.solver][INFO] - Valid | Epoch 47 | 42/79 | 0.21 it/sec | ce 0.203 | ppl 1.227
[05-26 06:34:53][flashy.solver][INFO] - Valid | Epoch 47 | 49/79 | 0.21 it/sec | ce 0.274 | ppl 1.318
[05-26 06:35:27][flashy.solver][INFO] - Valid | Epoch 47 | 56/79 | 0.21 it/sec | ce 0.205 | ppl 1.229
[05-26 06:36:01][flashy.solver][INFO] - Valid | Epoch 47 | 63/79 | 0.21 it/sec | ce 0.260 | ppl 1.297
[05-26 06:36:35][flashy.solver][INFO] - Valid | Epoch 47 | 70/79 | 0.21 it/sec | ce 0.240 | ppl 1.273
[05-26 06:37:10][flashy.solver][INFO] - Valid | Epoch 47 | 77/79 | 0.21 it/sec | ce 0.275 | ppl 1.317
[05-26 06:37:10][flashy.solver][INFO] - Valid Summary | Epoch 47 | ce=0.231 | ppl=1.262 | duration=380.415
[05-26 06:37:10][flashy.solver][INFO] - New best state with ce=0.231 (was 0.232)
[05-26 06:37:10][flashy.solver][INFO] - Model hash: ed35dd4ab736527bd7e0850a1779a2da0e669f53
[05-26 06:37:11][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 06:53:53][flashy.solver][INFO] - Train | Epoch 48 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.553E-01 | ce 0.237 | ppl 1.270
[05-26 07:09:57][flashy.solver][INFO] - Train | Epoch 48 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.556E-01 | ce 0.237 | ppl 1.270
[05-26 07:26:19][flashy.solver][INFO] - Train | Epoch 48 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.824E-01 | ce 0.240 | ppl 1.274
[05-26 07:42:41][flashy.solver][INFO] - Train | Epoch 48 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.527E-01 | ce 0.249 | ppl 1.286
[05-26 07:59:04][flashy.solver][INFO] - Train | Epoch 48 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.142E-01 | ce 0.244 | ppl 1.279
[05-26 08:15:22][flashy.solver][INFO] - Train | Epoch 48 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.341E-01 | ce 0.235 | ppl 1.267
[05-26 08:31:17][flashy.solver][INFO] - Train | Epoch 48 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.184E-01 | ce 0.241 | ppl 1.275
[05-26 08:47:20][flashy.solver][INFO] - Train | Epoch 48 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.140E-01 | ce 0.245 | ppl 1.279
[05-26 09:03:34][flashy.solver][INFO] - Train | Epoch 48 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.133E-01 | ce 0.236 | ppl 1.269
[05-26 09:19:47][flashy.solver][INFO] - Train Summary | Epoch 48 | lr=1.00E-04 | grad_norm=7.072E-01 | ce=0.241 | ppl=1.274 | duration=9756.736
[05-26 09:20:25][flashy.solver][INFO] - Valid | Epoch 48 | 7/79 | 0.21 it/sec | ce 0.216 | ppl 1.242
[05-26 09:20:58][flashy.solver][INFO] - Valid | Epoch 48 | 14/79 | 0.21 it/sec | ce 0.199 | ppl 1.223
[05-26 09:21:31][flashy.solver][INFO] - Valid | Epoch 48 | 21/79 | 0.21 it/sec | ce 0.239 | ppl 1.272
[05-26 09:22:03][flashy.solver][INFO] - Valid | Epoch 48 | 28/79 | 0.21 it/sec | ce 0.204 | ppl 1.229
[05-26 09:22:36][flashy.solver][INFO] - Valid | Epoch 48 | 35/79 | 0.21 it/sec | ce 0.235 | ppl 1.267
[05-26 09:23:09][flashy.solver][INFO] - Valid | Epoch 48 | 42/79 | 0.21 it/sec | ce 0.203 | ppl 1.227
[05-26 09:23:43][flashy.solver][INFO] - Valid | Epoch 48 | 49/79 | 0.21 it/sec | ce 0.274 | ppl 1.318
[05-26 09:24:16][flashy.solver][INFO] - Valid | Epoch 48 | 56/79 | 0.21 it/sec | ce 0.206 | ppl 1.229
[05-26 09:24:49][flashy.solver][INFO] - Valid | Epoch 48 | 63/79 | 0.21 it/sec | ce 0.259 | ppl 1.297
[05-26 09:25:22][flashy.solver][INFO] - Valid | Epoch 48 | 70/79 | 0.21 it/sec | ce 0.240 | ppl 1.273
[05-26 09:25:55][flashy.solver][INFO] - Valid | Epoch 48 | 77/79 | 0.21 it/sec | ce 0.275 | ppl 1.318
[05-26 09:25:56][flashy.solver][INFO] - Valid Summary | Epoch 48 | ce=0.231 | ppl=1.262 | duration=368.529
[05-26 09:25:56][flashy.solver][INFO] - New best state with ce=0.231 (was 0.231)
[05-26 09:25:56][flashy.solver][INFO] - Model hash: fd432eda6e8333f8f90f0494ac063c5d1525a6eb
[05-26 09:25:56][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 09:42:16][flashy.solver][INFO] - Train | Epoch 49 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.320E-01 | ce 0.237 | ppl 1.270
[05-26 09:58:34][flashy.solver][INFO] - Train | Epoch 49 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.495E-01 | ce 0.239 | ppl 1.272
[05-26 10:14:48][flashy.solver][INFO] - Train | Epoch 49 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.040E-01 | ce 0.233 | ppl 1.264
[05-26 10:30:54][flashy.solver][INFO] - Train | Epoch 49 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.359E-01 | ce 0.235 | ppl 1.268
[05-26 10:47:18][flashy.solver][INFO] - Train | Epoch 49 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.492E-01 | ce 0.241 | ppl 1.276
[05-26 11:03:38][flashy.solver][INFO] - Train | Epoch 49 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.113E-01 | ce 0.242 | ppl 1.276
[05-26 11:20:02][flashy.solver][INFO] - Train | Epoch 49 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.166E-01 | ce 0.246 | ppl 1.281
[05-26 11:36:05][flashy.solver][INFO] - Train | Epoch 49 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.639E-01 | ce 0.240 | ppl 1.274
[05-26 11:52:02][flashy.solver][INFO] - Train | Epoch 49 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.295E-01 | ce 0.238 | ppl 1.271
[05-26 12:07:55][flashy.solver][INFO] - Train Summary | Epoch 49 | lr=1.00E-04 | grad_norm=7.122E-01 | ce=0.239 | ppl=1.273 | duration=9719.254
[05-26 12:08:32][flashy.solver][INFO] - Valid | Epoch 49 | 7/79 | 0.22 it/sec | ce 0.215 | ppl 1.242
[05-26 12:09:04][flashy.solver][INFO] - Valid | Epoch 49 | 14/79 | 0.22 it/sec | ce 0.199 | ppl 1.223
[05-26 12:09:36][flashy.solver][INFO] - Valid | Epoch 49 | 21/79 | 0.22 it/sec | ce 0.238 | ppl 1.270
[05-26 12:10:08][flashy.solver][INFO] - Valid | Epoch 49 | 28/79 | 0.22 it/sec | ce 0.204 | ppl 1.229
[05-26 12:10:40][flashy.solver][INFO] - Valid | Epoch 49 | 35/79 | 0.22 it/sec | ce 0.234 | ppl 1.265
[05-26 12:11:13][flashy.solver][INFO] - Valid | Epoch 49 | 42/79 | 0.22 it/sec | ce 0.202 | ppl 1.226
[05-26 12:11:45][flashy.solver][INFO] - Valid | Epoch 49 | 49/79 | 0.22 it/sec | ce 0.273 | ppl 1.317
[05-26 12:12:18][flashy.solver][INFO] - Valid | Epoch 49 | 56/79 | 0.22 it/sec | ce 0.205 | ppl 1.228
[05-26 12:12:50][flashy.solver][INFO] - Valid | Epoch 49 | 63/79 | 0.22 it/sec | ce 0.259 | ppl 1.296
[05-26 12:13:22][flashy.solver][INFO] - Valid | Epoch 49 | 70/79 | 0.22 it/sec | ce 0.240 | ppl 1.272
[05-26 12:13:55][flashy.solver][INFO] - Valid | Epoch 49 | 77/79 | 0.22 it/sec | ce 0.274 | ppl 1.317
[05-26 12:13:55][flashy.solver][INFO] - Valid Summary | Epoch 49 | ce=0.230 | ppl=1.261 | duration=360.200
[05-26 12:13:55][flashy.solver][INFO] - New best state with ce=0.230 (was 0.231)
[05-26 12:13:55][flashy.solver][INFO] - Model hash: d8f8c2d895976c03be82dcb14b842397a3e31f5b
[05-26 12:13:55][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 12:29:53][flashy.solver][INFO] - Train | Epoch 50 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.751E-01 | ce 0.234 | ppl 1.266
[05-26 12:45:45][flashy.solver][INFO] - Train | Epoch 50 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.688E-01 | ce 0.240 | ppl 1.274
[05-26 13:01:39][flashy.solver][INFO] - Train | Epoch 50 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.274E-01 | ce 0.237 | ppl 1.269
[05-26 13:17:35][flashy.solver][INFO] - Train | Epoch 50 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.640E-01 | ce 0.242 | ppl 1.276
[05-26 13:33:48][flashy.solver][INFO] - Train | Epoch 50 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.346E-01 | ce 0.243 | ppl 1.277
[05-26 13:50:56][flashy.solver][INFO] - Train | Epoch 50 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.505E-01 | ce 0.240 | ppl 1.274
[05-26 14:07:47][flashy.solver][INFO] - Train | Epoch 50 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.486E-01 | ce 0.232 | ppl 1.264
[05-26 14:24:27][flashy.solver][INFO] - Train | Epoch 50 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.159E-01 | ce 0.232 | ppl 1.263
[05-26 14:40:22][flashy.solver][INFO] - Train | Epoch 50 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.417E-01 | ce 0.232 | ppl 1.264
[05-26 14:56:28][flashy.solver][INFO] - Train Summary | Epoch 50 | lr=1.00E-04 | grad_norm=7.181E-01 | ce=0.237 | ppl=1.270 | duration=9752.171
[05-26 14:57:04][flashy.solver][INFO] - Valid | Epoch 50 | 7/79 | 0.22 it/sec | ce 0.215 | ppl 1.241
[05-26 14:57:36][flashy.solver][INFO] - Valid | Epoch 50 | 14/79 | 0.22 it/sec | ce 0.199 | ppl 1.222
[05-26 14:58:09][flashy.solver][INFO] - Valid | Epoch 50 | 21/79 | 0.22 it/sec | ce 0.238 | ppl 1.270
[05-26 14:58:41][flashy.solver][INFO] - Valid | Epoch 50 | 28/79 | 0.22 it/sec | ce 0.203 | ppl 1.229
[05-26 14:59:13][flashy.solver][INFO] - Valid | Epoch 50 | 35/79 | 0.22 it/sec | ce 0.234 | ppl 1.265
[05-26 14:59:45][flashy.solver][INFO] - Valid | Epoch 50 | 42/79 | 0.22 it/sec | ce 0.202 | ppl 1.225
[05-26 15:00:17][flashy.solver][INFO] - Valid | Epoch 50 | 49/79 | 0.22 it/sec | ce 0.273 | ppl 1.316
[05-26 15:00:50][flashy.solver][INFO] - Valid | Epoch 50 | 56/79 | 0.22 it/sec | ce 0.205 | ppl 1.228
[05-26 15:01:22][flashy.solver][INFO] - Valid | Epoch 50 | 63/79 | 0.22 it/sec | ce 0.258 | ppl 1.295
[05-26 15:01:54][flashy.solver][INFO] - Valid | Epoch 50 | 70/79 | 0.22 it/sec | ce 0.239 | ppl 1.272
[05-26 15:02:27][flashy.solver][INFO] - Valid | Epoch 50 | 77/79 | 0.22 it/sec | ce 0.274 | ppl 1.316
[05-26 15:02:27][flashy.solver][INFO] - Valid Summary | Epoch 50 | ce=0.230 | ppl=1.261 | duration=359.721
[05-26 15:02:27][flashy.solver][INFO] - New best state with ce=0.230 (was 0.230)
[05-26 15:02:27][flashy.solver][INFO] - Evaluate Summary | Epoch 50 | duration=0.000
[05-26 15:02:27][flashy.solver][INFO] - Generating samples in /tmp/audiocraft_root/xps/a4911d1c/samples
[05-26 15:02:34][flashy.solver][INFO] - Generate Summary | Epoch 50 | rtf=0.134 | duration=6.334
[05-26 15:02:34][flashy.solver][INFO] - Model hash: 04f565585da273b6a85f9f42feb18186074dbf7e
[05-26 15:02:34][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 15:19:02][flashy.solver][INFO] - Train | Epoch 51 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.134E-01 | ce 0.240 | ppl 1.274
[05-26 15:35:20][flashy.solver][INFO] - Train | Epoch 51 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.048E-01 | ce 0.241 | ppl 1.275
[05-26 15:51:49][flashy.solver][INFO] - Train | Epoch 51 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.435E-01 | ce 0.243 | ppl 1.277
[05-26 16:08:19][flashy.solver][INFO] - Train | Epoch 51 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.547E-01 | ce 0.239 | ppl 1.272
[05-26 16:24:38][flashy.solver][INFO] - Train | Epoch 51 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.223E-01 | ce 0.246 | ppl 1.281
[05-26 16:40:34][flashy.solver][INFO] - Train | Epoch 51 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.070E-01 | ce 0.246 | ppl 1.281
[05-26 16:56:30][flashy.solver][INFO] - Train | Epoch 51 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.519E-01 | ce 0.247 | ppl 1.282
[05-26 17:12:25][flashy.solver][INFO] - Train | Epoch 51 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.257E-01 | ce 0.248 | ppl 1.283
[05-26 17:28:19][flashy.solver][INFO] - Train | Epoch 51 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.490E-01 | ce 0.236 | ppl 1.268
[05-26 17:44:14][flashy.solver][INFO] - Train Summary | Epoch 51 | lr=1.00E-04 | grad_norm=7.459E-01 | ce=0.243 | ppl=1.277 | duration=9700.451
[05-26 17:44:51][flashy.solver][INFO] - Valid | Epoch 51 | 7/79 | 0.22 it/sec | ce 0.215 | ppl 1.241
[05-26 17:45:23][flashy.solver][INFO] - Valid | Epoch 51 | 14/79 | 0.22 it/sec | ce 0.199 | ppl 1.222
[05-26 17:45:56][flashy.solver][INFO] - Valid | Epoch 51 | 21/79 | 0.22 it/sec | ce 0.238 | ppl 1.270
[05-26 17:46:28][flashy.solver][INFO] - Valid | Epoch 51 | 28/79 | 0.22 it/sec | ce 0.203 | ppl 1.228
[05-26 17:47:00][flashy.solver][INFO] - Valid | Epoch 51 | 35/79 | 0.22 it/sec | ce 0.234 | ppl 1.265
[05-26 17:47:32][flashy.solver][INFO] - Valid | Epoch 51 | 42/79 | 0.22 it/sec | ce 0.202 | ppl 1.225
[05-26 17:48:05][flashy.solver][INFO] - Valid | Epoch 51 | 49/79 | 0.22 it/sec | ce 0.272 | ppl 1.316
[05-26 17:48:37][flashy.solver][INFO] - Valid | Epoch 51 | 56/79 | 0.22 it/sec | ce 0.204 | ppl 1.228
[05-26 17:49:10][flashy.solver][INFO] - Valid | Epoch 51 | 63/79 | 0.22 it/sec | ce 0.258 | ppl 1.295
[05-26 17:49:42][flashy.solver][INFO] - Valid | Epoch 51 | 70/79 | 0.22 it/sec | ce 0.239 | ppl 1.271
[05-26 17:50:14][flashy.solver][INFO] - Valid | Epoch 51 | 77/79 | 0.22 it/sec | ce 0.273 | ppl 1.315
[05-26 17:50:15][flashy.solver][INFO] - Valid Summary | Epoch 51 | ce=0.230 | ppl=1.260 | duration=360.665
[05-26 17:50:15][flashy.solver][INFO] - New best state with ce=0.230 (was 0.230)
[05-26 17:50:15][flashy.solver][INFO] - Model hash: 84e1d37faa7af97650e56fa7d9cdc1f99834a314
[05-26 17:50:15][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 18:06:14][flashy.solver][INFO] - Train | Epoch 52 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.172E-01 | ce 0.234 | ppl 1.266
[05-26 18:22:05][flashy.solver][INFO] - Train | Epoch 52 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.931E-01 | ce 0.236 | ppl 1.268
[05-26 18:37:54][flashy.solver][INFO] - Train | Epoch 52 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.362E-01 | ce 0.238 | ppl 1.271
[05-26 18:53:46][flashy.solver][INFO] - Train | Epoch 52 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.092E-01 | ce 0.237 | ppl 1.270
[05-26 19:09:37][flashy.solver][INFO] - Train | Epoch 52 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.181E-01 | ce 0.244 | ppl 1.278
[05-26 19:25:29][flashy.solver][INFO] - Train | Epoch 52 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.673E-01 | ce 0.242 | ppl 1.277
[05-26 19:41:23][flashy.solver][INFO] - Train | Epoch 52 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.286E-01 | ce 0.242 | ppl 1.276
[05-26 19:57:17][flashy.solver][INFO] - Train | Epoch 52 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.105E-01 | ce 0.251 | ppl 1.288
[05-26 20:13:11][flashy.solver][INFO] - Train | Epoch 52 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.209E-01 | ce 0.239 | ppl 1.273
[05-26 20:29:04][flashy.solver][INFO] - Train Summary | Epoch 52 | lr=1.00E-04 | grad_norm=7.595E-01 | ce=0.241 | ppl=1.274 | duration=9529.466
[05-26 20:29:41][flashy.solver][INFO] - Valid | Epoch 52 | 7/79 | 0.22 it/sec | ce 0.214 | ppl 1.241
[05-26 20:30:13][flashy.solver][INFO] - Valid | Epoch 52 | 14/79 | 0.22 it/sec | ce 0.198 | ppl 1.222
[05-26 20:30:46][flashy.solver][INFO] - Valid | Epoch 52 | 21/79 | 0.22 it/sec | ce 0.237 | ppl 1.269
[05-26 20:31:17][flashy.solver][INFO] - Valid | Epoch 52 | 28/79 | 0.22 it/sec | ce 0.203 | ppl 1.228
[05-26 20:31:50][flashy.solver][INFO] - Valid | Epoch 52 | 35/79 | 0.22 it/sec | ce 0.233 | ppl 1.264
[05-26 20:32:22][flashy.solver][INFO] - Valid | Epoch 52 | 42/79 | 0.22 it/sec | ce 0.202 | ppl 1.225
[05-26 20:32:54][flashy.solver][INFO] - Valid | Epoch 52 | 49/79 | 0.22 it/sec | ce 0.272 | ppl 1.316
[05-26 20:33:27][flashy.solver][INFO] - Valid | Epoch 52 | 56/79 | 0.22 it/sec | ce 0.204 | ppl 1.228
[05-26 20:33:59][flashy.solver][INFO] - Valid | Epoch 52 | 63/79 | 0.22 it/sec | ce 0.258 | ppl 1.295
[05-26 20:34:31][flashy.solver][INFO] - Valid | Epoch 52 | 70/79 | 0.22 it/sec | ce 0.239 | ppl 1.271
[05-26 20:35:04][flashy.solver][INFO] - Valid | Epoch 52 | 77/79 | 0.22 it/sec | ce 0.273 | ppl 1.315
[05-26 20:35:05][flashy.solver][INFO] - Valid Summary | Epoch 52 | ce=0.229 | ppl=1.260 | duration=360.312
[05-26 20:35:05][flashy.solver][INFO] - New best state with ce=0.229 (was 0.230)
[05-26 20:35:05][flashy.solver][INFO] - Model hash: e688fb0490b3eb3f95bbf66898f1efc5e7ff2c89
[05-26 20:35:05][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 20:51:04][flashy.solver][INFO] - Train | Epoch 53 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.337E-01 | ce 0.239 | ppl 1.272
[05-26 21:07:00][flashy.solver][INFO] - Train | Epoch 53 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.014E-01 | ce 0.237 | ppl 1.270
[05-26 21:22:56][flashy.solver][INFO] - Train | Epoch 53 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.486E-01 | ce 0.247 | ppl 1.283
[05-26 21:38:47][flashy.solver][INFO] - Train | Epoch 53 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.526E-01 | ce 0.245 | ppl 1.280
[05-26 21:54:38][flashy.solver][INFO] - Train | Epoch 53 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.894E-01 | ce 0.236 | ppl 1.269
[05-26 22:10:33][flashy.solver][INFO] - Train | Epoch 53 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.514E-01 | ce 0.240 | ppl 1.274
[05-26 22:26:30][flashy.solver][INFO] - Train | Epoch 53 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.293E-01 | ce 0.233 | ppl 1.264
[05-26 22:42:21][flashy.solver][INFO] - Train | Epoch 53 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.660E-01 | ce 0.245 | ppl 1.280
[05-26 22:58:12][flashy.solver][INFO] - Train | Epoch 53 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.559E-01 | ce 0.238 | ppl 1.272
[05-26 23:14:04][flashy.solver][INFO] - Train Summary | Epoch 53 | lr=1.00E-04 | grad_norm=7.351E-01 | ce=0.239 | ppl=1.273 | duration=9539.188
[05-26 23:14:41][flashy.solver][INFO] - Valid | Epoch 53 | 7/79 | 0.22 it/sec | ce 0.214 | ppl 1.240
[05-26 23:15:13][flashy.solver][INFO] - Valid | Epoch 53 | 14/79 | 0.22 it/sec | ce 0.198 | ppl 1.221
[05-26 23:15:45][flashy.solver][INFO] - Valid | Epoch 53 | 21/79 | 0.22 it/sec | ce 0.237 | ppl 1.268
[05-26 23:16:17][flashy.solver][INFO] - Valid | Epoch 53 | 28/79 | 0.22 it/sec | ce 0.202 | ppl 1.227
[05-26 23:16:49][flashy.solver][INFO] - Valid | Epoch 53 | 35/79 | 0.22 it/sec | ce 0.233 | ppl 1.264
[05-26 23:17:21][flashy.solver][INFO] - Valid | Epoch 53 | 42/79 | 0.22 it/sec | ce 0.201 | ppl 1.224
[05-26 23:17:54][flashy.solver][INFO] - Valid | Epoch 53 | 49/79 | 0.22 it/sec | ce 0.271 | ppl 1.315
[05-26 23:18:26][flashy.solver][INFO] - Valid | Epoch 53 | 56/79 | 0.22 it/sec | ce 0.204 | ppl 1.227
[05-26 23:18:59][flashy.solver][INFO] - Valid | Epoch 53 | 63/79 | 0.22 it/sec | ce 0.257 | ppl 1.294
[05-26 23:19:31][flashy.solver][INFO] - Valid | Epoch 53 | 70/79 | 0.22 it/sec | ce 0.238 | ppl 1.270
[05-26 23:20:03][flashy.solver][INFO] - Valid | Epoch 53 | 77/79 | 0.22 it/sec | ce 0.272 | ppl 1.314
[05-26 23:20:04][flashy.solver][INFO] - Valid Summary | Epoch 53 | ce=0.229 | ppl=1.259 | duration=359.881
[05-26 23:20:04][flashy.solver][INFO] - New best state with ce=0.229 (was 0.229)
[05-26 23:20:04][flashy.solver][INFO] - Model hash: fd90381b3e1902f5ca42c62ad9f6b5e2b2dbba4f
[05-26 23:20:04][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-26 23:36:02][flashy.solver][INFO] - Train | Epoch 54 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.911E-01 | ce 0.237 | ppl 1.270
[05-26 23:51:54][flashy.solver][INFO] - Train | Epoch 54 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.572E-01 | ce 0.240 | ppl 1.274
[05-27 00:07:47][flashy.solver][INFO] - Train | Epoch 54 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.517E-01 | ce 0.236 | ppl 1.269
[05-27 00:23:40][flashy.solver][INFO] - Train | Epoch 54 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.862E-01 | ce 0.247 | ppl 1.283
[05-27 00:39:33][flashy.solver][INFO] - Train | Epoch 54 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.373E-01 | ce 0.247 | ppl 1.283
[05-27 00:55:26][flashy.solver][INFO] - Train | Epoch 54 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.278E-01 | ce 0.239 | ppl 1.272
[05-27 01:11:24][flashy.solver][INFO] - Train | Epoch 54 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.973E-01 | ce 0.232 | ppl 1.264
[05-27 01:27:21][flashy.solver][INFO] - Train | Epoch 54 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.145E-01 | ce 0.232 | ppl 1.263
[05-27 01:43:15][flashy.solver][INFO] - Train | Epoch 54 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.093E-01 | ce 0.239 | ppl 1.272
[05-27 01:59:07][flashy.solver][INFO] - Train Summary | Epoch 54 | lr=1.00E-04 | grad_norm=7.549E-01 | ce=0.239 | ppl=1.272 | duration=9542.972
[05-27 01:59:44][flashy.solver][INFO] - Valid | Epoch 54 | 7/79 | 0.22 it/sec | ce 0.213 | ppl 1.239
[05-27 02:00:16][flashy.solver][INFO] - Valid | Epoch 54 | 14/79 | 0.22 it/sec | ce 0.197 | ppl 1.220
[05-27 02:00:49][flashy.solver][INFO] - Valid | Epoch 54 | 21/79 | 0.22 it/sec | ce 0.236 | ppl 1.268
[05-27 02:01:21][flashy.solver][INFO] - Valid | Epoch 54 | 28/79 | 0.22 it/sec | ce 0.202 | ppl 1.227
[05-27 02:01:54][flashy.solver][INFO] - Valid | Epoch 54 | 35/79 | 0.21 it/sec | ce 0.233 | ppl 1.263
[05-27 02:02:27][flashy.solver][INFO] - Valid | Epoch 54 | 42/79 | 0.21 it/sec | ce 0.201 | ppl 1.224
[05-27 02:03:01][flashy.solver][INFO] - Valid | Epoch 54 | 49/79 | 0.21 it/sec | ce 0.270 | ppl 1.314
[05-27 02:03:34][flashy.solver][INFO] - Valid | Epoch 54 | 56/79 | 0.21 it/sec | ce 0.203 | ppl 1.226
[05-27 02:04:08][flashy.solver][INFO] - Valid | Epoch 54 | 63/79 | 0.21 it/sec | ce 0.256 | ppl 1.293
[05-27 02:04:40][flashy.solver][INFO] - Valid | Epoch 54 | 70/79 | 0.21 it/sec | ce 0.237 | ppl 1.269
[05-27 02:05:14][flashy.solver][INFO] - Valid | Epoch 54 | 77/79 | 0.21 it/sec | ce 0.272 | ppl 1.313
[05-27 02:05:15][flashy.solver][INFO] - Valid Summary | Epoch 54 | ce=0.228 | ppl=1.258 | duration=368.039
[05-27 02:05:15][flashy.solver][INFO] - New best state with ce=0.228 (was 0.229)
[05-27 02:05:15][flashy.solver][INFO] - Model hash: 43f552eef01fd32a263ff40880e33638d3339981
[05-27 02:05:15][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 02:21:19][flashy.solver][INFO] - Train | Epoch 55 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.202E-01 | ce 0.238 | ppl 1.271
[05-27 02:37:12][flashy.solver][INFO] - Train | Epoch 55 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 6.991E-01 | ce 0.236 | ppl 1.268
[05-27 02:53:03][flashy.solver][INFO] - Train | Epoch 55 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.696E-01 | ce 0.242 | ppl 1.276
[05-27 03:08:55][flashy.solver][INFO] - Train | Epoch 55 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.320E-01 | ce 0.235 | ppl 1.267
[05-27 03:24:49][flashy.solver][INFO] - Train | Epoch 55 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.800E-01 | ce 0.240 | ppl 1.273
[05-27 03:40:42][flashy.solver][INFO] - Train | Epoch 55 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.138E-01 | ce 0.236 | ppl 1.269
[05-27 03:56:38][flashy.solver][INFO] - Train | Epoch 55 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.901E-01 | ce 0.237 | ppl 1.270
[05-27 04:12:31][flashy.solver][INFO] - Train | Epoch 55 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.795E-01 | ce 0.238 | ppl 1.271
[05-27 04:28:28][flashy.solver][INFO] - Train | Epoch 55 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.949E-01 | ce 0.235 | ppl 1.267
[05-27 04:44:55][flashy.solver][INFO] - Train Summary | Epoch 55 | lr=1.00E-04 | grad_norm=7.593E-01 | ce=0.237 | ppl=1.269 | duration=9580.192
[05-27 04:45:34][flashy.solver][INFO] - Valid | Epoch 55 | 7/79 | 0.20 it/sec | ce 0.213 | ppl 1.239
[05-27 04:46:08][flashy.solver][INFO] - Valid | Epoch 55 | 14/79 | 0.20 it/sec | ce 0.197 | ppl 1.220
[05-27 04:46:43][flashy.solver][INFO] - Valid | Epoch 55 | 21/79 | 0.20 it/sec | ce 0.236 | ppl 1.267
[05-27 04:47:17][flashy.solver][INFO] - Valid | Epoch 55 | 28/79 | 0.20 it/sec | ce 0.202 | ppl 1.226
[05-27 04:47:50][flashy.solver][INFO] - Valid | Epoch 55 | 35/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-27 04:48:23][flashy.solver][INFO] - Valid | Epoch 55 | 42/79 | 0.21 it/sec | ce 0.200 | ppl 1.223
[05-27 04:48:58][flashy.solver][INFO] - Valid | Epoch 55 | 49/79 | 0.21 it/sec | ce 0.270 | ppl 1.313
[05-27 04:49:32][flashy.solver][INFO] - Valid | Epoch 55 | 56/79 | 0.21 it/sec | ce 0.203 | ppl 1.226
[05-27 04:50:07][flashy.solver][INFO] - Valid | Epoch 55 | 63/79 | 0.21 it/sec | ce 0.256 | ppl 1.293
[05-27 04:50:41][flashy.solver][INFO] - Valid | Epoch 55 | 70/79 | 0.21 it/sec | ce 0.237 | ppl 1.269
[05-27 04:51:15][flashy.solver][INFO] - Valid | Epoch 55 | 77/79 | 0.21 it/sec | ce 0.271 | ppl 1.312
[05-27 04:51:16][flashy.solver][INFO] - Valid Summary | Epoch 55 | ce=0.228 | ppl=1.258 | duration=380.776
[05-27 04:51:16][flashy.solver][INFO] - New best state with ce=0.228 (was 0.228)
[05-27 04:51:16][flashy.solver][INFO] - Model hash: bcc625252a2d3e5fb2173d9c7f070b45b1ecc816
[05-27 04:51:16][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 05:08:13][flashy.solver][INFO] - Train | Epoch 56 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.560E-01 | ce 0.240 | ppl 1.273
[05-27 05:25:01][flashy.solver][INFO] - Train | Epoch 56 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.211E-01 | ce 0.235 | ppl 1.267
[05-27 05:41:57][flashy.solver][INFO] - Train | Epoch 56 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.640E-01 | ce 0.237 | ppl 1.269
[05-27 05:58:10][flashy.solver][INFO] - Train | Epoch 56 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.752E-01 | ce 0.237 | ppl 1.269
[05-27 06:14:18][flashy.solver][INFO] - Train | Epoch 56 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.932E-01 | ce 0.238 | ppl 1.271
[05-27 06:24:39][dora.distrib][INFO] - world_size is 1, skipping init.
[05-27 06:24:39][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-27 06:24:39][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-27 06:24:39][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-27 06:24:39][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-27 06:24:39][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-27 06:24:39][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-27 06:24:39][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-27 06:24:40][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-27 06:24:40][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-27 06:24:41][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-27 06:24:41][flashy.solver][INFO] - No EMA on the model.
[05-27 06:24:41][flashy.solver][INFO] - Model size: 1.21 M params
[05-27 06:24:41][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-27 06:24:41][flashy.solver][INFO] - Restoring weights and history.
[05-27 06:24:41][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 06:24:41][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 06:24:41][flashy.solver][INFO] - Model hash: bcc625252a2d3e5fb2173d9c7f070b45b1ecc816
[05-27 06:24:41][flashy.solver][INFO] - Replaying past metrics...
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 16 | lr=1.00E-04 | grad_norm=5.373E-01 | ce=0.274 | ppl=1.318 | duration=9552.291
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 16 | ce=0.265 | ppl=1.306 | duration=362.290
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 17 | lr=1.00E-04 | grad_norm=5.495E-01 | ce=0.274 | ppl=1.318 | duration=9593.252
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 17 | ce=0.263 | ppl=1.303 | duration=361.603
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 18 | lr=1.00E-04 | grad_norm=5.303E-01 | ce=0.270 | ppl=1.314 | duration=9601.733
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 18 | ce=0.261 | ppl=1.301 | duration=362.833
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 19 | lr=1.00E-04 | grad_norm=5.436E-01 | ce=0.271 | ppl=1.314 | duration=9603.793
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 19 | ce=0.259 | ppl=1.298 | duration=362.807
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 20 | lr=1.00E-04 | grad_norm=5.256E-01 | ce=0.267 | ppl=1.309 | duration=9513.591
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 20 | ce=0.256 | ppl=1.295 | duration=359.164
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 21 | lr=1.00E-04 | grad_norm=5.398E-01 | ce=0.267 | ppl=1.309 | duration=9521.433
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 21 | ce=0.255 | ppl=1.293 | duration=359.736
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 22 | lr=1.00E-04 | grad_norm=5.273E-01 | ce=0.264 | ppl=1.305 | duration=9522.932
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 22 | ce=0.254 | ppl=1.291 | duration=359.952
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 23 | lr=1.00E-04 | grad_norm=5.359E-01 | ce=0.263 | ppl=1.304 | duration=9520.848
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 23 | ce=0.252 | ppl=1.290 | duration=359.946
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 24 | lr=1.00E-04 | grad_norm=5.356E-01 | ce=0.260 | ppl=1.300 | duration=9517.164
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 24 | ce=0.250 | ppl=1.287 | duration=360.223
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 25 | lr=1.00E-04 | grad_norm=5.350E-01 | ce=0.258 | ppl=1.297 | duration=9526.316
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 25 | ce=0.249 | ppl=1.286 | duration=359.858
[05-27 06:24:41][flashy.solver][INFO] - Evaluate Summary | Epoch 25 | duration=0.000
[05-27 06:24:41][flashy.solver][INFO] - Generate Summary | Epoch 25 | rtf=0.143 | duration=7.398
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 26 | lr=1.00E-04 | grad_norm=5.344E-01 | ce=0.255 | ppl=1.294 | duration=9505.304
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 26 | ce=0.248 | ppl=1.284 | duration=359.841
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 27 | lr=1.00E-04 | grad_norm=5.476E-01 | ce=0.256 | ppl=1.294 | duration=9515.969
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 27 | ce=0.247 | ppl=1.282 | duration=359.513
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 28 | lr=1.00E-04 | grad_norm=5.444E-01 | ce=0.254 | ppl=1.292 | duration=9521.965
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 28 | ce=0.245 | ppl=1.281 | duration=360.116
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 29 | lr=1.00E-04 | grad_norm=5.705E-01 | ce=0.255 | ppl=1.293 | duration=9536.583
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 29 | ce=0.245 | ppl=1.280 | duration=361.077
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 30 | lr=1.00E-04 | grad_norm=5.520E-01 | ce=0.253 | ppl=1.291 | duration=9517.157
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 30 | ce=0.243 | ppl=1.278 | duration=360.175
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 31 | lr=1.00E-04 | grad_norm=5.536E-01 | ce=0.252 | ppl=1.290 | duration=9515.999
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 31 | ce=0.242 | ppl=1.277 | duration=359.757
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 32 | lr=1.00E-04 | grad_norm=5.734E-01 | ce=0.251 | ppl=1.288 | duration=9525.455
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 32 | ce=0.241 | ppl=1.275 | duration=359.850
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 33 | lr=1.00E-04 | grad_norm=5.841E-01 | ce=0.252 | ppl=1.289 | duration=9522.251
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 33 | ce=0.240 | ppl=1.274 | duration=360.855
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 34 | lr=1.00E-04 | grad_norm=5.932E-01 | ce=0.250 | ppl=1.287 | duration=9530.122
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 34 | ce=0.240 | ppl=1.273 | duration=360.152
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 35 | lr=1.00E-04 | grad_norm=5.957E-01 | ce=0.248 | ppl=1.284 | duration=9524.631
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 35 | ce=0.239 | ppl=1.272 | duration=359.587
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 36 | lr=1.00E-04 | grad_norm=6.259E-01 | ce=0.251 | ppl=1.288 | duration=9531.088
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 36 | ce=0.239 | ppl=1.272 | duration=360.096
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 37 | lr=1.00E-04 | grad_norm=6.270E-01 | ce=0.248 | ppl=1.284 | duration=9533.307
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 37 | ce=0.237 | ppl=1.270 | duration=360.546
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 38 | lr=1.00E-04 | grad_norm=6.272E-01 | ce=0.248 | ppl=1.284 | duration=9529.414
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 38 | ce=0.237 | ppl=1.270 | duration=360.767
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 39 | lr=1.00E-04 | grad_norm=6.386E-01 | ce=0.246 | ppl=1.281 | duration=9524.562
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 39 | ce=0.236 | ppl=1.268 | duration=360.385
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 40 | lr=1.00E-04 | grad_norm=6.360E-01 | ce=0.244 | ppl=1.279 | duration=9642.435
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 40 | ce=0.235 | ppl=1.267 | duration=385.528
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 41 | lr=1.00E-04 | grad_norm=6.410E-01 | ce=0.246 | ppl=1.281 | duration=9668.847
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 41 | ce=0.235 | ppl=1.267 | duration=363.285
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 42 | lr=1.00E-04 | grad_norm=6.642E-01 | ce=0.244 | ppl=1.278 | duration=9572.143
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 42 | ce=0.234 | ppl=1.266 | duration=362.071
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 43 | lr=1.00E-04 | grad_norm=6.578E-01 | ce=0.244 | ppl=1.278 | duration=9564.111
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 43 | ce=0.234 | ppl=1.265 | duration=362.399
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 44 | lr=1.00E-04 | grad_norm=6.778E-01 | ce=0.244 | ppl=1.279 | duration=9545.633
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 44 | ce=0.233 | ppl=1.264 | duration=360.226
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 45 | lr=1.00E-04 | grad_norm=6.707E-01 | ce=0.240 | ppl=1.273 | duration=9558.510
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 45 | ce=0.232 | ppl=1.263 | duration=361.292
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 46 | lr=1.00E-04 | grad_norm=7.081E-01 | ce=0.243 | ppl=1.277 | duration=9549.454
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 46 | ce=0.232 | ppl=1.263 | duration=361.476
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 47 | lr=1.00E-04 | grad_norm=6.983E-01 | ce=0.241 | ppl=1.275 | duration=9994.908
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 47 | ce=0.231 | ppl=1.262 | duration=380.415
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 48 | lr=1.00E-04 | grad_norm=7.072E-01 | ce=0.241 | ppl=1.274 | duration=9756.736
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 48 | ce=0.231 | ppl=1.262 | duration=368.529
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 49 | lr=1.00E-04 | grad_norm=7.122E-01 | ce=0.239 | ppl=1.273 | duration=9719.254
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 49 | ce=0.230 | ppl=1.261 | duration=360.200
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 50 | lr=1.00E-04 | grad_norm=7.181E-01 | ce=0.237 | ppl=1.270 | duration=9752.171
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 50 | ce=0.230 | ppl=1.261 | duration=359.721
[05-27 06:24:41][flashy.solver][INFO] - Evaluate Summary | Epoch 50 | duration=0.000
[05-27 06:24:41][flashy.solver][INFO] - Generate Summary | Epoch 50 | rtf=0.134 | duration=6.334
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 51 | lr=1.00E-04 | grad_norm=7.459E-01 | ce=0.243 | ppl=1.277 | duration=9700.451
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 51 | ce=0.230 | ppl=1.260 | duration=360.665
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 52 | lr=1.00E-04 | grad_norm=7.595E-01 | ce=0.241 | ppl=1.274 | duration=9529.466
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 52 | ce=0.229 | ppl=1.260 | duration=360.312
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 53 | lr=1.00E-04 | grad_norm=7.351E-01 | ce=0.239 | ppl=1.273 | duration=9539.188
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 53 | ce=0.229 | ppl=1.259 | duration=359.881
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 54 | lr=1.00E-04 | grad_norm=7.549E-01 | ce=0.239 | ppl=1.272 | duration=9542.972
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 54 | ce=0.228 | ppl=1.258 | duration=368.039
[05-27 06:24:41][flashy.solver][INFO] - Train Summary | Epoch 55 | lr=1.00E-04 | grad_norm=7.593E-01 | ce=0.237 | ppl=1.269 | duration=9580.192
[05-27 06:24:41][flashy.solver][INFO] - Valid Summary | Epoch 55 | ce=0.228 | ppl=1.258 | duration=380.776
[05-27 06:24:47][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-27 06:31:21][flashy.solver][INFO] - Train | Epoch 56 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.696E-01 | ce 0.235 | ppl 1.267
[05-27 06:42:04][flashy.solver][INFO] - Train | Epoch 56 | 200/2000 | 0.19 it/sec | lr 1.00E-04 | grad_norm 7.462E-01 | ce 0.237 | ppl 1.270
[05-27 06:48:41][flashy.solver][INFO] - Train | Epoch 56 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.246E-01 | ce 0.243 | ppl 1.277
[05-27 06:59:37][flashy.solver][INFO] - Train | Epoch 56 | 400/2000 | 0.19 it/sec | lr 1.00E-04 | grad_norm 7.431E-01 | ce 0.241 | ppl 1.274
[05-27 07:05:56][flashy.solver][INFO] - Train | Epoch 56 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.590E-01 | ce 0.234 | ppl 1.266
[05-27 07:16:45][flashy.solver][INFO] - Train | Epoch 56 | 600/2000 | 0.19 it/sec | lr 1.00E-04 | grad_norm 7.973E-01 | ce 0.232 | ppl 1.263
[05-27 07:23:02][flashy.solver][INFO] - Train | Epoch 56 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.321E-01 | ce 0.236 | ppl 1.269
[05-27 07:33:20][flashy.solver][INFO] - Train | Epoch 56 | 800/2000 | 0.19 it/sec | lr 1.00E-04 | grad_norm 7.749E-01 | ce 0.246 | ppl 1.280
[05-27 07:39:33][flashy.solver][INFO] - Train Summary | Epoch 56 | lr=1.00E-04 | grad_norm=7.741E-01 | ce=0.237 | ppl=1.270 | duration=10097.090
[05-27 07:40:11][flashy.solver][INFO] - Valid | Epoch 56 | 7/79 | 0.21 it/sec | ce 0.213 | ppl 1.239
[05-27 07:40:43][flashy.solver][INFO] - Valid | Epoch 56 | 14/79 | 0.22 it/sec | ce 0.197 | ppl 1.220
[05-27 07:41:16][flashy.solver][INFO] - Valid | Epoch 56 | 21/79 | 0.21 it/sec | ce 0.236 | ppl 1.267
[05-27 07:41:50][flashy.solver][INFO] - Valid | Epoch 56 | 28/79 | 0.21 it/sec | ce 0.202 | ppl 1.226
[05-27 07:42:24][flashy.solver][INFO] - Valid | Epoch 56 | 35/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-27 07:42:59][flashy.solver][INFO] - Valid | Epoch 56 | 42/79 | 0.21 it/sec | ce 0.200 | ppl 1.223
[05-27 07:43:35][flashy.solver][INFO] - Valid | Epoch 56 | 49/79 | 0.21 it/sec | ce 0.270 | ppl 1.313
[05-27 07:44:09][flashy.solver][INFO] - Valid | Epoch 56 | 56/79 | 0.21 it/sec | ce 0.203 | ppl 1.226
[05-27 07:44:44][flashy.solver][INFO] - Valid | Epoch 56 | 63/79 | 0.21 it/sec | ce 0.256 | ppl 1.292
[05-27 07:45:18][flashy.solver][INFO] - Valid | Epoch 56 | 70/79 | 0.21 it/sec | ce 0.237 | ppl 1.269
[05-27 07:45:53][flashy.solver][INFO] - Valid | Epoch 56 | 77/79 | 0.21 it/sec | ce 0.271 | ppl 1.312
[05-27 07:45:54][flashy.solver][INFO] - Valid Summary | Epoch 56 | ce=0.228 | ppl=1.258 | duration=380.956
[05-27 07:45:54][flashy.solver][INFO] - New best state with ce=0.228 (was 0.228)
[05-27 07:45:54][flashy.solver][INFO] - Model hash: a6f85f29cbc103603f8f3bafe92011b30b8e48b6
[05-27 07:45:54][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 07:50:14][flashy.solver][INFO] - Train | Epoch 56 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.363E-01 | ce 0.239 | ppl 1.271
[05-27 08:02:47][flashy.solver][INFO] - Train | Epoch 57 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.529E-01 | ce 0.242 | ppl 1.276
[05-27 08:06:52][flashy.solver][INFO] - Train | Epoch 56 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.467E-01 | ce 0.236 | ppl 1.269
[05-27 08:19:38][flashy.solver][INFO] - Train | Epoch 57 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.917E-01 | ce 0.238 | ppl 1.270
[05-27 08:23:51][flashy.solver][INFO] - Train | Epoch 56 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.401E-01 | ce 0.238 | ppl 1.271
[05-27 08:36:21][flashy.solver][INFO] - Train | Epoch 57 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.201E-01 | ce 0.234 | ppl 1.266
[05-27 08:40:53][flashy.solver][INFO] - Train | Epoch 56 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.425E-01 | ce 0.230 | ppl 1.260
[05-27 08:53:19][flashy.solver][INFO] - Train | Epoch 57 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.680E-01 | ce 0.249 | ppl 1.285
[05-27 08:57:39][flashy.solver][INFO] - Train | Epoch 56 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.847E-01 | ce 0.235 | ppl 1.267
[05-27 09:10:13][flashy.solver][INFO] - Train | Epoch 57 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.810E-01 | ce 0.236 | ppl 1.268
[05-27 09:14:14][flashy.solver][INFO] - Train Summary | Epoch 56 | lr=1.00E-04 | grad_norm=7.556E-01 | ce=0.237 | ppl=1.270 | duration=10172.497
[05-27 09:14:51][flashy.solver][INFO] - Valid | Epoch 56 | 7/79 | 0.21 it/sec | ce 0.212 | ppl 1.238
[05-27 09:15:23][flashy.solver][INFO] - Valid | Epoch 56 | 14/79 | 0.22 it/sec | ce 0.197 | ppl 1.220
[05-27 09:15:57][flashy.solver][INFO] - Valid | Epoch 56 | 21/79 | 0.21 it/sec | ce 0.235 | ppl 1.267
[05-27 09:16:29][flashy.solver][INFO] - Valid | Epoch 56 | 28/79 | 0.21 it/sec | ce 0.201 | ppl 1.226
[05-27 09:17:02][flashy.solver][INFO] - Valid | Epoch 56 | 35/79 | 0.21 it/sec | ce 0.231 | ppl 1.262
[05-27 09:17:35][flashy.solver][INFO] - Valid | Epoch 56 | 42/79 | 0.21 it/sec | ce 0.200 | ppl 1.223
[05-27 09:18:09][flashy.solver][INFO] - Valid | Epoch 56 | 49/79 | 0.21 it/sec | ce 0.270 | ppl 1.313
[05-27 09:18:42][flashy.solver][INFO] - Valid | Epoch 56 | 56/79 | 0.21 it/sec | ce 0.202 | ppl 1.225
[05-27 09:19:14][flashy.solver][INFO] - Valid | Epoch 56 | 63/79 | 0.21 it/sec | ce 0.255 | ppl 1.292
[05-27 09:19:47][flashy.solver][INFO] - Valid | Epoch 56 | 70/79 | 0.21 it/sec | ce 0.237 | ppl 1.268
[05-27 09:20:22][flashy.solver][INFO] - Valid | Epoch 56 | 77/79 | 0.21 it/sec | ce 0.271 | ppl 1.312
[05-27 09:20:22][flashy.solver][INFO] - Valid Summary | Epoch 56 | ce=0.227 | ppl=1.257 | duration=368.761
[05-27 09:20:22][flashy.solver][INFO] - New best state with ce=0.227 (was 0.228)
[05-27 09:20:22][flashy.solver][INFO] - Model hash: 71961736ee79202b167ac0fe53b43f354427d989
[05-27 09:20:22][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 09:26:28][flashy.solver][INFO] - Train | Epoch 57 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 6.970E-01 | ce 0.239 | ppl 1.272
[05-27 09:36:46][flashy.solver][INFO] - Train | Epoch 57 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.562E-01 | ce 0.234 | ppl 1.266
[05-27 09:42:49][flashy.solver][INFO] - Train | Epoch 57 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.863E-01 | ce 0.239 | ppl 1.272
[05-27 09:52:56][flashy.solver][INFO] - Train | Epoch 57 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.589E-01 | ce 0.230 | ppl 1.261
[05-27 09:59:03][flashy.solver][INFO] - Train | Epoch 57 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.257E-01 | ce 0.225 | ppl 1.255
[05-27 10:09:00][flashy.solver][INFO] - Train | Epoch 57 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.598E-01 | ce 0.242 | ppl 1.276
[05-27 10:15:19][flashy.solver][INFO] - Train | Epoch 57 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.237E-01 | ce 0.239 | ppl 1.272
[05-27 10:25:07][flashy.solver][INFO] - Train | Epoch 57 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.708E-01 | ce 0.242 | ppl 1.275
[05-27 10:31:36][flashy.solver][INFO] - Train Summary | Epoch 57 | lr=1.00E-04 | grad_norm=7.629E-01 | ce=0.238 | ppl=1.271 | duration=9941.644
[05-27 10:32:14][flashy.solver][INFO] - Valid | Epoch 57 | 7/79 | 0.21 it/sec | ce 0.212 | ppl 1.238
[05-27 10:32:46][flashy.solver][INFO] - Valid | Epoch 57 | 14/79 | 0.21 it/sec | ce 0.197 | ppl 1.220
[05-27 10:33:19][flashy.solver][INFO] - Valid | Epoch 57 | 21/79 | 0.21 it/sec | ce 0.235 | ppl 1.267
[05-27 10:33:53][flashy.solver][INFO] - Valid | Epoch 57 | 28/79 | 0.21 it/sec | ce 0.201 | ppl 1.226
[05-27 10:34:26][flashy.solver][INFO] - Valid | Epoch 57 | 35/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-27 10:34:58][flashy.solver][INFO] - Valid | Epoch 57 | 42/79 | 0.21 it/sec | ce 0.200 | ppl 1.223
[05-27 10:35:31][flashy.solver][INFO] - Valid | Epoch 57 | 49/79 | 0.21 it/sec | ce 0.269 | ppl 1.312
[05-27 10:36:04][flashy.solver][INFO] - Valid | Epoch 57 | 56/79 | 0.21 it/sec | ce 0.202 | ppl 1.225
[05-27 10:36:37][flashy.solver][INFO] - Valid | Epoch 57 | 63/79 | 0.21 it/sec | ce 0.255 | ppl 1.292
[05-27 10:37:11][flashy.solver][INFO] - Valid | Epoch 57 | 70/79 | 0.21 it/sec | ce 0.236 | ppl 1.268
[05-27 10:37:45][flashy.solver][INFO] - Valid | Epoch 57 | 77/79 | 0.21 it/sec | ce 0.270 | ppl 1.312
[05-27 10:37:46][flashy.solver][INFO] - Valid Summary | Epoch 57 | ce=0.227 | ppl=1.257 | duration=369.864
[05-27 10:37:46][flashy.solver][INFO] - New best state with ce=0.227 (was 0.228)
[05-27 10:37:46][flashy.solver][INFO] - Model hash: 1966c72b9fae6a2dd3a4490c5a715e468de3fba5
[05-27 10:37:46][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 10:41:25][flashy.solver][INFO] - Train | Epoch 57 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.708E-01 | ce 0.241 | ppl 1.275
[05-27 10:53:57][flashy.solver][INFO] - Train | Epoch 58 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.049E-01 | ce 0.231 | ppl 1.261
[05-27 10:57:25][flashy.solver][INFO] - Train | Epoch 57 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.243E-01 | ce 0.242 | ppl 1.276
[05-27 11:10:03][flashy.solver][INFO] - Train | Epoch 58 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.620E-01 | ce 0.241 | ppl 1.275
[05-27 11:13:41][flashy.solver][INFO] - Train | Epoch 57 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.953E-01 | ce 0.236 | ppl 1.269
[05-27 11:26:17][flashy.solver][INFO] - Train | Epoch 58 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.560E-01 | ce 0.237 | ppl 1.270
[05-27 11:29:46][flashy.solver][INFO] - Train | Epoch 57 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.166E-01 | ce 0.231 | ppl 1.262
[05-27 11:42:27][flashy.solver][INFO] - Train | Epoch 58 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.516E-01 | ce 0.243 | ppl 1.278
[05-27 11:45:57][flashy.solver][INFO] - Train | Epoch 57 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.915E-01 | ce 0.237 | ppl 1.270
[05-27 11:58:50][flashy.solver][INFO] - Train | Epoch 58 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.387E-01 | ce 0.233 | ppl 1.264
[05-27 12:02:05][flashy.solver][INFO] - Train Summary | Epoch 57 | lr=1.00E-04 | grad_norm=7.810E-01 | ce=0.238 | ppl=1.271 | duration=9702.417
[05-27 12:02:42][flashy.solver][INFO] - Valid | Epoch 57 | 7/79 | 0.22 it/sec | ce 0.212 | ppl 1.238
[05-27 12:03:15][flashy.solver][INFO] - Valid | Epoch 57 | 14/79 | 0.21 it/sec | ce 0.196 | ppl 1.219
[05-27 12:03:47][flashy.solver][INFO] - Valid | Epoch 57 | 21/79 | 0.22 it/sec | ce 0.235 | ppl 1.267
[05-27 12:04:19][flashy.solver][INFO] - Valid | Epoch 57 | 28/79 | 0.22 it/sec | ce 0.201 | ppl 1.226
[05-27 12:04:52][flashy.solver][INFO] - Valid | Epoch 57 | 35/79 | 0.21 it/sec | ce 0.231 | ppl 1.262
[05-27 12:05:25][flashy.solver][INFO] - Valid | Epoch 57 | 42/79 | 0.21 it/sec | ce 0.200 | ppl 1.223
[05-27 12:05:57][flashy.solver][INFO] - Valid | Epoch 57 | 49/79 | 0.21 it/sec | ce 0.270 | ppl 1.313
[05-27 12:06:30][flashy.solver][INFO] - Valid | Epoch 57 | 56/79 | 0.22 it/sec | ce 0.202 | ppl 1.225
[05-27 12:07:04][flashy.solver][INFO] - Valid | Epoch 57 | 63/79 | 0.21 it/sec | ce 0.255 | ppl 1.292
[05-27 12:07:36][flashy.solver][INFO] - Valid | Epoch 57 | 70/79 | 0.21 it/sec | ce 0.237 | ppl 1.268
[05-27 12:08:09][flashy.solver][INFO] - Valid | Epoch 57 | 77/79 | 0.21 it/sec | ce 0.271 | ppl 1.312
[05-27 12:08:10][flashy.solver][INFO] - Valid Summary | Epoch 57 | ce=0.227 | ppl=1.257 | duration=364.849
[05-27 12:08:10][flashy.solver][INFO] - Model hash: 2ce87b5b53be6f457e7f219ebe84c39f2dc3f4a6
[05-27 12:08:10][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 12:15:00][flashy.solver][INFO] - Train | Epoch 58 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.832E-01 | ce 0.236 | ppl 1.268
[05-27 12:24:32][flashy.solver][INFO] - Train | Epoch 58 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.466E-01 | ce 0.234 | ppl 1.266
[05-27 12:31:35][flashy.solver][INFO] - Train | Epoch 58 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.977E-01 | ce 0.235 | ppl 1.268
[05-27 12:41:01][flashy.solver][INFO] - Train | Epoch 58 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.778E-01 | ce 0.237 | ppl 1.270
[05-27 12:47:46][flashy.solver][INFO] - Train | Epoch 58 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.025E-01 | ce 0.235 | ppl 1.267
[05-27 12:56:59][flashy.solver][INFO] - Train | Epoch 58 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.547E-01 | ce 0.241 | ppl 1.276
[05-27 13:03:50][flashy.solver][INFO] - Train | Epoch 58 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.487E-01 | ce 0.244 | ppl 1.279
[05-27 13:13:22][flashy.solver][INFO] - Train | Epoch 58 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.036E-01 | ce 0.234 | ppl 1.267
[05-27 13:20:20][flashy.solver][INFO] - Train Summary | Epoch 58 | lr=1.00E-04 | grad_norm=7.822E-01 | ce=0.237 | ppl=1.270 | duration=9753.884
[05-27 13:20:57][flashy.solver][INFO] - Valid | Epoch 58 | 7/79 | 0.22 it/sec | ce 0.212 | ppl 1.238
[05-27 13:21:29][flashy.solver][INFO] - Valid | Epoch 58 | 14/79 | 0.21 it/sec | ce 0.196 | ppl 1.219
[05-27 13:22:04][flashy.solver][INFO] - Valid | Epoch 58 | 21/79 | 0.21 it/sec | ce 0.235 | ppl 1.266
[05-27 13:22:36][flashy.solver][INFO] - Valid | Epoch 58 | 28/79 | 0.21 it/sec | ce 0.201 | ppl 1.225
[05-27 13:23:08][flashy.solver][INFO] - Valid | Epoch 58 | 35/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-27 13:23:42][flashy.solver][INFO] - Valid | Epoch 58 | 42/79 | 0.21 it/sec | ce 0.199 | ppl 1.222
[05-27 13:24:16][flashy.solver][INFO] - Valid | Epoch 58 | 49/79 | 0.21 it/sec | ce 0.269 | ppl 1.312
[05-27 13:24:48][flashy.solver][INFO] - Valid | Epoch 58 | 56/79 | 0.21 it/sec | ce 0.202 | ppl 1.225
[05-27 13:25:21][flashy.solver][INFO] - Valid | Epoch 58 | 63/79 | 0.21 it/sec | ce 0.255 | ppl 1.291
[05-27 13:25:53][flashy.solver][INFO] - Valid | Epoch 58 | 70/79 | 0.21 it/sec | ce 0.236 | ppl 1.267
[05-27 13:26:28][flashy.solver][INFO] - Valid | Epoch 58 | 77/79 | 0.21 it/sec | ce 0.270 | ppl 1.311
[05-27 13:26:28][flashy.solver][INFO] - Valid Summary | Epoch 58 | ce=0.227 | ppl=1.257 | duration=368.705
[05-27 13:26:28][flashy.solver][INFO] - New best state with ce=0.227 (was 0.227)
[05-27 13:26:28][flashy.solver][INFO] - Model hash: 2af75a907db64290fc767377a3d9d94224e6d242
[05-27 13:26:28][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 13:29:54][flashy.solver][INFO] - Train | Epoch 58 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.693E-01 | ce 0.235 | ppl 1.268
[05-27 13:43:04][flashy.solver][INFO] - Train | Epoch 59 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.067E-01 | ce 0.232 | ppl 1.264
[05-27 13:46:11][flashy.solver][INFO] - Train | Epoch 58 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.366E-01 | ce 0.237 | ppl 1.269
[05-27 13:59:14][flashy.solver][INFO] - Train | Epoch 59 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.158E-01 | ce 0.240 | ppl 1.274
[05-27 14:02:14][flashy.solver][INFO] - Train | Epoch 58 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.162E-01 | ce 0.242 | ppl 1.277
[05-27 14:15:23][flashy.solver][INFO] - Train | Epoch 59 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.020E-01 | ce 0.228 | ppl 1.258
[05-27 14:18:18][flashy.solver][INFO] - Train | Epoch 58 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.126E-01 | ce 0.236 | ppl 1.268
[05-27 14:31:29][flashy.solver][INFO] - Train | Epoch 59 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.611E-01 | ce 0.244 | ppl 1.278
[05-27 14:34:25][flashy.solver][INFO] - Train | Epoch 58 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.281E-01 | ce 0.240 | ppl 1.274
[05-27 14:48:09][flashy.solver][INFO] - Train | Epoch 59 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.280E-01 | ce 0.237 | ppl 1.269
[05-27 14:50:58][flashy.solver][INFO] - Train Summary | Epoch 58 | lr=1.00E-04 | grad_norm=7.805E-01 | ce=0.237 | ppl=1.269 | duration=9768.635
[05-27 14:51:36][flashy.solver][INFO] - Valid | Epoch 58 | 7/79 | 0.21 it/sec | ce 0.211 | ppl 1.237
[05-27 14:52:09][flashy.solver][INFO] - Valid | Epoch 58 | 14/79 | 0.21 it/sec | ce 0.196 | ppl 1.219
[05-27 14:52:42][flashy.solver][INFO] - Valid | Epoch 58 | 21/79 | 0.21 it/sec | ce 0.235 | ppl 1.266
[05-27 14:53:16][flashy.solver][INFO] - Valid | Epoch 58 | 28/79 | 0.21 it/sec | ce 0.201 | ppl 1.225
[05-27 14:53:50][flashy.solver][INFO] - Valid | Epoch 58 | 35/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-27 14:54:23][flashy.solver][INFO] - Valid | Epoch 58 | 42/79 | 0.21 it/sec | ce 0.199 | ppl 1.222
[05-27 14:54:56][flashy.solver][INFO] - Valid | Epoch 58 | 49/79 | 0.21 it/sec | ce 0.269 | ppl 1.312
[05-27 14:55:28][flashy.solver][INFO] - Valid | Epoch 58 | 56/79 | 0.21 it/sec | ce 0.202 | ppl 1.224
[05-27 14:56:01][flashy.solver][INFO] - Valid | Epoch 58 | 63/79 | 0.21 it/sec | ce 0.255 | ppl 1.291
[05-27 14:56:34][flashy.solver][INFO] - Valid | Epoch 58 | 70/79 | 0.21 it/sec | ce 0.236 | ppl 1.267
[05-27 14:57:06][flashy.solver][INFO] - Valid | Epoch 58 | 77/79 | 0.21 it/sec | ce 0.270 | ppl 1.311
[05-27 14:57:07][flashy.solver][INFO] - Valid Summary | Epoch 58 | ce=0.227 | ppl=1.256 | duration=368.489
[05-27 14:57:07][flashy.solver][INFO] - New best state with ce=0.227 (was 0.227)
[05-27 14:57:07][flashy.solver][INFO] - Model hash: 9393e77b094676269bb22e3a9376e51503f09b93
[05-27 14:57:07][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 15:04:16][flashy.solver][INFO] - Train | Epoch 59 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.630E-01 | ce 0.238 | ppl 1.271
[05-27 15:13:20][flashy.solver][INFO] - Train | Epoch 59 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.821E-01 | ce 0.236 | ppl 1.268
[05-27 15:20:42][flashy.solver][INFO] - Train | Epoch 59 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.453E-01 | ce 0.238 | ppl 1.271
[05-27 15:29:45][flashy.solver][INFO] - Train | Epoch 59 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.299E-01 | ce 0.236 | ppl 1.268
[05-27 15:32:30][dora.distrib][INFO] - world_size is 1, skipping init.
[05-27 15:32:30][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-27 15:32:30][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-27 15:32:30][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-27 15:32:30][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-27 15:32:30][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-27 15:32:30][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-27 15:32:30][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-27 15:32:31][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-27 15:32:31][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-27 15:32:32][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-27 15:32:32][flashy.solver][INFO] - No EMA on the model.
[05-27 15:32:32][flashy.solver][INFO] - Model size: 1.21 M params
[05-27 15:32:32][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-27 15:32:32][flashy.solver][INFO] - Restoring weights and history.
[05-27 15:32:32][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 15:32:32][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 15:32:32][flashy.solver][INFO] - Model hash: 9393e77b094676269bb22e3a9376e51503f09b93
[05-27 15:32:32][flashy.solver][INFO] - Replaying past metrics...
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 16 | lr=1.00E-04 | grad_norm=5.373E-01 | ce=0.274 | ppl=1.318 | duration=9552.291
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 16 | ce=0.265 | ppl=1.306 | duration=362.290
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 17 | lr=1.00E-04 | grad_norm=5.495E-01 | ce=0.274 | ppl=1.318 | duration=9593.252
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 17 | ce=0.263 | ppl=1.303 | duration=361.603
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 18 | lr=1.00E-04 | grad_norm=5.303E-01 | ce=0.270 | ppl=1.314 | duration=9601.733
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 18 | ce=0.261 | ppl=1.301 | duration=362.833
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 19 | lr=1.00E-04 | grad_norm=5.436E-01 | ce=0.271 | ppl=1.314 | duration=9603.793
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 19 | ce=0.259 | ppl=1.298 | duration=362.807
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 20 | lr=1.00E-04 | grad_norm=5.256E-01 | ce=0.267 | ppl=1.309 | duration=9513.591
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 20 | ce=0.256 | ppl=1.295 | duration=359.164
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 21 | lr=1.00E-04 | grad_norm=5.398E-01 | ce=0.267 | ppl=1.309 | duration=9521.433
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 21 | ce=0.255 | ppl=1.293 | duration=359.736
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 22 | lr=1.00E-04 | grad_norm=5.273E-01 | ce=0.264 | ppl=1.305 | duration=9522.932
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 22 | ce=0.254 | ppl=1.291 | duration=359.952
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 23 | lr=1.00E-04 | grad_norm=5.359E-01 | ce=0.263 | ppl=1.304 | duration=9520.848
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 23 | ce=0.252 | ppl=1.290 | duration=359.946
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 24 | lr=1.00E-04 | grad_norm=5.356E-01 | ce=0.260 | ppl=1.300 | duration=9517.164
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 24 | ce=0.250 | ppl=1.287 | duration=360.223
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 25 | lr=1.00E-04 | grad_norm=5.350E-01 | ce=0.258 | ppl=1.297 | duration=9526.316
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 25 | ce=0.249 | ppl=1.286 | duration=359.858
[05-27 15:32:32][flashy.solver][INFO] - Evaluate Summary | Epoch 25 | duration=0.000
[05-27 15:32:32][flashy.solver][INFO] - Generate Summary | Epoch 25 | rtf=0.143 | duration=7.398
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 26 | lr=1.00E-04 | grad_norm=5.344E-01 | ce=0.255 | ppl=1.294 | duration=9505.304
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 26 | ce=0.248 | ppl=1.284 | duration=359.841
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 27 | lr=1.00E-04 | grad_norm=5.476E-01 | ce=0.256 | ppl=1.294 | duration=9515.969
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 27 | ce=0.247 | ppl=1.282 | duration=359.513
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 28 | lr=1.00E-04 | grad_norm=5.444E-01 | ce=0.254 | ppl=1.292 | duration=9521.965
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 28 | ce=0.245 | ppl=1.281 | duration=360.116
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 29 | lr=1.00E-04 | grad_norm=5.705E-01 | ce=0.255 | ppl=1.293 | duration=9536.583
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 29 | ce=0.245 | ppl=1.280 | duration=361.077
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 30 | lr=1.00E-04 | grad_norm=5.520E-01 | ce=0.253 | ppl=1.291 | duration=9517.157
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 30 | ce=0.243 | ppl=1.278 | duration=360.175
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 31 | lr=1.00E-04 | grad_norm=5.536E-01 | ce=0.252 | ppl=1.290 | duration=9515.999
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 31 | ce=0.242 | ppl=1.277 | duration=359.757
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 32 | lr=1.00E-04 | grad_norm=5.734E-01 | ce=0.251 | ppl=1.288 | duration=9525.455
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 32 | ce=0.241 | ppl=1.275 | duration=359.850
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 33 | lr=1.00E-04 | grad_norm=5.841E-01 | ce=0.252 | ppl=1.289 | duration=9522.251
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 33 | ce=0.240 | ppl=1.274 | duration=360.855
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 34 | lr=1.00E-04 | grad_norm=5.932E-01 | ce=0.250 | ppl=1.287 | duration=9530.122
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 34 | ce=0.240 | ppl=1.273 | duration=360.152
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 35 | lr=1.00E-04 | grad_norm=5.957E-01 | ce=0.248 | ppl=1.284 | duration=9524.631
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 35 | ce=0.239 | ppl=1.272 | duration=359.587
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 36 | lr=1.00E-04 | grad_norm=6.259E-01 | ce=0.251 | ppl=1.288 | duration=9531.088
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 36 | ce=0.239 | ppl=1.272 | duration=360.096
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 37 | lr=1.00E-04 | grad_norm=6.270E-01 | ce=0.248 | ppl=1.284 | duration=9533.307
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 37 | ce=0.237 | ppl=1.270 | duration=360.546
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 38 | lr=1.00E-04 | grad_norm=6.272E-01 | ce=0.248 | ppl=1.284 | duration=9529.414
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 38 | ce=0.237 | ppl=1.270 | duration=360.767
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 39 | lr=1.00E-04 | grad_norm=6.386E-01 | ce=0.246 | ppl=1.281 | duration=9524.562
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 39 | ce=0.236 | ppl=1.268 | duration=360.385
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 40 | lr=1.00E-04 | grad_norm=6.360E-01 | ce=0.244 | ppl=1.279 | duration=9642.435
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 40 | ce=0.235 | ppl=1.267 | duration=385.528
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 41 | lr=1.00E-04 | grad_norm=6.410E-01 | ce=0.246 | ppl=1.281 | duration=9668.847
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 41 | ce=0.235 | ppl=1.267 | duration=363.285
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 42 | lr=1.00E-04 | grad_norm=6.642E-01 | ce=0.244 | ppl=1.278 | duration=9572.143
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 42 | ce=0.234 | ppl=1.266 | duration=362.071
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 43 | lr=1.00E-04 | grad_norm=6.578E-01 | ce=0.244 | ppl=1.278 | duration=9564.111
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 43 | ce=0.234 | ppl=1.265 | duration=362.399
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 44 | lr=1.00E-04 | grad_norm=6.778E-01 | ce=0.244 | ppl=1.279 | duration=9545.633
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 44 | ce=0.233 | ppl=1.264 | duration=360.226
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 45 | lr=1.00E-04 | grad_norm=6.707E-01 | ce=0.240 | ppl=1.273 | duration=9558.510
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 45 | ce=0.232 | ppl=1.263 | duration=361.292
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 46 | lr=1.00E-04 | grad_norm=7.081E-01 | ce=0.243 | ppl=1.277 | duration=9549.454
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 46 | ce=0.232 | ppl=1.263 | duration=361.476
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 47 | lr=1.00E-04 | grad_norm=6.983E-01 | ce=0.241 | ppl=1.275 | duration=9994.908
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 47 | ce=0.231 | ppl=1.262 | duration=380.415
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 48 | lr=1.00E-04 | grad_norm=7.072E-01 | ce=0.241 | ppl=1.274 | duration=9756.736
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 48 | ce=0.231 | ppl=1.262 | duration=368.529
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 49 | lr=1.00E-04 | grad_norm=7.122E-01 | ce=0.239 | ppl=1.273 | duration=9719.254
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 49 | ce=0.230 | ppl=1.261 | duration=360.200
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 50 | lr=1.00E-04 | grad_norm=7.181E-01 | ce=0.237 | ppl=1.270 | duration=9752.171
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 50 | ce=0.230 | ppl=1.261 | duration=359.721
[05-27 15:32:32][flashy.solver][INFO] - Evaluate Summary | Epoch 50 | duration=0.000
[05-27 15:32:32][flashy.solver][INFO] - Generate Summary | Epoch 50 | rtf=0.134 | duration=6.334
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 51 | lr=1.00E-04 | grad_norm=7.459E-01 | ce=0.243 | ppl=1.277 | duration=9700.451
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 51 | ce=0.230 | ppl=1.260 | duration=360.665
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 52 | lr=1.00E-04 | grad_norm=7.595E-01 | ce=0.241 | ppl=1.274 | duration=9529.466
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 52 | ce=0.229 | ppl=1.260 | duration=360.312
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 53 | lr=1.00E-04 | grad_norm=7.351E-01 | ce=0.239 | ppl=1.273 | duration=9539.188
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 53 | ce=0.229 | ppl=1.259 | duration=359.881
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 54 | lr=1.00E-04 | grad_norm=7.549E-01 | ce=0.239 | ppl=1.272 | duration=9542.972
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 54 | ce=0.228 | ppl=1.258 | duration=368.039
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 55 | lr=1.00E-04 | grad_norm=7.593E-01 | ce=0.237 | ppl=1.269 | duration=9580.192
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 55 | ce=0.228 | ppl=1.258 | duration=380.776
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 56 | lr=1.00E-04 | grad_norm=7.556E-01 | ce=0.237 | ppl=1.270 | duration=10172.497
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 56 | ce=0.227 | ppl=1.257 | duration=368.761
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 57 | lr=1.00E-04 | grad_norm=7.810E-01 | ce=0.238 | ppl=1.271 | duration=9702.417
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 57 | ce=0.227 | ppl=1.257 | duration=364.849
[05-27 15:32:32][flashy.solver][INFO] - Train Summary | Epoch 58 | lr=1.00E-04 | grad_norm=7.805E-01 | ce=0.237 | ppl=1.269 | duration=9768.635
[05-27 15:32:32][flashy.solver][INFO] - Valid Summary | Epoch 58 | ce=0.227 | ppl=1.256 | duration=368.489
[05-27 15:32:38][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-27 15:37:04][flashy.solver][INFO] - Train | Epoch 59 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.932E-01 | ce 0.240 | ppl 1.273
[05-27 15:48:40][flashy.solver][INFO] - Train | Epoch 59 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.051E-01 | ce 0.236 | ppl 1.268
[05-27 15:53:01][flashy.solver][INFO] - Train | Epoch 59 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.414E-01 | ce 0.231 | ppl 1.262
[05-27 16:04:47][flashy.solver][INFO] - Train | Epoch 59 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.968E-01 | ce 0.235 | ppl 1.268
[05-27 16:09:11][flashy.solver][INFO] - Train Summary | Epoch 59 | lr=1.00E-04 | grad_norm=7.903E-01 | ce=0.237 | ppl=1.269 | duration=9762.634
[05-27 16:09:49][flashy.solver][INFO] - Valid | Epoch 59 | 7/79 | 0.21 it/sec | ce 0.212 | ppl 1.237
[05-27 16:10:23][flashy.solver][INFO] - Valid | Epoch 59 | 14/79 | 0.21 it/sec | ce 0.196 | ppl 1.218
[05-27 16:10:57][flashy.solver][INFO] - Valid | Epoch 59 | 21/79 | 0.21 it/sec | ce 0.234 | ppl 1.266
[05-27 16:11:30][flashy.solver][INFO] - Valid | Epoch 59 | 28/79 | 0.21 it/sec | ce 0.200 | ppl 1.225
[05-27 16:12:03][flashy.solver][INFO] - Valid | Epoch 59 | 35/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-27 16:12:36][flashy.solver][INFO] - Valid | Epoch 59 | 42/79 | 0.21 it/sec | ce 0.199 | ppl 1.221
[05-27 16:13:11][flashy.solver][INFO] - Valid | Epoch 59 | 49/79 | 0.21 it/sec | ce 0.269 | ppl 1.311
[05-27 16:13:43][flashy.solver][INFO] - Valid | Epoch 59 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.224
[05-27 16:14:16][flashy.solver][INFO] - Valid | Epoch 59 | 63/79 | 0.21 it/sec | ce 0.254 | ppl 1.290
[05-27 16:14:48][flashy.solver][INFO] - Valid | Epoch 59 | 70/79 | 0.21 it/sec | ce 0.236 | ppl 1.267
[05-27 16:15:21][flashy.solver][INFO] - Valid | Epoch 59 | 77/79 | 0.21 it/sec | ce 0.269 | ppl 1.310
[05-27 16:15:22][flashy.solver][INFO] - Valid Summary | Epoch 59 | ce=0.226 | ppl=1.256 | duration=370.989
[05-27 16:15:22][flashy.solver][INFO] - New best state with ce=0.226 (was 0.227)
[05-27 16:15:22][flashy.solver][INFO] - Model hash: b290d6ee25ee95040ae9ed90300ac79654ef7432
[05-27 16:15:22][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 16:21:18][flashy.solver][INFO] - Train | Epoch 59 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.027E-01 | ce 0.236 | ppl 1.269
[05-27 16:32:10][flashy.solver][INFO] - Train | Epoch 60 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.035E-01 | ce 0.244 | ppl 1.279
[05-27 16:37:36][flashy.solver][INFO] - Train | Epoch 59 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.915E-01 | ce 0.224 | ppl 1.253
[05-27 16:48:43][flashy.solver][INFO] - Train | Epoch 60 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.446E-01 | ce 0.238 | ppl 1.271
[05-27 16:54:09][flashy.solver][INFO] - Train | Epoch 59 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.491E-01 | ce 0.235 | ppl 1.267
[05-27 17:05:11][flashy.solver][INFO] - Train | Epoch 60 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.699E-01 | ce 0.232 | ppl 1.263
[05-27 17:10:32][flashy.solver][INFO] - Train | Epoch 59 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.815E-01 | ce 0.240 | ppl 1.274
[05-27 17:21:30][flashy.solver][INFO] - Train | Epoch 60 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.878E-01 | ce 0.243 | ppl 1.278
[05-27 17:26:46][flashy.solver][INFO] - Train | Epoch 59 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.761E-01 | ce 0.239 | ppl 1.271
[05-27 17:37:54][flashy.solver][INFO] - Train | Epoch 60 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.730E-01 | ce 0.238 | ppl 1.271
[05-27 17:43:06][flashy.solver][INFO] - Train | Epoch 59 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.248E-01 | ce 0.240 | ppl 1.273
[05-27 17:54:11][flashy.solver][INFO] - Train | Epoch 60 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.208E-01 | ce 0.229 | ppl 1.259
[05-27 17:59:17][flashy.solver][INFO] - Train | Epoch 59 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.644E-01 | ce 0.230 | ppl 1.262
[05-27 18:10:37][flashy.solver][INFO] - Train | Epoch 60 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.449E-01 | ce 0.234 | ppl 1.266
[05-27 18:15:42][flashy.solver][INFO] - Train Summary | Epoch 59 | lr=1.00E-04 | grad_norm=7.934E-01 | ce=0.234 | ppl=1.266 | duration=9789.248
[05-27 18:16:19][flashy.solver][INFO] - Valid | Epoch 59 | 7/79 | 0.21 it/sec | ce 0.211 | ppl 1.237
[05-27 18:16:51][flashy.solver][INFO] - Valid | Epoch 59 | 14/79 | 0.22 it/sec | ce 0.196 | ppl 1.219
[05-27 18:17:25][flashy.solver][INFO] - Valid | Epoch 59 | 21/79 | 0.21 it/sec | ce 0.234 | ppl 1.266
[05-27 18:17:58][flashy.solver][INFO] - Valid | Epoch 59 | 28/79 | 0.21 it/sec | ce 0.200 | ppl 1.225
[05-27 18:18:30][flashy.solver][INFO] - Valid | Epoch 59 | 35/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-27 18:19:04][flashy.solver][INFO] - Valid | Epoch 59 | 42/79 | 0.21 it/sec | ce 0.199 | ppl 1.221
[05-27 18:19:37][flashy.solver][INFO] - Valid | Epoch 59 | 49/79 | 0.21 it/sec | ce 0.268 | ppl 1.311
[05-27 18:20:10][flashy.solver][INFO] - Valid | Epoch 59 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.224
[05-27 18:20:43][flashy.solver][INFO] - Valid | Epoch 59 | 63/79 | 0.21 it/sec | ce 0.255 | ppl 1.291
[05-27 18:21:17][flashy.solver][INFO] - Valid | Epoch 59 | 70/79 | 0.21 it/sec | ce 0.236 | ppl 1.267
[05-27 18:21:50][flashy.solver][INFO] - Valid | Epoch 59 | 77/79 | 0.21 it/sec | ce 0.269 | ppl 1.310
[05-27 18:21:50][flashy.solver][INFO] - Valid Summary | Epoch 59 | ce=0.226 | ppl=1.256 | duration=368.743
[05-27 18:21:50][flashy.solver][INFO] - New best state with ce=0.226 (was 0.227)
[05-27 18:21:50][flashy.solver][INFO] - Model hash: 989467131ee0d489ea525d3aec97556ff907317d
[05-27 18:21:50][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 18:27:11][flashy.solver][INFO] - Train | Epoch 60 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.226E-01 | ce 0.235 | ppl 1.267
[05-27 18:38:21][flashy.solver][INFO] - Train | Epoch 60 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.627E-01 | ce 0.232 | ppl 1.264
[05-27 18:43:30][flashy.solver][INFO] - Train | Epoch 60 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.759E-01 | ce 0.238 | ppl 1.271
[05-27 18:54:39][flashy.solver][INFO] - Train | Epoch 60 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.398E-01 | ce 0.230 | ppl 1.261
[05-27 18:59:56][flashy.solver][INFO] - Train Summary | Epoch 60 | lr=1.00E-04 | grad_norm=8.007E-01 | ce=0.237 | ppl=1.269 | duration=9874.322
[05-27 19:00:35][flashy.solver][INFO] - Valid | Epoch 60 | 7/79 | 0.21 it/sec | ce 0.212 | ppl 1.237
[05-27 19:01:08][flashy.solver][INFO] - Valid | Epoch 60 | 14/79 | 0.21 it/sec | ce 0.196 | ppl 1.219
[05-27 19:01:41][flashy.solver][INFO] - Valid | Epoch 60 | 21/79 | 0.21 it/sec | ce 0.234 | ppl 1.266
[05-27 19:02:15][flashy.solver][INFO] - Valid | Epoch 60 | 28/79 | 0.21 it/sec | ce 0.200 | ppl 1.225
[05-27 19:02:48][flashy.solver][INFO] - Valid | Epoch 60 | 35/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-27 19:03:22][flashy.solver][INFO] - Valid | Epoch 60 | 42/79 | 0.21 it/sec | ce 0.199 | ppl 1.222
[05-27 19:03:56][flashy.solver][INFO] - Valid | Epoch 60 | 49/79 | 0.21 it/sec | ce 0.268 | ppl 1.311
[05-27 19:04:29][flashy.solver][INFO] - Valid | Epoch 60 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.224
[05-27 19:05:03][flashy.solver][INFO] - Valid | Epoch 60 | 63/79 | 0.21 it/sec | ce 0.255 | ppl 1.291
[05-27 19:05:37][flashy.solver][INFO] - Valid | Epoch 60 | 70/79 | 0.21 it/sec | ce 0.236 | ppl 1.267
[05-27 19:06:11][flashy.solver][INFO] - Valid | Epoch 60 | 77/79 | 0.21 it/sec | ce 0.269 | ppl 1.310
[05-27 19:06:12][flashy.solver][INFO] - Valid Summary | Epoch 60 | ce=0.227 | ppl=1.256 | duration=375.685
[05-27 19:06:12][flashy.solver][INFO] - Model hash: 9c5a4841943a0b59f6ccf1c27f6be331302da9ca
[05-27 19:06:12][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 19:11:21][flashy.solver][INFO] - Train | Epoch 60 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.853E-01 | ce 0.238 | ppl 1.270
[05-27 19:23:16][flashy.solver][INFO] - Train | Epoch 61 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.900E-01 | ce 0.241 | ppl 1.275
[05-27 19:28:05][flashy.solver][INFO] - Train | Epoch 60 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.030E-01 | ce 0.238 | ppl 1.271
[05-27 19:39:41][flashy.solver][INFO] - Train | Epoch 61 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.131E-01 | ce 0.237 | ppl 1.270
[05-27 19:44:46][flashy.solver][INFO] - Train | Epoch 60 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.099E-01 | ce 0.237 | ppl 1.270
[05-27 19:56:03][flashy.solver][INFO] - Train | Epoch 61 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.725E-01 | ce 0.234 | ppl 1.266
[05-27 20:00:57][flashy.solver][INFO] - Train | Epoch 60 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.403E-01 | ce 0.238 | ppl 1.271
[05-27 20:12:21][flashy.solver][INFO] - Train | Epoch 61 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.304E-01 | ce 0.243 | ppl 1.278
[05-27 20:17:34][flashy.solver][INFO] - Train | Epoch 60 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.860E-01 | ce 0.235 | ppl 1.266
[05-27 20:29:09][flashy.solver][INFO] - Train | Epoch 61 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.879E-01 | ce 0.239 | ppl 1.272
[05-27 20:33:58][flashy.solver][INFO] - Train | Epoch 60 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.207E-01 | ce 0.243 | ppl 1.277
[05-27 20:45:27][flashy.solver][INFO] - Train | Epoch 61 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.976E-01 | ce 0.236 | ppl 1.269
[05-27 20:50:12][flashy.solver][INFO] - Train | Epoch 60 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.658E-01 | ce 0.233 | ppl 1.265
[05-27 21:01:42][flashy.solver][INFO] - Train | Epoch 61 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.671E-01 | ce 0.240 | ppl 1.273
[05-27 21:06:30][flashy.solver][INFO] - Train Summary | Epoch 60 | lr=1.00E-04 | grad_norm=7.924E-01 | ce=0.235 | ppl=1.267 | duration=9879.320
[05-27 21:07:07][flashy.solver][INFO] - Valid | Epoch 60 | 7/79 | 0.21 it/sec | ce 0.212 | ppl 1.238
[05-27 21:07:40][flashy.solver][INFO] - Valid | Epoch 60 | 14/79 | 0.21 it/sec | ce 0.196 | ppl 1.219
[05-27 21:08:14][flashy.solver][INFO] - Valid | Epoch 60 | 21/79 | 0.21 it/sec | ce 0.235 | ppl 1.266
[05-27 21:08:47][flashy.solver][INFO] - Valid | Epoch 60 | 28/79 | 0.21 it/sec | ce 0.201 | ppl 1.225
[05-27 21:09:19][flashy.solver][INFO] - Valid | Epoch 60 | 35/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-27 21:09:54][flashy.solver][INFO] - Valid | Epoch 60 | 42/79 | 0.21 it/sec | ce 0.199 | ppl 1.222
[05-27 21:10:27][flashy.solver][INFO] - Valid | Epoch 60 | 49/79 | 0.21 it/sec | ce 0.269 | ppl 1.311
[05-27 21:11:00][flashy.solver][INFO] - Valid | Epoch 60 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.224
[05-27 21:11:33][flashy.solver][INFO] - Valid | Epoch 60 | 63/79 | 0.21 it/sec | ce 0.255 | ppl 1.291
[05-27 21:12:06][flashy.solver][INFO] - Valid | Epoch 60 | 70/79 | 0.21 it/sec | ce 0.236 | ppl 1.267
[05-27 21:12:40][flashy.solver][INFO] - Valid | Epoch 60 | 77/79 | 0.21 it/sec | ce 0.269 | ppl 1.310
[05-27 21:12:40][flashy.solver][INFO] - Valid Summary | Epoch 60 | ce=0.227 | ppl=1.256 | duration=370.751
[05-27 21:12:40][flashy.solver][INFO] - Model hash: 93768a0c48382aa3e89fb578a6538b2eb7a6b12b
[05-27 21:12:41][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 21:18:03][flashy.solver][INFO] - Train | Epoch 61 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.680E-01 | ce 0.234 | ppl 1.266
[05-27 21:28:57][flashy.solver][INFO] - Train | Epoch 61 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.287E-01 | ce 0.242 | ppl 1.276
[05-27 21:34:28][flashy.solver][INFO] - Train | Epoch 61 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.519E-01 | ce 0.234 | ppl 1.265
[05-27 21:45:36][flashy.solver][INFO] - Train | Epoch 61 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.459E-01 | ce 0.238 | ppl 1.271
[05-27 21:50:57][flashy.solver][INFO] - Train Summary | Epoch 61 | lr=1.00E-04 | grad_norm=8.122E-01 | ce=0.238 | ppl=1.271 | duration=9884.318
[05-27 21:51:36][flashy.solver][INFO] - Valid | Epoch 61 | 7/79 | 0.20 it/sec | ce 0.211 | ppl 1.236
[05-27 21:52:09][flashy.solver][INFO] - Valid | Epoch 61 | 14/79 | 0.21 it/sec | ce 0.195 | ppl 1.218
[05-27 21:52:42][flashy.solver][INFO] - Valid | Epoch 61 | 21/79 | 0.21 it/sec | ce 0.234 | ppl 1.265
[05-27 21:53:14][flashy.solver][INFO] - Valid | Epoch 61 | 28/79 | 0.21 it/sec | ce 0.200 | ppl 1.224
[05-27 21:53:48][flashy.solver][INFO] - Valid | Epoch 61 | 35/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-27 21:54:21][flashy.solver][INFO] - Valid | Epoch 61 | 42/79 | 0.21 it/sec | ce 0.198 | ppl 1.221
[05-27 21:54:54][flashy.solver][INFO] - Valid | Epoch 61 | 49/79 | 0.21 it/sec | ce 0.267 | ppl 1.310
[05-27 21:55:27][flashy.solver][INFO] - Valid | Epoch 61 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.224
[05-27 21:56:00][flashy.solver][INFO] - Valid | Epoch 61 | 63/79 | 0.21 it/sec | ce 0.254 | ppl 1.290
[05-27 21:56:33][flashy.solver][INFO] - Valid | Epoch 61 | 70/79 | 0.21 it/sec | ce 0.235 | ppl 1.266
[05-27 21:57:08][flashy.solver][INFO] - Valid | Epoch 61 | 77/79 | 0.21 it/sec | ce 0.268 | ppl 1.309
[05-27 21:57:09][flashy.solver][INFO] - Valid Summary | Epoch 61 | ce=0.226 | ppl=1.255 | duration=372.385
[05-27 21:57:09][flashy.solver][INFO] - New best state with ce=0.226 (was 0.226)
[05-27 21:57:09][flashy.solver][INFO] - Model hash: 593fdc5798e5426a448e8aec3860d23e2648434c
[05-27 21:57:09][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-27 22:02:12][flashy.solver][INFO] - Train | Epoch 61 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.813E-01 | ce 0.242 | ppl 1.277
[05-27 22:14:14][flashy.solver][INFO] - Train | Epoch 62 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.559E-01 | ce 0.235 | ppl 1.267
[05-27 22:18:55][flashy.solver][INFO] - Train | Epoch 61 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.569E-01 | ce 0.235 | ppl 1.267
[05-27 22:30:25][flashy.solver][INFO] - Train | Epoch 62 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.060E-01 | ce 0.239 | ppl 1.273
[05-27 22:35:03][flashy.solver][INFO] - Train | Epoch 61 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.344E-01 | ce 0.234 | ppl 1.265
[05-27 22:46:49][flashy.solver][INFO] - Train | Epoch 62 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.919E-01 | ce 0.235 | ppl 1.267
[05-27 22:51:36][flashy.solver][INFO] - Train | Epoch 61 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.743E-01 | ce 0.232 | ppl 1.264
[05-27 23:03:10][flashy.solver][INFO] - Train | Epoch 62 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.269E-01 | ce 0.232 | ppl 1.264
[05-27 23:07:58][flashy.solver][INFO] - Train | Epoch 61 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.096E-01 | ce 0.235 | ppl 1.268
[05-27 23:19:23][flashy.solver][INFO] - Train | Epoch 62 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.450E-01 | ce 0.222 | ppl 1.250
[05-27 23:24:12][flashy.solver][INFO] - Train | Epoch 61 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.197E-01 | ce 0.239 | ppl 1.272
[05-27 23:35:46][flashy.solver][INFO] - Train | Epoch 62 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.986E-01 | ce 0.241 | ppl 1.274
[05-27 23:40:30][flashy.solver][INFO] - Train | Epoch 61 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.649E-01 | ce 0.235 | ppl 1.267
[05-27 23:52:00][flashy.solver][INFO] - Train | Epoch 62 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.240E-01 | ce 0.233 | ppl 1.264
[05-27 23:56:57][flashy.solver][INFO] - Train Summary | Epoch 61 | lr=1.00E-04 | grad_norm=8.156E-01 | ce=0.237 | ppl=1.269 | duration=9856.496
[05-27 23:57:34][flashy.solver][INFO] - Valid | Epoch 61 | 7/79 | 0.22 it/sec | ce 0.211 | ppl 1.236
[05-27 23:58:08][flashy.solver][INFO] - Valid | Epoch 61 | 14/79 | 0.21 it/sec | ce 0.195 | ppl 1.218
[05-27 23:58:41][flashy.solver][INFO] - Valid | Epoch 61 | 21/79 | 0.21 it/sec | ce 0.233 | ppl 1.264
[05-27 23:59:13][flashy.solver][INFO] - Valid | Epoch 61 | 28/79 | 0.21 it/sec | ce 0.200 | ppl 1.224
[05-27 23:59:46][flashy.solver][INFO] - Valid | Epoch 61 | 35/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-28 00:00:20][flashy.solver][INFO] - Valid | Epoch 61 | 42/79 | 0.21 it/sec | ce 0.198 | ppl 1.220
[05-28 00:00:53][flashy.solver][INFO] - Valid | Epoch 61 | 49/79 | 0.21 it/sec | ce 0.267 | ppl 1.309
[05-28 00:01:26][flashy.solver][INFO] - Valid | Epoch 61 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.223
[05-28 00:02:00][flashy.solver][INFO] - Valid | Epoch 61 | 63/79 | 0.21 it/sec | ce 0.254 | ppl 1.290
[05-28 00:02:33][flashy.solver][INFO] - Valid | Epoch 61 | 70/79 | 0.21 it/sec | ce 0.235 | ppl 1.266
[05-28 00:03:07][flashy.solver][INFO] - Valid | Epoch 61 | 77/79 | 0.21 it/sec | ce 0.268 | ppl 1.309
[05-28 00:03:07][flashy.solver][INFO] - Valid Summary | Epoch 61 | ce=0.226 | ppl=1.255 | duration=370.156
[05-28 00:03:07][flashy.solver][INFO] - New best state with ce=0.226 (was 0.226)
[05-28 00:03:07][flashy.solver][INFO] - Model hash: bbbbcaacd1ce46a4be9d6d3abb6232293090be4a
[05-28 00:03:07][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 00:08:35][flashy.solver][INFO] - Train | Epoch 62 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.557E-01 | ce 0.230 | ppl 1.260
[05-28 00:19:22][flashy.solver][INFO] - Train | Epoch 62 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.006E-01 | ce 0.234 | ppl 1.266
[05-28 00:24:46][flashy.solver][INFO] - Train | Epoch 62 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.922E-01 | ce 0.235 | ppl 1.267
[05-28 00:35:35][flashy.solver][INFO] - Train | Epoch 62 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.053E-01 | ce 0.229 | ppl 1.259
[05-28 00:41:10][flashy.solver][INFO] - Train Summary | Epoch 62 | lr=1.00E-04 | grad_norm=7.932E-01 | ce=0.234 | ppl=1.266 | duration=9840.462
[05-28 00:41:48][flashy.solver][INFO] - Valid | Epoch 62 | 7/79 | 0.21 it/sec | ce 0.210 | ppl 1.236
[05-28 00:42:21][flashy.solver][INFO] - Valid | Epoch 62 | 14/79 | 0.21 it/sec | ce 0.195 | ppl 1.217
[05-28 00:42:54][flashy.solver][INFO] - Valid | Epoch 62 | 21/79 | 0.21 it/sec | ce 0.234 | ppl 1.264
[05-28 00:43:27][flashy.solver][INFO] - Valid | Epoch 62 | 28/79 | 0.21 it/sec | ce 0.200 | ppl 1.224
[05-28 00:44:02][flashy.solver][INFO] - Valid | Epoch 62 | 35/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-28 00:44:35][flashy.solver][INFO] - Valid | Epoch 62 | 42/79 | 0.21 it/sec | ce 0.198 | ppl 1.221
[05-28 00:45:08][flashy.solver][INFO] - Valid | Epoch 62 | 49/79 | 0.21 it/sec | ce 0.267 | ppl 1.310
[05-28 00:45:41][flashy.solver][INFO] - Valid | Epoch 62 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.224
[05-28 00:46:14][flashy.solver][INFO] - Valid | Epoch 62 | 63/79 | 0.21 it/sec | ce 0.254 | ppl 1.290
[05-28 00:46:47][flashy.solver][INFO] - Valid | Epoch 62 | 70/79 | 0.21 it/sec | ce 0.235 | ppl 1.266
[05-28 00:47:21][flashy.solver][INFO] - Valid | Epoch 62 | 77/79 | 0.21 it/sec | ce 0.268 | ppl 1.309
[05-28 00:47:21][flashy.solver][INFO] - Valid Summary | Epoch 62 | ce=0.225 | ppl=1.255 | duration=371.850
[05-28 00:47:21][flashy.solver][INFO] - New best state with ce=0.225 (was 0.226)
[05-28 00:47:21][flashy.solver][INFO] - Model hash: 87a7036f043275fa656b489d8dd1aea85e89fd0a
[05-28 00:47:21][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 00:51:59][flashy.solver][INFO] - Train | Epoch 62 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.109E-01 | ce 0.237 | ppl 1.270
[05-28 01:03:52][flashy.solver][INFO] - Train | Epoch 63 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.302E-01 | ce 0.233 | ppl 1.264
[05-28 01:08:11][flashy.solver][INFO] - Train | Epoch 62 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.213E-01 | ce 0.234 | ppl 1.266
[05-28 01:20:17][flashy.solver][INFO] - Train | Epoch 63 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.984E-01 | ce 0.225 | ppl 1.255
[05-28 01:24:35][flashy.solver][INFO] - Train | Epoch 62 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.337E-01 | ce 0.233 | ppl 1.264
[05-28 01:36:40][flashy.solver][INFO] - Train | Epoch 63 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.071E-01 | ce 0.230 | ppl 1.261
[05-28 01:41:01][flashy.solver][INFO] - Train | Epoch 62 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.864E-01 | ce 0.229 | ppl 1.259
[05-28 01:52:56][flashy.solver][INFO] - Train | Epoch 63 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.353E-01 | ce 0.239 | ppl 1.272
[05-28 01:57:27][flashy.solver][INFO] - Train | Epoch 62 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.911E-01 | ce 0.230 | ppl 1.262
[05-28 02:09:43][flashy.solver][INFO] - Train | Epoch 63 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.650E-01 | ce 0.235 | ppl 1.266
[05-28 02:14:07][flashy.solver][INFO] - Train | Epoch 62 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.268E-01 | ce 0.237 | ppl 1.269
[05-28 02:26:07][flashy.solver][INFO] - Train | Epoch 63 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.730E-01 | ce 0.236 | ppl 1.269
[05-28 02:30:41][flashy.solver][INFO] - Train | Epoch 62 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.306E-01 | ce 0.236 | ppl 1.269
[05-28 02:42:34][flashy.solver][INFO] - Train | Epoch 63 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.142E-01 | ce 0.230 | ppl 1.261
[05-28 02:46:55][flashy.solver][INFO] - Train Summary | Epoch 62 | lr=1.00E-04 | grad_norm=8.115E-01 | ce=0.233 | ppl=1.265 | duration=9827.438
[05-28 02:47:32][flashy.solver][INFO] - Valid | Epoch 62 | 7/79 | 0.21 it/sec | ce 0.210 | ppl 1.236
[05-28 02:48:04][flashy.solver][INFO] - Valid | Epoch 62 | 14/79 | 0.22 it/sec | ce 0.195 | ppl 1.218
[05-28 02:48:37][flashy.solver][INFO] - Valid | Epoch 62 | 21/79 | 0.22 it/sec | ce 0.234 | ppl 1.264
[05-28 02:49:10][flashy.solver][INFO] - Valid | Epoch 62 | 28/79 | 0.21 it/sec | ce 0.200 | ppl 1.224
[05-28 02:49:43][flashy.solver][INFO] - Valid | Epoch 62 | 35/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-28 02:50:15][flashy.solver][INFO] - Valid | Epoch 62 | 42/79 | 0.21 it/sec | ce 0.198 | ppl 1.220
[05-28 02:50:48][flashy.solver][INFO] - Valid | Epoch 62 | 49/79 | 0.21 it/sec | ce 0.267 | ppl 1.309
[05-28 02:51:21][flashy.solver][INFO] - Valid | Epoch 62 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.223
[05-28 02:51:56][flashy.solver][INFO] - Valid | Epoch 62 | 63/79 | 0.21 it/sec | ce 0.253 | ppl 1.289
[05-28 02:52:28][flashy.solver][INFO] - Valid | Epoch 62 | 70/79 | 0.21 it/sec | ce 0.235 | ppl 1.266
[05-28 02:53:01][flashy.solver][INFO] - Valid | Epoch 62 | 77/79 | 0.21 it/sec | ce 0.268 | ppl 1.308
[05-28 02:53:01][flashy.solver][INFO] - Valid Summary | Epoch 62 | ce=0.225 | ppl=1.255 | duration=366.751
[05-28 02:53:01][flashy.solver][INFO] - New best state with ce=0.225 (was 0.226)
[05-28 02:53:01][flashy.solver][INFO] - Model hash: 8beb003e5864b88987ede3943aaddb4712a76b5a
[05-28 02:53:02][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 02:58:44][flashy.solver][INFO] - Train | Epoch 63 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.646E-01 | ce 0.229 | ppl 1.259
[05-28 03:09:16][flashy.solver][INFO] - Train | Epoch 63 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.050E-01 | ce 0.231 | ppl 1.262
[05-28 03:14:50][flashy.solver][INFO] - Train | Epoch 63 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.431E-01 | ce 0.237 | ppl 1.269
[05-28 03:25:16][flashy.solver][INFO] - Train | Epoch 63 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.230E-01 | ce 0.230 | ppl 1.260
[05-28 03:30:47][flashy.solver][INFO] - Train Summary | Epoch 63 | lr=1.00E-04 | grad_norm=8.240E-01 | ce=0.233 | ppl=1.265 | duration=9806.020
[05-28 03:31:26][flashy.solver][INFO] - Valid | Epoch 63 | 7/79 | 0.21 it/sec | ce 0.211 | ppl 1.236
[05-28 03:31:58][flashy.solver][INFO] - Valid | Epoch 63 | 14/79 | 0.21 it/sec | ce 0.195 | ppl 1.218
[05-28 03:32:31][flashy.solver][INFO] - Valid | Epoch 63 | 21/79 | 0.21 it/sec | ce 0.234 | ppl 1.265
[05-28 03:33:03][flashy.solver][INFO] - Valid | Epoch 63 | 28/79 | 0.21 it/sec | ce 0.200 | ppl 1.224
[05-28 03:33:37][flashy.solver][INFO] - Valid | Epoch 63 | 35/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-28 03:34:09][flashy.solver][INFO] - Valid | Epoch 63 | 42/79 | 0.21 it/sec | ce 0.198 | ppl 1.221
[05-28 03:34:42][flashy.solver][INFO] - Valid | Epoch 63 | 49/79 | 0.21 it/sec | ce 0.268 | ppl 1.310
[05-28 03:35:14][flashy.solver][INFO] - Valid | Epoch 63 | 56/79 | 0.21 it/sec | ce 0.201 | ppl 1.223
[05-28 03:35:47][flashy.solver][INFO] - Valid | Epoch 63 | 63/79 | 0.21 it/sec | ce 0.254 | ppl 1.290
[05-28 03:36:22][flashy.solver][INFO] - Valid | Epoch 63 | 70/79 | 0.21 it/sec | ce 0.235 | ppl 1.266
[05-28 03:36:56][flashy.solver][INFO] - Valid | Epoch 63 | 77/79 | 0.21 it/sec | ce 0.268 | ppl 1.309
[05-28 03:36:57][flashy.solver][INFO] - Valid Summary | Epoch 63 | ce=0.226 | ppl=1.255 | duration=369.038
[05-28 03:36:57][flashy.solver][INFO] - Model hash: 6ab333bc5690607556093a12b355a849b9d6c905
[05-28 03:36:57][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 03:41:45][flashy.solver][INFO] - Train | Epoch 63 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.081E-01 | ce 0.231 | ppl 1.263
[05-28 03:53:34][flashy.solver][INFO] - Train | Epoch 64 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.505E-01 | ce 0.238 | ppl 1.272
[05-28 03:57:57][flashy.solver][INFO] - Train | Epoch 63 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.707E-01 | ce 0.227 | ppl 1.257
[05-28 04:10:17][flashy.solver][INFO] - Train | Epoch 64 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.624E-01 | ce 0.242 | ppl 1.276
[05-28 04:14:37][flashy.solver][INFO] - Train | Epoch 63 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.124E-01 | ce 0.236 | ppl 1.268
[05-28 04:26:49][flashy.solver][INFO] - Train | Epoch 64 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.961E-01 | ce 0.240 | ppl 1.273
[05-28 04:30:58][flashy.solver][INFO] - Train | Epoch 63 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.311E-01 | ce 0.236 | ppl 1.268
[05-28 04:43:16][flashy.solver][INFO] - Train | Epoch 64 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.742E-01 | ce 0.240 | ppl 1.273
[05-28 04:47:22][flashy.solver][INFO] - Train | Epoch 63 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.796E-01 | ce 0.231 | ppl 1.262
[05-28 04:59:24][flashy.solver][INFO] - Train | Epoch 64 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.187E-01 | ce 0.235 | ppl 1.267
[05-28 05:03:22][flashy.solver][INFO] - Train | Epoch 63 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.688E-01 | ce 0.231 | ppl 1.262
[05-28 05:15:43][flashy.solver][INFO] - Train | Epoch 64 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.618E-01 | ce 0.234 | ppl 1.266
[05-28 05:20:01][flashy.solver][INFO] - Train | Epoch 63 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.311E-01 | ce 0.237 | ppl 1.270
[05-28 05:32:22][flashy.solver][INFO] - Train | Epoch 64 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.403E-01 | ce 0.244 | ppl 1.279
[05-28 05:36:16][flashy.solver][INFO] - Train Summary | Epoch 63 | lr=1.00E-04 | grad_norm=8.167E-01 | ce=0.232 | ppl=1.264 | duration=9794.900
[05-28 05:36:54][flashy.solver][INFO] - Valid | Epoch 63 | 7/79 | 0.21 it/sec | ce 0.210 | ppl 1.236
[05-28 05:37:28][flashy.solver][INFO] - Valid | Epoch 63 | 14/79 | 0.21 it/sec | ce 0.195 | ppl 1.217
[05-28 05:38:00][flashy.solver][INFO] - Valid | Epoch 63 | 21/79 | 0.21 it/sec | ce 0.233 | ppl 1.264
[05-28 05:38:33][flashy.solver][INFO] - Valid | Epoch 63 | 28/79 | 0.21 it/sec | ce 0.199 | ppl 1.224
[05-28 05:39:08][flashy.solver][INFO] - Valid | Epoch 63 | 35/79 | 0.21 it/sec | ce 0.229 | ppl 1.259
[05-28 05:39:40][flashy.solver][INFO] - Valid | Epoch 63 | 42/79 | 0.21 it/sec | ce 0.198 | ppl 1.220
[05-28 05:40:13][flashy.solver][INFO] - Valid | Epoch 63 | 49/79 | 0.21 it/sec | ce 0.267 | ppl 1.309
[05-28 05:40:47][flashy.solver][INFO] - Valid | Epoch 63 | 56/79 | 0.21 it/sec | ce 0.200 | ppl 1.223
[05-28 05:41:22][flashy.solver][INFO] - Valid | Epoch 63 | 63/79 | 0.21 it/sec | ce 0.253 | ppl 1.289
[05-28 05:41:54][flashy.solver][INFO] - Valid | Epoch 63 | 70/79 | 0.21 it/sec | ce 0.235 | ppl 1.265
[05-28 05:42:27][flashy.solver][INFO] - Valid | Epoch 63 | 77/79 | 0.21 it/sec | ce 0.268 | ppl 1.309
[05-28 05:42:28][flashy.solver][INFO] - Valid Summary | Epoch 63 | ce=0.225 | ppl=1.255 | duration=371.545
[05-28 05:42:28][flashy.solver][INFO] - New best state with ce=0.225 (was 0.225)
[05-28 05:42:28][flashy.solver][INFO] - Model hash: b9160e7a02ec13bc0d53d57f19323272f738d564
[05-28 05:42:28][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 05:48:45][flashy.solver][INFO] - Train | Epoch 64 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.422E-01 | ce 0.232 | ppl 1.263
[05-28 05:58:56][flashy.solver][INFO] - Train | Epoch 64 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.011E-01 | ce 0.232 | ppl 1.263
[05-28 06:05:10][flashy.solver][INFO] - Train | Epoch 64 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.043E-01 | ce 0.231 | ppl 1.262
[05-28 06:13:20][dora.distrib][INFO] - world_size is 1, skipping init.
[05-28 06:13:20][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[05-28 06:13:20][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[05-28 06:13:20][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[05-28 06:13:20][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[05-28 06:13:20][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[05-28 06:13:20][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[05-28 06:13:20][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[05-28 06:13:21][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[05-28 06:13:21][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[05-28 06:13:22][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[05-28 06:13:22][flashy.solver][INFO] - No EMA on the model.
[05-28 06:13:22][flashy.solver][INFO] - Model size: 1.21 M params
[05-28 06:13:22][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[05-28 06:13:22][flashy.solver][INFO] - Restoring weights and history.
[05-28 06:13:22][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 06:13:22][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 06:13:22][flashy.solver][INFO] - Model hash: b9160e7a02ec13bc0d53d57f19323272f738d564
[05-28 06:13:22][flashy.solver][INFO] - Replaying past metrics...
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 16 | lr=1.00E-04 | grad_norm=5.373E-01 | ce=0.274 | ppl=1.318 | duration=9552.291
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 16 | ce=0.265 | ppl=1.306 | duration=362.290
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 17 | lr=1.00E-04 | grad_norm=5.495E-01 | ce=0.274 | ppl=1.318 | duration=9593.252
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 17 | ce=0.263 | ppl=1.303 | duration=361.603
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 18 | lr=1.00E-04 | grad_norm=5.303E-01 | ce=0.270 | ppl=1.314 | duration=9601.733
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 18 | ce=0.261 | ppl=1.301 | duration=362.833
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 19 | lr=1.00E-04 | grad_norm=5.436E-01 | ce=0.271 | ppl=1.314 | duration=9603.793
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 19 | ce=0.259 | ppl=1.298 | duration=362.807
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 20 | lr=1.00E-04 | grad_norm=5.256E-01 | ce=0.267 | ppl=1.309 | duration=9513.591
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 20 | ce=0.256 | ppl=1.295 | duration=359.164
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 21 | lr=1.00E-04 | grad_norm=5.398E-01 | ce=0.267 | ppl=1.309 | duration=9521.433
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 21 | ce=0.255 | ppl=1.293 | duration=359.736
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 22 | lr=1.00E-04 | grad_norm=5.273E-01 | ce=0.264 | ppl=1.305 | duration=9522.932
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 22 | ce=0.254 | ppl=1.291 | duration=359.952
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 23 | lr=1.00E-04 | grad_norm=5.359E-01 | ce=0.263 | ppl=1.304 | duration=9520.848
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 23 | ce=0.252 | ppl=1.290 | duration=359.946
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 24 | lr=1.00E-04 | grad_norm=5.356E-01 | ce=0.260 | ppl=1.300 | duration=9517.164
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 24 | ce=0.250 | ppl=1.287 | duration=360.223
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 25 | lr=1.00E-04 | grad_norm=5.350E-01 | ce=0.258 | ppl=1.297 | duration=9526.316
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 25 | ce=0.249 | ppl=1.286 | duration=359.858
[05-28 06:13:22][flashy.solver][INFO] - Evaluate Summary | Epoch 25 | duration=0.000
[05-28 06:13:22][flashy.solver][INFO] - Generate Summary | Epoch 25 | rtf=0.143 | duration=7.398
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 26 | lr=1.00E-04 | grad_norm=5.344E-01 | ce=0.255 | ppl=1.294 | duration=9505.304
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 26 | ce=0.248 | ppl=1.284 | duration=359.841
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 27 | lr=1.00E-04 | grad_norm=5.476E-01 | ce=0.256 | ppl=1.294 | duration=9515.969
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 27 | ce=0.247 | ppl=1.282 | duration=359.513
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 28 | lr=1.00E-04 | grad_norm=5.444E-01 | ce=0.254 | ppl=1.292 | duration=9521.965
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 28 | ce=0.245 | ppl=1.281 | duration=360.116
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 29 | lr=1.00E-04 | grad_norm=5.705E-01 | ce=0.255 | ppl=1.293 | duration=9536.583
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 29 | ce=0.245 | ppl=1.280 | duration=361.077
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 30 | lr=1.00E-04 | grad_norm=5.520E-01 | ce=0.253 | ppl=1.291 | duration=9517.157
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 30 | ce=0.243 | ppl=1.278 | duration=360.175
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 31 | lr=1.00E-04 | grad_norm=5.536E-01 | ce=0.252 | ppl=1.290 | duration=9515.999
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 31 | ce=0.242 | ppl=1.277 | duration=359.757
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 32 | lr=1.00E-04 | grad_norm=5.734E-01 | ce=0.251 | ppl=1.288 | duration=9525.455
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 32 | ce=0.241 | ppl=1.275 | duration=359.850
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 33 | lr=1.00E-04 | grad_norm=5.841E-01 | ce=0.252 | ppl=1.289 | duration=9522.251
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 33 | ce=0.240 | ppl=1.274 | duration=360.855
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 34 | lr=1.00E-04 | grad_norm=5.932E-01 | ce=0.250 | ppl=1.287 | duration=9530.122
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 34 | ce=0.240 | ppl=1.273 | duration=360.152
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 35 | lr=1.00E-04 | grad_norm=5.957E-01 | ce=0.248 | ppl=1.284 | duration=9524.631
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 35 | ce=0.239 | ppl=1.272 | duration=359.587
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 36 | lr=1.00E-04 | grad_norm=6.259E-01 | ce=0.251 | ppl=1.288 | duration=9531.088
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 36 | ce=0.239 | ppl=1.272 | duration=360.096
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 37 | lr=1.00E-04 | grad_norm=6.270E-01 | ce=0.248 | ppl=1.284 | duration=9533.307
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 37 | ce=0.237 | ppl=1.270 | duration=360.546
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 38 | lr=1.00E-04 | grad_norm=6.272E-01 | ce=0.248 | ppl=1.284 | duration=9529.414
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 38 | ce=0.237 | ppl=1.270 | duration=360.767
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 39 | lr=1.00E-04 | grad_norm=6.386E-01 | ce=0.246 | ppl=1.281 | duration=9524.562
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 39 | ce=0.236 | ppl=1.268 | duration=360.385
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 40 | lr=1.00E-04 | grad_norm=6.360E-01 | ce=0.244 | ppl=1.279 | duration=9642.435
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 40 | ce=0.235 | ppl=1.267 | duration=385.528
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 41 | lr=1.00E-04 | grad_norm=6.410E-01 | ce=0.246 | ppl=1.281 | duration=9668.847
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 41 | ce=0.235 | ppl=1.267 | duration=363.285
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 42 | lr=1.00E-04 | grad_norm=6.642E-01 | ce=0.244 | ppl=1.278 | duration=9572.143
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 42 | ce=0.234 | ppl=1.266 | duration=362.071
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 43 | lr=1.00E-04 | grad_norm=6.578E-01 | ce=0.244 | ppl=1.278 | duration=9564.111
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 43 | ce=0.234 | ppl=1.265 | duration=362.399
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 44 | lr=1.00E-04 | grad_norm=6.778E-01 | ce=0.244 | ppl=1.279 | duration=9545.633
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 44 | ce=0.233 | ppl=1.264 | duration=360.226
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 45 | lr=1.00E-04 | grad_norm=6.707E-01 | ce=0.240 | ppl=1.273 | duration=9558.510
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 45 | ce=0.232 | ppl=1.263 | duration=361.292
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 46 | lr=1.00E-04 | grad_norm=7.081E-01 | ce=0.243 | ppl=1.277 | duration=9549.454
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 46 | ce=0.232 | ppl=1.263 | duration=361.476
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 47 | lr=1.00E-04 | grad_norm=6.983E-01 | ce=0.241 | ppl=1.275 | duration=9994.908
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 47 | ce=0.231 | ppl=1.262 | duration=380.415
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 48 | lr=1.00E-04 | grad_norm=7.072E-01 | ce=0.241 | ppl=1.274 | duration=9756.736
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 48 | ce=0.231 | ppl=1.262 | duration=368.529
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 49 | lr=1.00E-04 | grad_norm=7.122E-01 | ce=0.239 | ppl=1.273 | duration=9719.254
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 49 | ce=0.230 | ppl=1.261 | duration=360.200
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 50 | lr=1.00E-04 | grad_norm=7.181E-01 | ce=0.237 | ppl=1.270 | duration=9752.171
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 50 | ce=0.230 | ppl=1.261 | duration=359.721
[05-28 06:13:22][flashy.solver][INFO] - Evaluate Summary | Epoch 50 | duration=0.000
[05-28 06:13:22][flashy.solver][INFO] - Generate Summary | Epoch 50 | rtf=0.134 | duration=6.334
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 51 | lr=1.00E-04 | grad_norm=7.459E-01 | ce=0.243 | ppl=1.277 | duration=9700.451
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 51 | ce=0.230 | ppl=1.260 | duration=360.665
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 52 | lr=1.00E-04 | grad_norm=7.595E-01 | ce=0.241 | ppl=1.274 | duration=9529.466
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 52 | ce=0.229 | ppl=1.260 | duration=360.312
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 53 | lr=1.00E-04 | grad_norm=7.351E-01 | ce=0.239 | ppl=1.273 | duration=9539.188
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 53 | ce=0.229 | ppl=1.259 | duration=359.881
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 54 | lr=1.00E-04 | grad_norm=7.549E-01 | ce=0.239 | ppl=1.272 | duration=9542.972
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 54 | ce=0.228 | ppl=1.258 | duration=368.039
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 55 | lr=1.00E-04 | grad_norm=7.593E-01 | ce=0.237 | ppl=1.269 | duration=9580.192
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 55 | ce=0.228 | ppl=1.258 | duration=380.776
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 56 | lr=1.00E-04 | grad_norm=7.556E-01 | ce=0.237 | ppl=1.270 | duration=10172.497
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 56 | ce=0.227 | ppl=1.257 | duration=368.761
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 57 | lr=1.00E-04 | grad_norm=7.810E-01 | ce=0.238 | ppl=1.271 | duration=9702.417
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 57 | ce=0.227 | ppl=1.257 | duration=364.849
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 58 | lr=1.00E-04 | grad_norm=7.805E-01 | ce=0.237 | ppl=1.269 | duration=9768.635
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 58 | ce=0.227 | ppl=1.256 | duration=368.489
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 59 | lr=1.00E-04 | grad_norm=7.934E-01 | ce=0.234 | ppl=1.266 | duration=9789.248
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 59 | ce=0.226 | ppl=1.256 | duration=368.743
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 60 | lr=1.00E-04 | grad_norm=7.924E-01 | ce=0.235 | ppl=1.267 | duration=9879.320
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 60 | ce=0.227 | ppl=1.256 | duration=370.751
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 61 | lr=1.00E-04 | grad_norm=8.156E-01 | ce=0.237 | ppl=1.269 | duration=9856.496
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 61 | ce=0.226 | ppl=1.255 | duration=370.156
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 62 | lr=1.00E-04 | grad_norm=8.115E-01 | ce=0.233 | ppl=1.265 | duration=9827.438
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 62 | ce=0.225 | ppl=1.255 | duration=366.751
[05-28 06:13:22][flashy.solver][INFO] - Train Summary | Epoch 63 | lr=1.00E-04 | grad_norm=8.167E-01 | ce=0.232 | ppl=1.264 | duration=9794.900
[05-28 06:13:22][flashy.solver][INFO] - Valid Summary | Epoch 63 | ce=0.225 | ppl=1.255 | duration=371.545
[05-28 06:13:28][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[05-28 06:21:35][flashy.solver][INFO] - Train Summary | Epoch 64 | lr=1.00E-04 | grad_norm=8.545E-01 | ce=0.237 | ppl=1.269 | duration=9878.421
[05-28 06:22:13][flashy.solver][INFO] - Valid | Epoch 64 | 7/79 | 0.21 it/sec | ce 0.210 | ppl 1.235
[05-28 06:22:46][flashy.solver][INFO] - Valid | Epoch 64 | 14/79 | 0.21 it/sec | ce 0.194 | ppl 1.217
[05-28 06:23:19][flashy.solver][INFO] - Valid | Epoch 64 | 21/79 | 0.21 it/sec | ce 0.233 | ppl 1.263
[05-28 06:23:51][flashy.solver][INFO] - Valid | Epoch 64 | 28/79 | 0.21 it/sec | ce 0.199 | ppl 1.223
[05-28 06:24:25][flashy.solver][INFO] - Valid | Epoch 64 | 35/79 | 0.21 it/sec | ce 0.229 | ppl 1.259
[05-28 06:24:58][flashy.solver][INFO] - Valid | Epoch 64 | 42/79 | 0.21 it/sec | ce 0.197 | ppl 1.220
[05-28 06:25:31][flashy.solver][INFO] - Valid | Epoch 64 | 49/79 | 0.21 it/sec | ce 0.267 | ppl 1.309
[05-28 06:26:04][flashy.solver][INFO] - Valid | Epoch 64 | 56/79 | 0.21 it/sec | ce 0.200 | ppl 1.222
[05-28 06:26:37][flashy.solver][INFO] - Valid | Epoch 64 | 63/79 | 0.21 it/sec | ce 0.253 | ppl 1.289
[05-28 06:27:12][flashy.solver][INFO] - Valid | Epoch 64 | 70/79 | 0.21 it/sec | ce 0.234 | ppl 1.265
[05-28 06:27:45][flashy.solver][INFO] - Valid | Epoch 64 | 77/79 | 0.21 it/sec | ce 0.267 | ppl 1.307
[05-28 06:27:46][flashy.solver][INFO] - Valid Summary | Epoch 64 | ce=0.225 | ppl=1.254 | duration=370.405
[05-28 06:27:46][flashy.solver][INFO] - New best state with ce=0.225 (was 0.225)
[05-28 06:27:46][flashy.solver][INFO] - Model hash: 8f5e9f00a3fc353f9846c8bf4b1669eb45f45fce
[05-28 06:27:46][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 06:29:58][flashy.solver][INFO] - Train | Epoch 64 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.045E-01 | ce 0.232 | ppl 1.263
[05-28 06:44:43][flashy.solver][INFO] - Train | Epoch 65 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.750E-01 | ce 0.231 | ppl 1.262
[05-28 06:46:55][flashy.solver][INFO] - Train | Epoch 64 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.155E-01 | ce 0.237 | ppl 1.270
[05-28 07:01:11][flashy.solver][INFO] - Train | Epoch 65 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 7.853E-01 | ce 0.236 | ppl 1.268
[05-28 07:17:21][flashy.solver][INFO] - Train | Epoch 65 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.116E-01 | ce 0.239 | ppl 1.272
[05-28 07:33:31][flashy.solver][INFO] - Train | Epoch 65 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.654E-01 | ce 0.232 | ppl 1.264
[05-28 07:49:40][flashy.solver][INFO] - Train | Epoch 65 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.251E-01 | ce 0.231 | ppl 1.262
[05-28 08:05:52][flashy.solver][INFO] - Train | Epoch 65 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.458E-01 | ce 0.240 | ppl 1.273
[05-28 08:22:04][flashy.solver][INFO] - Train | Epoch 65 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.502E-01 | ce 0.232 | ppl 1.264
[05-28 08:38:13][flashy.solver][INFO] - Train | Epoch 65 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.859E-01 | ce 0.231 | ppl 1.262
[05-28 08:54:23][flashy.solver][INFO] - Train | Epoch 65 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.593E-01 | ce 0.238 | ppl 1.271
[05-28 09:10:34][flashy.solver][INFO] - Train Summary | Epoch 65 | lr=1.00E-04 | grad_norm=8.380E-01 | ce=0.235 | ppl=1.267 | duration=9768.115
[05-28 09:11:12][flashy.solver][INFO] - Valid | Epoch 65 | 7/79 | 0.21 it/sec | ce 0.210 | ppl 1.236
[05-28 09:11:44][flashy.solver][INFO] - Valid | Epoch 65 | 14/79 | 0.21 it/sec | ce 0.194 | ppl 1.217
[05-28 09:12:17][flashy.solver][INFO] - Valid | Epoch 65 | 21/79 | 0.21 it/sec | ce 0.233 | ppl 1.263
[05-28 09:12:50][flashy.solver][INFO] - Valid | Epoch 65 | 28/79 | 0.21 it/sec | ce 0.199 | ppl 1.223
[05-28 09:13:23][flashy.solver][INFO] - Valid | Epoch 65 | 35/79 | 0.21 it/sec | ce 0.229 | ppl 1.258
[05-28 09:13:56][flashy.solver][INFO] - Valid | Epoch 65 | 42/79 | 0.21 it/sec | ce 0.197 | ppl 1.219
[05-28 09:14:29][flashy.solver][INFO] - Valid | Epoch 65 | 49/79 | 0.21 it/sec | ce 0.266 | ppl 1.308
[05-28 09:15:02][flashy.solver][INFO] - Valid | Epoch 65 | 56/79 | 0.21 it/sec | ce 0.200 | ppl 1.222
[05-28 09:15:35][flashy.solver][INFO] - Valid | Epoch 65 | 63/79 | 0.21 it/sec | ce 0.253 | ppl 1.288
[05-28 09:16:08][flashy.solver][INFO] - Valid | Epoch 65 | 70/79 | 0.21 it/sec | ce 0.234 | ppl 1.265
[05-28 09:16:41][flashy.solver][INFO] - Valid | Epoch 65 | 77/79 | 0.21 it/sec | ce 0.267 | ppl 1.307
[05-28 09:16:42][flashy.solver][INFO] - Valid Summary | Epoch 65 | ce=0.225 | ppl=1.254 | duration=368.232
[05-28 09:16:42][flashy.solver][INFO] - New best state with ce=0.225 (was 0.225)
[05-28 09:16:42][flashy.solver][INFO] - Model hash: 3e790f4d8199d1d2aafbd9e36db5745335ca462d
[05-28 09:16:42][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 09:32:59][flashy.solver][INFO] - Train | Epoch 66 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.483E-01 | ce 0.233 | ppl 1.265
[05-28 09:49:11][flashy.solver][INFO] - Train | Epoch 66 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.036E-01 | ce 0.234 | ppl 1.266
[05-28 10:05:22][flashy.solver][INFO] - Train | Epoch 66 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.376E-01 | ce 0.234 | ppl 1.266
[05-28 10:21:28][flashy.solver][INFO] - Train | Epoch 66 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.464E-01 | ce 0.235 | ppl 1.267
[05-28 10:37:37][flashy.solver][INFO] - Train | Epoch 66 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.597E-01 | ce 0.236 | ppl 1.269
[05-28 10:53:47][flashy.solver][INFO] - Train | Epoch 66 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.554E-01 | ce 0.226 | ppl 1.256
[05-28 11:09:58][flashy.solver][INFO] - Train | Epoch 66 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.994E-01 | ce 0.235 | ppl 1.267
[05-28 11:26:08][flashy.solver][INFO] - Train | Epoch 66 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.409E-01 | ce 0.233 | ppl 1.266
[05-28 11:42:19][flashy.solver][INFO] - Train | Epoch 66 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.047E-01 | ce 0.240 | ppl 1.273
[05-28 11:58:28][flashy.solver][INFO] - Train Summary | Epoch 66 | lr=1.00E-04 | grad_norm=8.622E-01 | ce=0.234 | ppl=1.266 | duration=9705.978
[05-28 11:59:06][flashy.solver][INFO] - Valid | Epoch 66 | 7/79 | 0.21 it/sec | ce 0.210 | ppl 1.235
[05-28 11:59:38][flashy.solver][INFO] - Valid | Epoch 66 | 14/79 | 0.21 it/sec | ce 0.194 | ppl 1.216
[05-28 12:00:12][flashy.solver][INFO] - Valid | Epoch 66 | 21/79 | 0.21 it/sec | ce 0.233 | ppl 1.263
[05-28 12:00:44][flashy.solver][INFO] - Valid | Epoch 66 | 28/79 | 0.21 it/sec | ce 0.199 | ppl 1.223
[05-28 12:01:17][flashy.solver][INFO] - Valid | Epoch 66 | 35/79 | 0.21 it/sec | ce 0.229 | ppl 1.258
[05-28 12:01:50][flashy.solver][INFO] - Valid | Epoch 66 | 42/79 | 0.21 it/sec | ce 0.197 | ppl 1.220
[05-28 12:02:24][flashy.solver][INFO] - Valid | Epoch 66 | 49/79 | 0.21 it/sec | ce 0.266 | ppl 1.308
[05-28 12:02:57][flashy.solver][INFO] - Valid | Epoch 66 | 56/79 | 0.21 it/sec | ce 0.200 | ppl 1.222
[05-28 12:03:30][flashy.solver][INFO] - Valid | Epoch 66 | 63/79 | 0.21 it/sec | ce 0.253 | ppl 1.288
[05-28 12:04:03][flashy.solver][INFO] - Valid | Epoch 66 | 70/79 | 0.21 it/sec | ce 0.234 | ppl 1.265
[05-28 12:04:36][flashy.solver][INFO] - Valid | Epoch 66 | 77/79 | 0.21 it/sec | ce 0.267 | ppl 1.307
[05-28 12:04:36][flashy.solver][INFO] - Valid Summary | Epoch 66 | ce=0.225 | ppl=1.254 | duration=368.388
[05-28 12:04:36][flashy.solver][INFO] - New best state with ce=0.225 (was 0.225)
[05-28 12:04:36][flashy.solver][INFO] - Model hash: e494933e96d4111ed1605604c9c01bb28cac8e57
[05-28 12:04:37][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 12:20:53][flashy.solver][INFO] - Train | Epoch 67 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.584E-01 | ce 0.239 | ppl 1.272
[05-28 12:37:05][flashy.solver][INFO] - Train | Epoch 67 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.954E-01 | ce 0.239 | ppl 1.272
[05-28 12:53:17][flashy.solver][INFO] - Train | Epoch 67 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.888E-01 | ce 0.234 | ppl 1.266
[05-28 13:09:24][flashy.solver][INFO] - Train | Epoch 67 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.885E-01 | ce 0.232 | ppl 1.264
[05-28 13:25:32][flashy.solver][INFO] - Train | Epoch 67 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.972E-01 | ce 0.235 | ppl 1.267
[05-28 13:41:43][flashy.solver][INFO] - Train | Epoch 67 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.487E-01 | ce 0.233 | ppl 1.264
[05-28 13:57:53][flashy.solver][INFO] - Train | Epoch 67 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.860E-01 | ce 0.239 | ppl 1.272
[05-28 14:14:05][flashy.solver][INFO] - Train | Epoch 67 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.032E-01 | ce 0.234 | ppl 1.266
[05-28 14:30:16][flashy.solver][INFO] - Train | Epoch 67 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.690E-01 | ce 0.235 | ppl 1.267
[05-28 14:46:25][flashy.solver][INFO] - Train Summary | Epoch 67 | lr=1.00E-04 | grad_norm=8.433E-01 | ce=0.235 | ppl=1.267 | duration=9708.521
[05-28 14:47:03][flashy.solver][INFO] - Valid | Epoch 67 | 7/79 | 0.21 it/sec | ce 0.210 | ppl 1.235
[05-28 14:47:35][flashy.solver][INFO] - Valid | Epoch 67 | 14/79 | 0.21 it/sec | ce 0.194 | ppl 1.216
[05-28 14:48:08][flashy.solver][INFO] - Valid | Epoch 67 | 21/79 | 0.21 it/sec | ce 0.232 | ppl 1.263
[05-28 14:48:41][flashy.solver][INFO] - Valid | Epoch 67 | 28/79 | 0.21 it/sec | ce 0.199 | ppl 1.223
[05-28 14:49:13][flashy.solver][INFO] - Valid | Epoch 67 | 35/79 | 0.21 it/sec | ce 0.229 | ppl 1.258
[05-28 14:49:46][flashy.solver][INFO] - Valid | Epoch 67 | 42/79 | 0.21 it/sec | ce 0.197 | ppl 1.219
[05-28 14:50:19][flashy.solver][INFO] - Valid | Epoch 67 | 49/79 | 0.21 it/sec | ce 0.266 | ppl 1.307
[05-28 14:50:52][flashy.solver][INFO] - Valid | Epoch 67 | 56/79 | 0.21 it/sec | ce 0.199 | ppl 1.222
[05-28 14:51:25][flashy.solver][INFO] - Valid | Epoch 67 | 63/79 | 0.21 it/sec | ce 0.252 | ppl 1.287
[05-28 14:51:58][flashy.solver][INFO] - Valid | Epoch 67 | 70/79 | 0.21 it/sec | ce 0.233 | ppl 1.264
[05-28 14:52:31][flashy.solver][INFO] - Valid | Epoch 67 | 77/79 | 0.21 it/sec | ce 0.266 | ppl 1.306
[05-28 14:52:32][flashy.solver][INFO] - Valid Summary | Epoch 67 | ce=0.224 | ppl=1.253 | duration=366.592
[05-28 14:52:32][flashy.solver][INFO] - New best state with ce=0.224 (was 0.225)
[05-28 14:52:32][flashy.solver][INFO] - Model hash: 46e34496664573d6c2a7f86e2e854acb5f01b199
[05-28 14:52:32][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 15:08:48][flashy.solver][INFO] - Train | Epoch 68 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.637E-01 | ce 0.226 | ppl 1.256
[05-28 15:24:55][flashy.solver][INFO] - Train | Epoch 68 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.236E-01 | ce 0.231 | ppl 1.262
[05-28 15:41:02][flashy.solver][INFO] - Train | Epoch 68 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.175E-01 | ce 0.240 | ppl 1.273
[05-28 15:57:11][flashy.solver][INFO] - Train | Epoch 68 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.944E-01 | ce 0.234 | ppl 1.266
[05-28 16:13:19][flashy.solver][INFO] - Train | Epoch 68 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.893E-01 | ce 0.235 | ppl 1.267
[05-28 16:29:26][flashy.solver][INFO] - Train | Epoch 68 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.623E-01 | ce 0.234 | ppl 1.265
[05-28 16:45:32][flashy.solver][INFO] - Train | Epoch 68 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.578E-01 | ce 0.232 | ppl 1.263
[05-28 17:01:42][flashy.solver][INFO] - Train | Epoch 68 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.166E-01 | ce 0.231 | ppl 1.263
[05-28 17:17:48][flashy.solver][INFO] - Train | Epoch 68 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.710E-01 | ce 0.230 | ppl 1.261
[05-28 17:33:53][flashy.solver][INFO] - Train Summary | Epoch 68 | lr=1.00E-04 | grad_norm=8.658E-01 | ce=0.233 | ppl=1.264 | duration=9681.177
[05-28 17:34:30][flashy.solver][INFO] - Valid | Epoch 68 | 7/79 | 0.21 it/sec | ce 0.209 | ppl 1.234
[05-28 17:35:03][flashy.solver][INFO] - Valid | Epoch 68 | 14/79 | 0.21 it/sec | ce 0.194 | ppl 1.216
[05-28 17:35:36][flashy.solver][INFO] - Valid | Epoch 68 | 21/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-28 17:36:09][flashy.solver][INFO] - Valid | Epoch 68 | 28/79 | 0.21 it/sec | ce 0.198 | ppl 1.222
[05-28 17:36:41][flashy.solver][INFO] - Valid | Epoch 68 | 35/79 | 0.21 it/sec | ce 0.228 | ppl 1.258
[05-28 17:37:14][flashy.solver][INFO] - Valid | Epoch 68 | 42/79 | 0.21 it/sec | ce 0.196 | ppl 1.218
[05-28 17:37:47][flashy.solver][INFO] - Valid | Epoch 68 | 49/79 | 0.21 it/sec | ce 0.265 | ppl 1.307
[05-28 17:38:20][flashy.solver][INFO] - Valid | Epoch 68 | 56/79 | 0.21 it/sec | ce 0.199 | ppl 1.221
[05-28 17:38:53][flashy.solver][INFO] - Valid | Epoch 68 | 63/79 | 0.21 it/sec | ce 0.252 | ppl 1.287
[05-28 17:39:26][flashy.solver][INFO] - Valid | Epoch 68 | 70/79 | 0.21 it/sec | ce 0.233 | ppl 1.263
[05-28 17:39:59][flashy.solver][INFO] - Valid | Epoch 68 | 77/79 | 0.21 it/sec | ce 0.266 | ppl 1.305
[05-28 17:40:00][flashy.solver][INFO] - Valid Summary | Epoch 68 | ce=0.224 | ppl=1.253 | duration=366.955
[05-28 17:40:00][flashy.solver][INFO] - New best state with ce=0.224 (was 0.224)
[05-28 17:40:00][flashy.solver][INFO] - Model hash: b442aedb15b5b1ea6280c67d06c28b5802ad8de5
[05-28 17:40:00][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 17:56:14][flashy.solver][INFO] - Train | Epoch 69 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.258E-01 | ce 0.238 | ppl 1.271
[05-28 18:12:24][flashy.solver][INFO] - Train | Epoch 69 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.836E-01 | ce 0.232 | ppl 1.264
[05-28 18:28:34][flashy.solver][INFO] - Train | Epoch 69 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.190E-01 | ce 0.236 | ppl 1.268
[05-28 18:44:40][flashy.solver][INFO] - Train | Epoch 69 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.566E-01 | ce 0.236 | ppl 1.268
[05-28 19:00:46][flashy.solver][INFO] - Train | Epoch 69 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.771E-01 | ce 0.235 | ppl 1.267
[05-28 19:16:53][flashy.solver][INFO] - Train | Epoch 69 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.029E-01 | ce 0.230 | ppl 1.261
[05-28 19:32:58][flashy.solver][INFO] - Train | Epoch 69 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.903E-01 | ce 0.237 | ppl 1.269
[05-28 19:49:05][flashy.solver][INFO] - Train | Epoch 69 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.779E-01 | ce 0.232 | ppl 1.264
[05-28 20:05:11][flashy.solver][INFO] - Train | Epoch 69 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.626E-01 | ce 0.234 | ppl 1.265
[05-28 20:21:16][flashy.solver][INFO] - Train Summary | Epoch 69 | lr=1.00E-04 | grad_norm=8.897E-01 | ce=0.234 | ppl=1.266 | duration=9676.317
[05-28 20:21:54][flashy.solver][INFO] - Valid | Epoch 69 | 7/79 | 0.21 it/sec | ce 0.209 | ppl 1.234
[05-28 20:22:26][flashy.solver][INFO] - Valid | Epoch 69 | 14/79 | 0.21 it/sec | ce 0.193 | ppl 1.215
[05-28 20:22:59][flashy.solver][INFO] - Valid | Epoch 69 | 21/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-28 20:23:31][flashy.solver][INFO] - Valid | Epoch 69 | 28/79 | 0.21 it/sec | ce 0.198 | ppl 1.222
[05-28 20:24:04][flashy.solver][INFO] - Valid | Epoch 69 | 35/79 | 0.21 it/sec | ce 0.228 | ppl 1.257
[05-28 20:24:37][flashy.solver][INFO] - Valid | Epoch 69 | 42/79 | 0.21 it/sec | ce 0.196 | ppl 1.218
[05-28 20:25:10][flashy.solver][INFO] - Valid | Epoch 69 | 49/79 | 0.21 it/sec | ce 0.265 | ppl 1.307
[05-28 20:25:43][flashy.solver][INFO] - Valid | Epoch 69 | 56/79 | 0.21 it/sec | ce 0.199 | ppl 1.221
[05-28 20:26:16][flashy.solver][INFO] - Valid | Epoch 69 | 63/79 | 0.21 it/sec | ce 0.252 | ppl 1.287
[05-28 20:26:48][flashy.solver][INFO] - Valid | Epoch 69 | 70/79 | 0.21 it/sec | ce 0.233 | ppl 1.263
[05-28 20:27:22][flashy.solver][INFO] - Valid | Epoch 69 | 77/79 | 0.21 it/sec | ce 0.266 | ppl 1.305
[05-28 20:27:22][flashy.solver][INFO] - Valid Summary | Epoch 69 | ce=0.224 | ppl=1.253 | duration=365.945
[05-28 20:27:22][flashy.solver][INFO] - New best state with ce=0.224 (was 0.224)
[05-28 20:27:22][flashy.solver][INFO] - Model hash: f942efd95f622944510d4dc8594b2864dbf9b48e
[05-28 20:27:22][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 20:43:37][flashy.solver][INFO] - Train | Epoch 70 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.322E-01 | ce 0.229 | ppl 1.259
[05-28 20:59:49][flashy.solver][INFO] - Train | Epoch 70 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.412E-01 | ce 0.235 | ppl 1.268
[05-28 21:15:59][flashy.solver][INFO] - Train | Epoch 70 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.479E-01 | ce 0.227 | ppl 1.256
[05-28 21:32:09][flashy.solver][INFO] - Train | Epoch 70 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.807E-01 | ce 0.233 | ppl 1.265
[05-28 21:48:14][flashy.solver][INFO] - Train | Epoch 70 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.458E-01 | ce 0.228 | ppl 1.259
[05-28 22:04:21][flashy.solver][INFO] - Train | Epoch 70 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 7.797E-01 | ce 0.233 | ppl 1.265
[05-28 22:20:29][flashy.solver][INFO] - Train | Epoch 70 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.080E-01 | ce 0.232 | ppl 1.263
[05-28 22:36:37][flashy.solver][INFO] - Train | Epoch 70 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.599E-01 | ce 0.230 | ppl 1.261
[05-28 22:52:45][flashy.solver][INFO] - Train | Epoch 70 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.110E-01 | ce 0.232 | ppl 1.263
[05-28 23:08:56][flashy.solver][INFO] - Train Summary | Epoch 70 | lr=1.00E-04 | grad_norm=8.614E-01 | ce=0.231 | ppl=1.262 | duration=9693.598
[05-28 23:09:34][flashy.solver][INFO] - Valid | Epoch 70 | 7/79 | 0.21 it/sec | ce 0.209 | ppl 1.234
[05-28 23:10:07][flashy.solver][INFO] - Valid | Epoch 70 | 14/79 | 0.21 it/sec | ce 0.193 | ppl 1.215
[05-28 23:10:40][flashy.solver][INFO] - Valid | Epoch 70 | 21/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-28 23:11:12][flashy.solver][INFO] - Valid | Epoch 70 | 28/79 | 0.21 it/sec | ce 0.198 | ppl 1.221
[05-28 23:11:45][flashy.solver][INFO] - Valid | Epoch 70 | 35/79 | 0.21 it/sec | ce 0.227 | ppl 1.257
[05-28 23:12:18][flashy.solver][INFO] - Valid | Epoch 70 | 42/79 | 0.21 it/sec | ce 0.196 | ppl 1.218
[05-28 23:12:51][flashy.solver][INFO] - Valid | Epoch 70 | 49/79 | 0.21 it/sec | ce 0.265 | ppl 1.306
[05-28 23:13:24][flashy.solver][INFO] - Valid | Epoch 70 | 56/79 | 0.21 it/sec | ce 0.198 | ppl 1.221
[05-28 23:13:57][flashy.solver][INFO] - Valid | Epoch 70 | 63/79 | 0.21 it/sec | ce 0.251 | ppl 1.286
[05-28 23:14:29][flashy.solver][INFO] - Valid | Epoch 70 | 70/79 | 0.21 it/sec | ce 0.232 | ppl 1.263
[05-28 23:15:02][flashy.solver][INFO] - Valid | Epoch 70 | 77/79 | 0.21 it/sec | ce 0.265 | ppl 1.305
[05-28 23:15:03][flashy.solver][INFO] - Valid Summary | Epoch 70 | ce=0.223 | ppl=1.252 | duration=366.943
[05-28 23:15:03][flashy.solver][INFO] - New best state with ce=0.223 (was 0.224)
[05-28 23:15:03][flashy.solver][INFO] - Model hash: 7941118f2e783857ee9e7f61ad56389aa5fa6349
[05-28 23:15:03][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-28 23:31:15][flashy.solver][INFO] - Train | Epoch 71 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.999E-01 | ce 0.233 | ppl 1.264
[05-28 23:47:21][flashy.solver][INFO] - Train | Epoch 71 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.518E-01 | ce 0.240 | ppl 1.273
[05-29 00:03:28][flashy.solver][INFO] - Train | Epoch 71 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.560E-01 | ce 0.236 | ppl 1.268
[05-29 00:19:37][flashy.solver][INFO] - Train | Epoch 71 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.819E-01 | ce 0.232 | ppl 1.263
[05-29 00:35:43][flashy.solver][INFO] - Train | Epoch 71 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.820E-01 | ce 0.231 | ppl 1.262
[05-29 00:51:48][flashy.solver][INFO] - Train | Epoch 71 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.928E-01 | ce 0.228 | ppl 1.259
[05-29 01:07:56][flashy.solver][INFO] - Train | Epoch 71 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.808E-01 | ce 0.237 | ppl 1.269
[05-29 01:24:06][flashy.solver][INFO] - Train | Epoch 71 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.525E-01 | ce 0.241 | ppl 1.274
[05-29 01:40:17][flashy.solver][INFO] - Train | Epoch 71 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.640E-01 | ce 0.239 | ppl 1.272
[05-29 01:56:26][flashy.solver][INFO] - Train Summary | Epoch 71 | lr=1.00E-04 | grad_norm=9.080E-01 | ce=0.235 | ppl=1.268 | duration=9683.285
[05-29 01:57:04][flashy.solver][INFO] - Valid | Epoch 71 | 7/79 | 0.21 it/sec | ce 0.208 | ppl 1.233
[05-29 01:57:37][flashy.solver][INFO] - Valid | Epoch 71 | 14/79 | 0.21 it/sec | ce 0.193 | ppl 1.215
[05-29 01:58:10][flashy.solver][INFO] - Valid | Epoch 71 | 21/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-29 01:58:42][flashy.solver][INFO] - Valid | Epoch 71 | 28/79 | 0.21 it/sec | ce 0.198 | ppl 1.221
[05-29 01:59:15][flashy.solver][INFO] - Valid | Epoch 71 | 35/79 | 0.21 it/sec | ce 0.227 | ppl 1.256
[05-29 01:59:48][flashy.solver][INFO] - Valid | Epoch 71 | 42/79 | 0.21 it/sec | ce 0.195 | ppl 1.217
[05-29 02:00:21][flashy.solver][INFO] - Valid | Epoch 71 | 49/79 | 0.21 it/sec | ce 0.264 | ppl 1.306
[05-29 02:00:54][flashy.solver][INFO] - Valid | Epoch 71 | 56/79 | 0.21 it/sec | ce 0.198 | ppl 1.221
[05-29 02:01:27][flashy.solver][INFO] - Valid | Epoch 71 | 63/79 | 0.21 it/sec | ce 0.251 | ppl 1.286
[05-29 02:02:00][flashy.solver][INFO] - Valid | Epoch 71 | 70/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-29 02:02:34][flashy.solver][INFO] - Valid | Epoch 71 | 77/79 | 0.21 it/sec | ce 0.265 | ppl 1.304
[05-29 02:02:34][flashy.solver][INFO] - Valid Summary | Epoch 71 | ce=0.223 | ppl=1.252 | duration=368.008
[05-29 02:02:34][flashy.solver][INFO] - New best state with ce=0.223 (was 0.223)
[05-29 02:02:34][flashy.solver][INFO] - Model hash: 47bb4b7faf27c7b2ff88ffe5d74aa7ff4d45d019
[05-29 02:02:34][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-29 02:18:47][flashy.solver][INFO] - Train | Epoch 72 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.936E-01 | ce 0.230 | ppl 1.260
[05-29 02:34:54][flashy.solver][INFO] - Train | Epoch 72 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.677E-01 | ce 0.231 | ppl 1.262
[05-29 02:51:05][flashy.solver][INFO] - Train | Epoch 72 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.194E-01 | ce 0.237 | ppl 1.270
[05-29 03:07:12][flashy.solver][INFO] - Train | Epoch 72 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.369E-01 | ce 0.232 | ppl 1.264
[05-29 03:23:23][flashy.solver][INFO] - Train | Epoch 72 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.994E-01 | ce 0.234 | ppl 1.267
[05-29 03:39:32][flashy.solver][INFO] - Train | Epoch 72 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.753E-01 | ce 0.238 | ppl 1.270
[05-29 03:55:40][flashy.solver][INFO] - Train | Epoch 72 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.738E-01 | ce 0.230 | ppl 1.261
[05-29 04:11:51][flashy.solver][INFO] - Train | Epoch 72 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.166E-01 | ce 0.234 | ppl 1.266
[05-29 04:28:01][flashy.solver][INFO] - Train | Epoch 72 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.453E-01 | ce 0.233 | ppl 1.265
[05-29 04:44:09][flashy.solver][INFO] - Train Summary | Epoch 72 | lr=1.00E-04 | grad_norm=8.908E-01 | ce=0.233 | ppl=1.265 | duration=9695.132
[05-29 04:44:47][flashy.solver][INFO] - Valid | Epoch 72 | 7/79 | 0.21 it/sec | ce 0.208 | ppl 1.233
[05-29 04:45:19][flashy.solver][INFO] - Valid | Epoch 72 | 14/79 | 0.21 it/sec | ce 0.192 | ppl 1.214
[05-29 04:45:52][flashy.solver][INFO] - Valid | Epoch 72 | 21/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-29 04:46:25][flashy.solver][INFO] - Valid | Epoch 72 | 28/79 | 0.21 it/sec | ce 0.197 | ppl 1.221
[05-29 04:46:58][flashy.solver][INFO] - Valid | Epoch 72 | 35/79 | 0.21 it/sec | ce 0.227 | ppl 1.256
[05-29 04:47:31][flashy.solver][INFO] - Valid | Epoch 72 | 42/79 | 0.21 it/sec | ce 0.195 | ppl 1.217
[05-29 04:48:04][flashy.solver][INFO] - Valid | Epoch 72 | 49/79 | 0.21 it/sec | ce 0.264 | ppl 1.305
[05-29 04:48:37][flashy.solver][INFO] - Valid | Epoch 72 | 56/79 | 0.21 it/sec | ce 0.198 | ppl 1.220
[05-29 04:49:10][flashy.solver][INFO] - Valid | Epoch 72 | 63/79 | 0.21 it/sec | ce 0.250 | ppl 1.285
[05-29 04:49:42][flashy.solver][INFO] - Valid | Epoch 72 | 70/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-29 04:50:16][flashy.solver][INFO] - Valid | Epoch 72 | 77/79 | 0.21 it/sec | ce 0.264 | ppl 1.304
[05-29 04:50:16][flashy.solver][INFO] - Valid Summary | Epoch 72 | ce=0.222 | ppl=1.251 | duration=366.920
[05-29 04:50:16][flashy.solver][INFO] - New best state with ce=0.222 (was 0.223)
[05-29 04:50:16][flashy.solver][INFO] - Model hash: de07a6d77465c17f93a37df3d9ed0f37d0dcbd62
[05-29 04:50:16][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-29 05:06:32][flashy.solver][INFO] - Train | Epoch 73 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.617E-01 | ce 0.230 | ppl 1.260
[05-29 05:22:42][flashy.solver][INFO] - Train | Epoch 73 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.097E-01 | ce 0.238 | ppl 1.271
[05-29 05:38:51][flashy.solver][INFO] - Train | Epoch 73 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.113E-01 | ce 0.235 | ppl 1.267
[05-29 05:55:03][flashy.solver][INFO] - Train | Epoch 73 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.943E-01 | ce 0.233 | ppl 1.264
[05-29 06:11:11][flashy.solver][INFO] - Train | Epoch 73 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.750E-01 | ce 0.235 | ppl 1.267
[05-29 06:27:20][flashy.solver][INFO] - Train | Epoch 73 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.657E-01 | ce 0.236 | ppl 1.268
[05-29 06:43:32][flashy.solver][INFO] - Train | Epoch 73 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.252E-01 | ce 0.239 | ppl 1.272
[05-29 06:59:46][flashy.solver][INFO] - Train | Epoch 73 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.655E-01 | ce 0.238 | ppl 1.271
[05-29 07:15:54][flashy.solver][INFO] - Train | Epoch 73 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.953E-01 | ce 0.236 | ppl 1.268
[05-29 07:32:04][flashy.solver][INFO] - Train Summary | Epoch 73 | lr=1.00E-04 | grad_norm=9.097E-01 | ce=0.235 | ppl=1.267 | duration=9707.760
[05-29 07:32:42][flashy.solver][INFO] - Valid | Epoch 73 | 7/79 | 0.21 it/sec | ce 0.208 | ppl 1.233
[05-29 07:33:14][flashy.solver][INFO] - Valid | Epoch 73 | 14/79 | 0.21 it/sec | ce 0.193 | ppl 1.215
[05-29 07:33:48][flashy.solver][INFO] - Valid | Epoch 73 | 21/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-29 07:34:20][flashy.solver][INFO] - Valid | Epoch 73 | 28/79 | 0.21 it/sec | ce 0.197 | ppl 1.221
[05-29 07:34:53][flashy.solver][INFO] - Valid | Epoch 73 | 35/79 | 0.21 it/sec | ce 0.227 | ppl 1.256
[05-29 07:35:26][flashy.solver][INFO] - Valid | Epoch 73 | 42/79 | 0.21 it/sec | ce 0.195 | ppl 1.217
[05-29 07:35:59][flashy.solver][INFO] - Valid | Epoch 73 | 49/79 | 0.21 it/sec | ce 0.264 | ppl 1.305
[05-29 07:36:32][flashy.solver][INFO] - Valid | Epoch 73 | 56/79 | 0.21 it/sec | ce 0.198 | ppl 1.220
[05-29 07:37:05][flashy.solver][INFO] - Valid | Epoch 73 | 63/79 | 0.21 it/sec | ce 0.250 | ppl 1.285
[05-29 07:37:38][flashy.solver][INFO] - Valid | Epoch 73 | 70/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-29 07:38:12][flashy.solver][INFO] - Valid | Epoch 73 | 77/79 | 0.21 it/sec | ce 0.264 | ppl 1.304
[05-29 07:38:12][flashy.solver][INFO] - Valid Summary | Epoch 73 | ce=0.223 | ppl=1.251 | duration=368.145
[05-29 07:38:12][flashy.solver][INFO] - Model hash: ae713329f652ee45054732bfc8da526dd83a6571
[05-29 07:38:12][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-29 07:54:28][flashy.solver][INFO] - Train | Epoch 74 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.392E-01 | ce 0.233 | ppl 1.265
[05-29 08:10:31][flashy.solver][INFO] - Train | Epoch 74 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.696E-01 | ce 0.230 | ppl 1.260
[05-29 08:26:40][flashy.solver][INFO] - Train | Epoch 74 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.166E-01 | ce 0.238 | ppl 1.271
[05-29 08:42:45][flashy.solver][INFO] - Train | Epoch 74 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.526E-01 | ce 0.239 | ppl 1.272
[05-29 08:58:48][flashy.solver][INFO] - Train | Epoch 74 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.418E-01 | ce 0.233 | ppl 1.265
[05-29 09:14:56][flashy.solver][INFO] - Train | Epoch 74 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.657E-01 | ce 0.234 | ppl 1.266
[05-29 09:31:01][flashy.solver][INFO] - Train | Epoch 74 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.559E-01 | ce 0.230 | ppl 1.262
[05-29 09:47:11][flashy.solver][INFO] - Train | Epoch 74 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.892E-01 | ce 0.240 | ppl 1.274
[05-29 10:03:07][flashy.solver][INFO] - Train | Epoch 74 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.693E-01 | ce 0.232 | ppl 1.263
[05-29 10:19:03][flashy.solver][INFO] - Train Summary | Epoch 74 | lr=1.00E-04 | grad_norm=9.211E-01 | ce=0.234 | ppl=1.266 | duration=9650.362
[05-29 10:19:40][flashy.solver][INFO] - Valid | Epoch 74 | 7/79 | 0.22 it/sec | ce 0.208 | ppl 1.233
[05-29 10:20:12][flashy.solver][INFO] - Valid | Epoch 74 | 14/79 | 0.22 it/sec | ce 0.192 | ppl 1.214
[05-29 10:20:44][flashy.solver][INFO] - Valid | Epoch 74 | 21/79 | 0.22 it/sec | ce 0.231 | ppl 1.261
[05-29 10:21:16][flashy.solver][INFO] - Valid | Epoch 74 | 28/79 | 0.22 it/sec | ce 0.197 | ppl 1.221
[05-29 10:21:49][flashy.solver][INFO] - Valid | Epoch 74 | 35/79 | 0.22 it/sec | ce 0.227 | ppl 1.256
[05-29 10:22:21][flashy.solver][INFO] - Valid | Epoch 74 | 42/79 | 0.22 it/sec | ce 0.195 | ppl 1.217
[05-29 10:22:54][flashy.solver][INFO] - Valid | Epoch 74 | 49/79 | 0.22 it/sec | ce 0.264 | ppl 1.305
[05-29 10:23:26][flashy.solver][INFO] - Valid | Epoch 74 | 56/79 | 0.22 it/sec | ce 0.198 | ppl 1.220
[05-29 10:23:59][flashy.solver][INFO] - Valid | Epoch 74 | 63/79 | 0.22 it/sec | ce 0.250 | ppl 1.285
[05-29 10:24:31][flashy.solver][INFO] - Valid | Epoch 74 | 70/79 | 0.22 it/sec | ce 0.232 | ppl 1.262
[05-29 10:25:04][flashy.solver][INFO] - Valid | Epoch 74 | 77/79 | 0.22 it/sec | ce 0.264 | ppl 1.303
[05-29 10:25:04][flashy.solver][INFO] - Valid Summary | Epoch 74 | ce=0.222 | ppl=1.251 | duration=361.467
[05-29 10:25:04][flashy.solver][INFO] - New best state with ce=0.222 (was 0.222)
[05-29 10:25:04][flashy.solver][INFO] - Model hash: 651702dd5a866b7147960fd6d4a8d31b2af6f055
[05-29 10:25:04][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-29 10:41:05][flashy.solver][INFO] - Train | Epoch 75 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.759E-01 | ce 0.234 | ppl 1.266
[05-29 10:57:04][flashy.solver][INFO] - Train | Epoch 75 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.720E-01 | ce 0.236 | ppl 1.269
[05-29 11:13:01][flashy.solver][INFO] - Train | Epoch 75 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.070E-01 | ce 0.230 | ppl 1.261
[05-29 11:28:58][flashy.solver][INFO] - Train | Epoch 75 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.825E-01 | ce 0.236 | ppl 1.269
[05-29 11:44:58][flashy.solver][INFO] - Train | Epoch 75 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.139E-01 | ce 0.225 | ppl 1.255
[05-29 12:01:08][flashy.solver][INFO] - Train | Epoch 75 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.270E-01 | ce 0.235 | ppl 1.268
[05-29 12:17:17][flashy.solver][INFO] - Train | Epoch 75 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.922E-01 | ce 0.238 | ppl 1.270
[05-29 12:33:25][flashy.solver][INFO] - Train | Epoch 75 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.654E-01 | ce 0.227 | ppl 1.258
[05-29 12:49:31][flashy.solver][INFO] - Train | Epoch 75 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.145E-01 | ce 0.235 | ppl 1.267
[05-29 13:05:38][flashy.solver][INFO] - Train Summary | Epoch 75 | lr=1.00E-04 | grad_norm=9.272E-01 | ce=0.233 | ppl=1.265 | duration=9633.944
[05-29 13:06:16][flashy.solver][INFO] - Valid | Epoch 75 | 7/79 | 0.21 it/sec | ce 0.208 | ppl 1.233
[05-29 13:06:48][flashy.solver][INFO] - Valid | Epoch 75 | 14/79 | 0.21 it/sec | ce 0.192 | ppl 1.214
[05-29 13:07:21][flashy.solver][INFO] - Valid | Epoch 75 | 21/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-29 13:07:54][flashy.solver][INFO] - Valid | Epoch 75 | 28/79 | 0.21 it/sec | ce 0.197 | ppl 1.221
[05-29 13:08:27][flashy.solver][INFO] - Valid | Epoch 75 | 35/79 | 0.21 it/sec | ce 0.227 | ppl 1.256
[05-29 13:09:00][flashy.solver][INFO] - Valid | Epoch 75 | 42/79 | 0.21 it/sec | ce 0.195 | ppl 1.216
[05-29 13:09:33][flashy.solver][INFO] - Valid | Epoch 75 | 49/79 | 0.21 it/sec | ce 0.264 | ppl 1.305
[05-29 13:10:06][flashy.solver][INFO] - Valid | Epoch 75 | 56/79 | 0.21 it/sec | ce 0.198 | ppl 1.220
[05-29 13:10:39][flashy.solver][INFO] - Valid | Epoch 75 | 63/79 | 0.21 it/sec | ce 0.250 | ppl 1.285
[05-29 13:11:12][flashy.solver][INFO] - Valid | Epoch 75 | 70/79 | 0.21 it/sec | ce 0.232 | ppl 1.262
[05-29 13:11:45][flashy.solver][INFO] - Valid | Epoch 75 | 77/79 | 0.21 it/sec | ce 0.264 | ppl 1.303
[05-29 13:11:46][flashy.solver][INFO] - Valid Summary | Epoch 75 | ce=0.222 | ppl=1.251 | duration=367.500
[05-29 13:11:46][flashy.solver][INFO] - New best state with ce=0.222 (was 0.222)
[05-29 13:11:46][flashy.solver][INFO] - Evaluate Summary | Epoch 75 | duration=0.000
[05-29 13:11:46][flashy.solver][INFO] - Generating samples in /tmp/audiocraft_root/xps/a4911d1c/samples
[05-29 13:11:52][flashy.solver][INFO] - Generate Summary | Epoch 75 | rtf=0.144 | duration=6.500
[05-29 13:11:52][flashy.solver][INFO] - Model hash: 0e5317f2c203459a5308c54d307802bb85da86b6
[05-29 13:11:52][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-29 13:28:08][flashy.solver][INFO] - Train | Epoch 76 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.427E-01 | ce 0.234 | ppl 1.265
[05-29 13:44:11][flashy.solver][INFO] - Train | Epoch 76 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.309E-01 | ce 0.232 | ppl 1.264
[05-29 14:00:16][flashy.solver][INFO] - Train | Epoch 76 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.059E+00 | ce 0.238 | ppl 1.272
[05-29 14:16:21][flashy.solver][INFO] - Train | Epoch 76 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.096E-01 | ce 0.225 | ppl 1.254
[05-29 14:32:28][flashy.solver][INFO] - Train | Epoch 76 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.276E-01 | ce 0.232 | ppl 1.263
[05-29 14:48:37][flashy.solver][INFO] - Train | Epoch 76 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.335E-01 | ce 0.234 | ppl 1.266
[05-29 15:04:43][flashy.solver][INFO] - Train | Epoch 76 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.562E-01 | ce 0.231 | ppl 1.263
[05-29 15:20:49][flashy.solver][INFO] - Train | Epoch 76 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.672E-01 | ce 0.231 | ppl 1.262
[05-29 15:36:54][flashy.solver][INFO] - Train | Epoch 76 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.255E-01 | ce 0.232 | ppl 1.264
[05-29 15:52:59][flashy.solver][INFO] - Train Summary | Epoch 76 | lr=1.00E-04 | grad_norm=9.470E-01 | ce=0.232 | ppl=1.263 | duration=9666.764
[05-29 15:53:37][flashy.solver][INFO] - Valid | Epoch 76 | 7/79 | 0.21 it/sec | ce 0.207 | ppl 1.232
[05-29 15:54:09][flashy.solver][INFO] - Valid | Epoch 76 | 14/79 | 0.21 it/sec | ce 0.191 | ppl 1.213
[05-29 15:54:42][flashy.solver][INFO] - Valid | Epoch 76 | 21/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-29 15:55:14][flashy.solver][INFO] - Valid | Epoch 76 | 28/79 | 0.21 it/sec | ce 0.197 | ppl 1.220
[05-29 15:55:47][flashy.solver][INFO] - Valid | Epoch 76 | 35/79 | 0.21 it/sec | ce 0.226 | ppl 1.255
[05-29 15:56:20][flashy.solver][INFO] - Valid | Epoch 76 | 42/79 | 0.21 it/sec | ce 0.194 | ppl 1.216
[05-29 15:56:53][flashy.solver][INFO] - Valid | Epoch 76 | 49/79 | 0.21 it/sec | ce 0.263 | ppl 1.304
[05-29 15:57:25][flashy.solver][INFO] - Valid | Epoch 76 | 56/79 | 0.21 it/sec | ce 0.197 | ppl 1.219
[05-29 15:57:58][flashy.solver][INFO] - Valid | Epoch 76 | 63/79 | 0.21 it/sec | ce 0.249 | ppl 1.284
[05-29 15:58:31][flashy.solver][INFO] - Valid | Epoch 76 | 70/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-29 15:59:04][flashy.solver][INFO] - Valid | Epoch 76 | 77/79 | 0.21 it/sec | ce 0.263 | ppl 1.302
[05-29 15:59:05][flashy.solver][INFO] - Valid Summary | Epoch 76 | ce=0.222 | ppl=1.250 | duration=365.508
[05-29 15:59:05][flashy.solver][INFO] - New best state with ce=0.222 (was 0.222)
[05-29 15:59:05][flashy.solver][INFO] - Model hash: bdd6f5b9c24155683173c5254e856a47e70c31c4
[05-29 15:59:05][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-29 16:15:17][flashy.solver][INFO] - Train | Epoch 77 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.187E-01 | ce 0.232 | ppl 1.263
[05-29 16:31:26][flashy.solver][INFO] - Train | Epoch 77 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.121E-01 | ce 0.223 | ppl 1.252
[05-29 16:47:28][flashy.solver][INFO] - Train | Epoch 77 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.354E-01 | ce 0.230 | ppl 1.260
[05-29 17:03:36][flashy.solver][INFO] - Train | Epoch 77 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.244E-01 | ce 0.228 | ppl 1.259
[05-29 17:19:45][flashy.solver][INFO] - Train | Epoch 77 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.609E-01 | ce 0.231 | ppl 1.263
[05-29 17:35:53][flashy.solver][INFO] - Train | Epoch 77 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.752E-01 | ce 0.228 | ppl 1.259
[05-29 17:51:59][flashy.solver][INFO] - Train | Epoch 77 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.591E-01 | ce 0.228 | ppl 1.259
[05-29 18:08:03][flashy.solver][INFO] - Train | Epoch 77 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.367E-01 | ce 0.235 | ppl 1.268
[05-29 18:24:08][flashy.solver][INFO] - Train | Epoch 77 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.728E-01 | ce 0.229 | ppl 1.260
[05-29 18:40:17][flashy.solver][INFO] - Train Summary | Epoch 77 | lr=1.00E-04 | grad_norm=9.267E-01 | ce=0.230 | ppl=1.261 | duration=9671.955
[05-29 18:40:54][flashy.solver][INFO] - Valid | Epoch 77 | 7/79 | 0.21 it/sec | ce 0.208 | ppl 1.232
[05-29 18:41:26][flashy.solver][INFO] - Valid | Epoch 77 | 14/79 | 0.22 it/sec | ce 0.192 | ppl 1.213
[05-29 18:41:59][flashy.solver][INFO] - Valid | Epoch 77 | 21/79 | 0.21 it/sec | ce 0.230 | ppl 1.259
[05-29 18:42:32][flashy.solver][INFO] - Valid | Epoch 77 | 28/79 | 0.21 it/sec | ce 0.197 | ppl 1.220
[05-29 18:43:04][flashy.solver][INFO] - Valid | Epoch 77 | 35/79 | 0.21 it/sec | ce 0.226 | ppl 1.255
[05-29 18:43:37][flashy.solver][INFO] - Valid | Epoch 77 | 42/79 | 0.21 it/sec | ce 0.194 | ppl 1.216
[05-29 18:44:10][flashy.solver][INFO] - Valid | Epoch 77 | 49/79 | 0.21 it/sec | ce 0.263 | ppl 1.304
[05-29 18:44:43][flashy.solver][INFO] - Valid | Epoch 77 | 56/79 | 0.21 it/sec | ce 0.197 | ppl 1.219
[05-29 18:45:16][flashy.solver][INFO] - Valid | Epoch 77 | 63/79 | 0.21 it/sec | ce 0.249 | ppl 1.284
[05-29 18:45:49][flashy.solver][INFO] - Valid | Epoch 77 | 70/79 | 0.21 it/sec | ce 0.231 | ppl 1.261
[05-29 18:46:22][flashy.solver][INFO] - Valid | Epoch 77 | 77/79 | 0.21 it/sec | ce 0.263 | ppl 1.302
[05-29 18:46:22][flashy.solver][INFO] - Valid Summary | Epoch 77 | ce=0.222 | ppl=1.250 | duration=365.621
[05-29 18:46:22][flashy.solver][INFO] - New best state with ce=0.222 (was 0.222)
[05-29 18:46:22][flashy.solver][INFO] - Model hash: 73669a6a955c5e993b5ef6746056b6d3434ea578
[05-29 18:46:22][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-29 19:02:35][flashy.solver][INFO] - Train | Epoch 78 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.629E-01 | ce 0.227 | ppl 1.258
[05-29 19:18:41][flashy.solver][INFO] - Train | Epoch 78 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.195E-01 | ce 0.228 | ppl 1.257
[05-29 19:34:53][flashy.solver][INFO] - Train | Epoch 78 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.165E-01 | ce 0.244 | ppl 1.279
[05-29 19:50:52][flashy.solver][INFO] - Train | Epoch 78 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.759E-01 | ce 0.233 | ppl 1.265
[05-29 20:06:47][flashy.solver][INFO] - Train | Epoch 78 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.033E-01 | ce 0.229 | ppl 1.259
[05-29 20:22:42][flashy.solver][INFO] - Train | Epoch 78 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.096E+00 | ce 0.231 | ppl 1.263
[05-29 20:38:41][flashy.solver][INFO] - Train | Epoch 78 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.356E-01 | ce 0.226 | ppl 1.256
[05-29 20:54:36][flashy.solver][INFO] - Train | Epoch 78 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.040E-01 | ce 0.230 | ppl 1.260
[05-29 21:10:35][flashy.solver][INFO] - Train | Epoch 78 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.474E-01 | ce 0.237 | ppl 1.270
[05-29 21:26:30][flashy.solver][INFO] - Train Summary | Epoch 78 | lr=1.00E-04 | grad_norm=9.488E-01 | ce=0.232 | ppl=1.263 | duration=9607.441
[05-29 21:27:07][flashy.solver][INFO] - Valid | Epoch 78 | 7/79 | 0.22 it/sec | ce 0.207 | ppl 1.232
[05-29 21:27:39][flashy.solver][INFO] - Valid | Epoch 78 | 14/79 | 0.22 it/sec | ce 0.191 | ppl 1.213
[05-29 21:28:12][flashy.solver][INFO] - Valid | Epoch 78 | 21/79 | 0.22 it/sec | ce 0.229 | ppl 1.259
[05-29 21:28:43][flashy.solver][INFO] - Valid | Epoch 78 | 28/79 | 0.22 it/sec | ce 0.196 | ppl 1.220
[05-29 21:29:16][flashy.solver][INFO] - Valid | Epoch 78 | 35/79 | 0.22 it/sec | ce 0.226 | ppl 1.254
[05-29 21:29:48][flashy.solver][INFO] - Valid | Epoch 78 | 42/79 | 0.22 it/sec | ce 0.194 | ppl 1.216
[05-29 21:30:21][flashy.solver][INFO] - Valid | Epoch 78 | 49/79 | 0.22 it/sec | ce 0.263 | ppl 1.304
[05-29 21:30:53][flashy.solver][INFO] - Valid | Epoch 78 | 56/79 | 0.22 it/sec | ce 0.197 | ppl 1.219
[05-29 21:31:26][flashy.solver][INFO] - Valid | Epoch 78 | 63/79 | 0.22 it/sec | ce 0.249 | ppl 1.284
[05-29 21:31:58][flashy.solver][INFO] - Valid | Epoch 78 | 70/79 | 0.22 it/sec | ce 0.231 | ppl 1.261
[05-29 21:32:31][flashy.solver][INFO] - Valid | Epoch 78 | 77/79 | 0.22 it/sec | ce 0.263 | ppl 1.302
[05-29 21:32:31][flashy.solver][INFO] - Valid Summary | Epoch 78 | ce=0.222 | ppl=1.250 | duration=361.455
[05-29 21:32:31][flashy.solver][INFO] - New best state with ce=0.222 (was 0.222)
[05-29 21:32:31][flashy.solver][INFO] - Model hash: 318245bef308280ebb10533dc15f81dbbcbe92d4
[05-29 21:32:31][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-29 21:48:32][flashy.solver][INFO] - Train | Epoch 79 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.568E-01 | ce 0.231 | ppl 1.262
[05-29 22:04:31][flashy.solver][INFO] - Train | Epoch 79 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.677E-01 | ce 0.226 | ppl 1.256
[05-29 22:20:24][flashy.solver][INFO] - Train | Epoch 79 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.830E-01 | ce 0.230 | ppl 1.261
[05-29 22:36:21][flashy.solver][INFO] - Train | Epoch 79 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.893E-01 | ce 0.229 | ppl 1.259
[05-29 22:52:16][flashy.solver][INFO] - Train | Epoch 79 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.615E-01 | ce 0.229 | ppl 1.259
[05-29 23:08:14][flashy.solver][INFO] - Train | Epoch 79 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.265E-01 | ce 0.231 | ppl 1.262
[05-29 23:24:11][flashy.solver][INFO] - Train | Epoch 79 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.000E+00 | ce 0.231 | ppl 1.263
[05-29 23:40:07][flashy.solver][INFO] - Train | Epoch 79 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.710E-01 | ce 0.235 | ppl 1.267
[05-29 23:56:07][flashy.solver][INFO] - Train | Epoch 79 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.275E-01 | ce 0.235 | ppl 1.267
[05-30 00:12:01][flashy.solver][INFO] - Train Summary | Epoch 79 | lr=1.00E-04 | grad_norm=9.638E-01 | ce=0.230 | ppl=1.261 | duration=9569.696
[05-30 00:12:38][flashy.solver][INFO] - Valid | Epoch 79 | 7/79 | 0.22 it/sec | ce 0.207 | ppl 1.232
[05-30 00:13:10][flashy.solver][INFO] - Valid | Epoch 79 | 14/79 | 0.22 it/sec | ce 0.191 | ppl 1.213
[05-30 00:13:43][flashy.solver][INFO] - Valid | Epoch 79 | 21/79 | 0.22 it/sec | ce 0.229 | ppl 1.258
[05-30 00:14:15][flashy.solver][INFO] - Valid | Epoch 79 | 28/79 | 0.22 it/sec | ce 0.196 | ppl 1.219
[05-30 00:14:47][flashy.solver][INFO] - Valid | Epoch 79 | 35/79 | 0.22 it/sec | ce 0.225 | ppl 1.254
[05-30 00:15:19][flashy.solver][INFO] - Valid | Epoch 79 | 42/79 | 0.22 it/sec | ce 0.194 | ppl 1.215
[05-30 00:15:52][flashy.solver][INFO] - Valid | Epoch 79 | 49/79 | 0.22 it/sec | ce 0.262 | ppl 1.303
[05-30 00:16:24][flashy.solver][INFO] - Valid | Epoch 79 | 56/79 | 0.22 it/sec | ce 0.197 | ppl 1.219
[05-30 00:16:57][flashy.solver][INFO] - Valid | Epoch 79 | 63/79 | 0.22 it/sec | ce 0.249 | ppl 1.283
[05-30 00:17:29][flashy.solver][INFO] - Valid | Epoch 79 | 70/79 | 0.22 it/sec | ce 0.231 | ppl 1.260
[05-30 00:18:02][flashy.solver][INFO] - Valid | Epoch 79 | 77/79 | 0.22 it/sec | ce 0.263 | ppl 1.301
[05-30 00:18:02][flashy.solver][INFO] - Valid Summary | Epoch 79 | ce=0.221 | ppl=1.250 | duration=361.342
[05-30 00:18:02][flashy.solver][INFO] - New best state with ce=0.221 (was 0.222)
[05-30 00:18:02][flashy.solver][INFO] - Model hash: c636e9f33786aacead0ad929ca00ed238f373001
[05-30 00:18:02][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 00:34:03][flashy.solver][INFO] - Train | Epoch 80 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.409E-01 | ce 0.234 | ppl 1.266
[05-30 00:49:58][flashy.solver][INFO] - Train | Epoch 80 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.539E-01 | ce 0.225 | ppl 1.255
[05-30 01:05:53][flashy.solver][INFO] - Train | Epoch 80 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.575E-01 | ce 0.228 | ppl 1.257
[05-30 01:21:49][flashy.solver][INFO] - Train | Epoch 80 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.035E-01 | ce 0.230 | ppl 1.260
[05-30 01:37:43][flashy.solver][INFO] - Train | Epoch 80 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.170E-01 | ce 0.231 | ppl 1.262
[05-30 01:53:40][flashy.solver][INFO] - Train | Epoch 80 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.400E-01 | ce 0.228 | ppl 1.258
[05-30 02:09:45][flashy.solver][INFO] - Train | Epoch 80 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.532E-01 | ce 0.227 | ppl 1.257
[05-30 02:25:51][flashy.solver][INFO] - Train | Epoch 80 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.378E-01 | ce 0.230 | ppl 1.261
[05-30 02:41:59][flashy.solver][INFO] - Train | Epoch 80 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.634E-01 | ce 0.223 | ppl 1.253
[05-30 02:58:05][flashy.solver][INFO] - Train Summary | Epoch 80 | lr=1.00E-04 | grad_norm=9.486E-01 | ce=0.229 | ppl=1.260 | duration=9602.914
[05-30 02:58:43][flashy.solver][INFO] - Valid | Epoch 80 | 7/79 | 0.21 it/sec | ce 0.207 | ppl 1.231
[05-30 02:59:15][flashy.solver][INFO] - Valid | Epoch 80 | 14/79 | 0.21 it/sec | ce 0.191 | ppl 1.212
[05-30 02:59:48][flashy.solver][INFO] - Valid | Epoch 80 | 21/79 | 0.21 it/sec | ce 0.229 | ppl 1.259
[05-30 03:00:20][flashy.solver][INFO] - Valid | Epoch 80 | 28/79 | 0.21 it/sec | ce 0.196 | ppl 1.219
[05-30 03:00:53][flashy.solver][INFO] - Valid | Epoch 80 | 35/79 | 0.21 it/sec | ce 0.225 | ppl 1.254
[05-30 03:01:26][flashy.solver][INFO] - Valid | Epoch 80 | 42/79 | 0.21 it/sec | ce 0.194 | ppl 1.215
[05-30 03:01:59][flashy.solver][INFO] - Valid | Epoch 80 | 49/79 | 0.21 it/sec | ce 0.262 | ppl 1.302
[05-30 03:02:32][flashy.solver][INFO] - Valid | Epoch 80 | 56/79 | 0.21 it/sec | ce 0.196 | ppl 1.218
[05-30 03:03:05][flashy.solver][INFO] - Valid | Epoch 80 | 63/79 | 0.21 it/sec | ce 0.248 | ppl 1.283
[05-30 03:03:37][flashy.solver][INFO] - Valid | Epoch 80 | 70/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-30 03:04:10][flashy.solver][INFO] - Valid | Epoch 80 | 77/79 | 0.21 it/sec | ce 0.262 | ppl 1.301
[05-30 03:04:11][flashy.solver][INFO] - Valid Summary | Epoch 80 | ce=0.221 | ppl=1.249 | duration=365.637
[05-30 03:04:11][flashy.solver][INFO] - New best state with ce=0.221 (was 0.221)
[05-30 03:04:11][flashy.solver][INFO] - Model hash: 6447919cc86f35802553790c2a08384b009ad5b5
[05-30 03:04:11][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 03:20:25][flashy.solver][INFO] - Train | Epoch 81 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.182E-01 | ce 0.239 | ppl 1.272
[05-30 03:36:28][flashy.solver][INFO] - Train | Epoch 81 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.771E-01 | ce 0.227 | ppl 1.258
[05-30 03:52:39][flashy.solver][INFO] - Train | Epoch 81 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.675E-01 | ce 0.234 | ppl 1.265
[05-30 04:08:45][flashy.solver][INFO] - Train | Epoch 81 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.049E+00 | ce 0.230 | ppl 1.261
[05-30 04:24:51][flashy.solver][INFO] - Train | Epoch 81 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.060E+00 | ce 0.230 | ppl 1.261
[05-30 04:40:57][flashy.solver][INFO] - Train | Epoch 81 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.889E-01 | ce 0.229 | ppl 1.259
[05-30 04:57:06][flashy.solver][INFO] - Train | Epoch 81 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.526E-01 | ce 0.235 | ppl 1.267
[05-30 05:13:16][flashy.solver][INFO] - Train | Epoch 81 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.366E-01 | ce 0.232 | ppl 1.263
[05-30 05:29:27][flashy.solver][INFO] - Train | Epoch 81 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.997E-01 | ce 0.235 | ppl 1.267
[05-30 05:45:31][flashy.solver][INFO] - Train Summary | Epoch 81 | lr=1.00E-04 | grad_norm=9.651E-01 | ce=0.232 | ppl=1.263 | duration=9679.771
[05-30 05:46:08][flashy.solver][INFO] - Valid | Epoch 81 | 7/79 | 0.21 it/sec | ce 0.206 | ppl 1.230
[05-30 05:46:41][flashy.solver][INFO] - Valid | Epoch 81 | 14/79 | 0.21 it/sec | ce 0.190 | ppl 1.212
[05-30 05:47:14][flashy.solver][INFO] - Valid | Epoch 81 | 21/79 | 0.21 it/sec | ce 0.228 | ppl 1.257
[05-30 05:47:46][flashy.solver][INFO] - Valid | Epoch 81 | 28/79 | 0.21 it/sec | ce 0.195 | ppl 1.219
[05-30 05:48:19][flashy.solver][INFO] - Valid | Epoch 81 | 35/79 | 0.21 it/sec | ce 0.225 | ppl 1.253
[05-30 05:48:52][flashy.solver][INFO] - Valid | Epoch 81 | 42/79 | 0.21 it/sec | ce 0.193 | ppl 1.215
[05-30 05:49:25][flashy.solver][INFO] - Valid | Epoch 81 | 49/79 | 0.21 it/sec | ce 0.262 | ppl 1.302
[05-30 05:49:58][flashy.solver][INFO] - Valid | Epoch 81 | 56/79 | 0.21 it/sec | ce 0.196 | ppl 1.218
[05-30 05:50:31][flashy.solver][INFO] - Valid | Epoch 81 | 63/79 | 0.21 it/sec | ce 0.248 | ppl 1.282
[05-30 05:51:04][flashy.solver][INFO] - Valid | Epoch 81 | 70/79 | 0.21 it/sec | ce 0.229 | ppl 1.259
[05-30 05:51:37][flashy.solver][INFO] - Valid | Epoch 81 | 77/79 | 0.21 it/sec | ce 0.262 | ppl 1.300
[05-30 05:51:38][flashy.solver][INFO] - Valid Summary | Epoch 81 | ce=0.220 | ppl=1.249 | duration=366.636
[05-30 05:51:38][flashy.solver][INFO] - New best state with ce=0.220 (was 0.221)
[05-30 05:51:38][flashy.solver][INFO] - Model hash: 402d2a0b25f13500f29d92e92c82be46272519d7
[05-30 05:51:38][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 06:07:50][flashy.solver][INFO] - Train | Epoch 82 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 8.736E-01 | ce 0.232 | ppl 1.263
[05-30 06:23:54][flashy.solver][INFO] - Train | Epoch 82 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.017E+00 | ce 0.232 | ppl 1.263
[05-30 06:40:03][flashy.solver][INFO] - Train | Epoch 82 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.532E-01 | ce 0.229 | ppl 1.259
[05-30 06:56:12][flashy.solver][INFO] - Train | Epoch 82 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.064E+00 | ce 0.232 | ppl 1.264
[05-30 07:12:22][flashy.solver][INFO] - Train | Epoch 82 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.631E-01 | ce 0.229 | ppl 1.260
[05-30 07:28:29][flashy.solver][INFO] - Train | Epoch 82 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.023E+00 | ce 0.231 | ppl 1.262
[05-30 07:44:38][flashy.solver][INFO] - Train | Epoch 82 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.278E-01 | ce 0.232 | ppl 1.263
[05-30 08:00:47][flashy.solver][INFO] - Train | Epoch 82 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.469E-01 | ce 0.228 | ppl 1.259
[05-30 08:16:56][flashy.solver][INFO] - Train | Epoch 82 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.012E+00 | ce 0.237 | ppl 1.270
[05-30 08:33:03][flashy.solver][INFO] - Train Summary | Epoch 82 | lr=1.00E-04 | grad_norm=9.775E-01 | ce=0.231 | ppl=1.262 | duration=9685.887
[05-30 08:33:41][flashy.solver][INFO] - Valid | Epoch 82 | 7/79 | 0.21 it/sec | ce 0.207 | ppl 1.231
[05-30 08:34:14][flashy.solver][INFO] - Valid | Epoch 82 | 14/79 | 0.21 it/sec | ce 0.191 | ppl 1.212
[05-30 08:34:46][flashy.solver][INFO] - Valid | Epoch 82 | 21/79 | 0.21 it/sec | ce 0.228 | ppl 1.258
[05-30 08:35:19][flashy.solver][INFO] - Valid | Epoch 82 | 28/79 | 0.21 it/sec | ce 0.196 | ppl 1.219
[05-30 08:35:51][flashy.solver][INFO] - Valid | Epoch 82 | 35/79 | 0.21 it/sec | ce 0.225 | ppl 1.253
[05-30 08:36:24][flashy.solver][INFO] - Valid | Epoch 82 | 42/79 | 0.21 it/sec | ce 0.194 | ppl 1.215
[05-30 08:36:57][flashy.solver][INFO] - Valid | Epoch 82 | 49/79 | 0.21 it/sec | ce 0.262 | ppl 1.302
[05-30 08:37:30][flashy.solver][INFO] - Valid | Epoch 82 | 56/79 | 0.21 it/sec | ce 0.197 | ppl 1.218
[05-30 08:38:03][flashy.solver][INFO] - Valid | Epoch 82 | 63/79 | 0.21 it/sec | ce 0.248 | ppl 1.283
[05-30 08:38:36][flashy.solver][INFO] - Valid | Epoch 82 | 70/79 | 0.21 it/sec | ce 0.230 | ppl 1.260
[05-30 08:39:09][flashy.solver][INFO] - Valid | Epoch 82 | 77/79 | 0.21 it/sec | ce 0.262 | ppl 1.301
[05-30 08:39:09][flashy.solver][INFO] - Valid Summary | Epoch 82 | ce=0.221 | ppl=1.249 | duration=365.850
[05-30 08:39:09][flashy.solver][INFO] - Model hash: d6996887512494f76c4b2321bbc94969c3535230
[05-30 08:39:09][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 08:55:18][flashy.solver][INFO] - Train | Epoch 83 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.856E-01 | ce 0.225 | ppl 1.255
[05-30 09:11:25][flashy.solver][INFO] - Train | Epoch 83 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.877E-01 | ce 0.223 | ppl 1.252
[05-30 09:27:34][flashy.solver][INFO] - Train | Epoch 83 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.573E-01 | ce 0.228 | ppl 1.259
[05-30 09:43:38][flashy.solver][INFO] - Train | Epoch 83 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.038E+00 | ce 0.234 | ppl 1.266
[05-30 09:59:46][flashy.solver][INFO] - Train | Epoch 83 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.067E+00 | ce 0.231 | ppl 1.262
[05-30 10:16:39][flashy.solver][INFO] - Train | Epoch 83 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.282E-01 | ce 0.230 | ppl 1.261
[05-30 10:33:36][flashy.solver][INFO] - Train | Epoch 83 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 1.124E+00 | ce 0.232 | ppl 1.264
[05-30 10:50:32][flashy.solver][INFO] - Train | Epoch 83 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 1.043E+00 | ce 0.222 | ppl 1.250
[05-30 11:07:23][flashy.solver][INFO] - Train | Epoch 83 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.645E-01 | ce 0.233 | ppl 1.264
[05-30 11:24:24][flashy.solver][INFO] - Train Summary | Epoch 83 | lr=1.00E-04 | grad_norm=1.019E+00 | ce=0.229 | ppl=1.260 | duration=9914.915
[05-30 11:25:04][flashy.solver][INFO] - Valid | Epoch 83 | 7/79 | 0.20 it/sec | ce 0.206 | ppl 1.230
[05-30 11:25:38][flashy.solver][INFO] - Valid | Epoch 83 | 14/79 | 0.20 it/sec | ce 0.190 | ppl 1.212
[05-30 11:26:13][flashy.solver][INFO] - Valid | Epoch 83 | 21/79 | 0.20 it/sec | ce 0.229 | ppl 1.258
[05-30 11:26:47][flashy.solver][INFO] - Valid | Epoch 83 | 28/79 | 0.20 it/sec | ce 0.195 | ppl 1.219
[05-30 11:27:22][flashy.solver][INFO] - Valid | Epoch 83 | 35/79 | 0.20 it/sec | ce 0.225 | ppl 1.253
[05-30 11:27:57][flashy.solver][INFO] - Valid | Epoch 83 | 42/79 | 0.20 it/sec | ce 0.193 | ppl 1.214
[05-30 11:28:32][flashy.solver][INFO] - Valid | Epoch 83 | 49/79 | 0.20 it/sec | ce 0.262 | ppl 1.302
[05-30 11:29:06][flashy.solver][INFO] - Valid | Epoch 83 | 56/79 | 0.20 it/sec | ce 0.196 | ppl 1.218
[05-30 11:29:41][flashy.solver][INFO] - Valid | Epoch 83 | 63/79 | 0.20 it/sec | ce 0.248 | ppl 1.282
[05-30 11:30:16][flashy.solver][INFO] - Valid | Epoch 83 | 70/79 | 0.20 it/sec | ce 0.230 | ppl 1.259
[05-30 11:30:51][flashy.solver][INFO] - Valid | Epoch 83 | 77/79 | 0.20 it/sec | ce 0.262 | ppl 1.300
[05-30 11:30:51][flashy.solver][INFO] - Valid Summary | Epoch 83 | ce=0.220 | ppl=1.249 | duration=387.122
[05-30 11:30:51][flashy.solver][INFO] - Model hash: 1945393f4fc6570c33798d3958894a82c45aa31d
[05-30 11:30:52][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 11:47:53][flashy.solver][INFO] - Train | Epoch 84 | 200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.451E-01 | ce 0.228 | ppl 1.258
[05-30 12:04:51][flashy.solver][INFO] - Train | Epoch 84 | 400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.612E-01 | ce 0.228 | ppl 1.259
[05-30 12:21:47][flashy.solver][INFO] - Train | Epoch 84 | 600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.812E-01 | ce 0.233 | ppl 1.265
[05-30 12:38:42][flashy.solver][INFO] - Train | Epoch 84 | 800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.648E-01 | ce 0.225 | ppl 1.254
[05-30 12:55:36][flashy.solver][INFO] - Train | Epoch 84 | 1000/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 8.916E-01 | ce 0.232 | ppl 1.263
[05-30 13:12:35][flashy.solver][INFO] - Train | Epoch 84 | 1200/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 1.026E+00 | ce 0.235 | ppl 1.268
[05-30 13:29:01][flashy.solver][INFO] - Train | Epoch 84 | 1400/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.648E-01 | ce 0.232 | ppl 1.263
[05-30 13:45:10][flashy.solver][INFO] - Train | Epoch 84 | 1600/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.886E-01 | ce 0.232 | ppl 1.264
[05-30 14:01:18][flashy.solver][INFO] - Train | Epoch 84 | 1800/2000 | 0.20 it/sec | lr 1.00E-04 | grad_norm 9.531E-01 | ce 0.235 | ppl 1.267
[05-30 14:17:25][flashy.solver][INFO] - Train Summary | Epoch 84 | lr=1.00E-04 | grad_norm=9.639E-01 | ce=0.231 | ppl=1.262 | duration=9993.127
[05-30 14:18:02][flashy.solver][INFO] - Valid | Epoch 84 | 7/79 | 0.21 it/sec | ce 0.206 | ppl 1.230
[05-30 14:18:35][flashy.solver][INFO] - Valid | Epoch 84 | 14/79 | 0.21 it/sec | ce 0.190 | ppl 1.212
[05-30 14:19:07][flashy.solver][INFO] - Valid | Epoch 84 | 21/79 | 0.21 it/sec | ce 0.228 | ppl 1.257
[05-30 14:19:40][flashy.solver][INFO] - Valid | Epoch 84 | 28/79 | 0.21 it/sec | ce 0.195 | ppl 1.218
[05-30 14:20:12][flashy.solver][INFO] - Valid | Epoch 84 | 35/79 | 0.21 it/sec | ce 0.224 | ppl 1.253
[05-30 14:20:45][flashy.solver][INFO] - Valid | Epoch 84 | 42/79 | 0.21 it/sec | ce 0.193 | ppl 1.214
[05-30 14:21:18][flashy.solver][INFO] - Valid | Epoch 84 | 49/79 | 0.21 it/sec | ce 0.261 | ppl 1.301
[05-30 14:21:51][flashy.solver][INFO] - Valid | Epoch 84 | 56/79 | 0.21 it/sec | ce 0.196 | ppl 1.217
[05-30 14:22:24][flashy.solver][INFO] - Valid | Epoch 84 | 63/79 | 0.21 it/sec | ce 0.247 | ppl 1.281
[05-30 14:22:57][flashy.solver][INFO] - Valid | Epoch 84 | 70/79 | 0.21 it/sec | ce 0.229 | ppl 1.258
[05-30 14:23:30][flashy.solver][INFO] - Valid | Epoch 84 | 77/79 | 0.21 it/sec | ce 0.261 | ppl 1.300
[05-30 14:23:30][flashy.solver][INFO] - Valid Summary | Epoch 84 | ce=0.220 | ppl=1.248 | duration=365.554
[05-30 14:23:30][flashy.solver][INFO] - New best state with ce=0.220 (was 0.220)
[05-30 14:23:30][flashy.solver][INFO] - Model hash: 17c8efec54abf64061fb5053444acd8e9b8e01bd
[05-30 14:23:30][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 14:39:43][flashy.solver][INFO] - Train | Epoch 85 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.066E-01 | ce 0.227 | ppl 1.257
[05-30 14:55:50][flashy.solver][INFO] - Train | Epoch 85 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.827E-01 | ce 0.230 | ppl 1.261
[05-30 15:11:55][flashy.solver][INFO] - Train | Epoch 85 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.022E+00 | ce 0.233 | ppl 1.264
[05-30 15:27:58][flashy.solver][INFO] - Train | Epoch 85 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.075E+00 | ce 0.240 | ppl 1.273
[05-30 15:44:01][flashy.solver][INFO] - Train | Epoch 85 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.027E+00 | ce 0.229 | ppl 1.260
[05-30 16:00:09][flashy.solver][INFO] - Train | Epoch 85 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.059E-01 | ce 0.233 | ppl 1.265
[05-30 16:16:14][flashy.solver][INFO] - Train | Epoch 85 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.053E+00 | ce 0.230 | ppl 1.260
[05-30 16:32:21][flashy.solver][INFO] - Train | Epoch 85 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.974E-01 | ce 0.236 | ppl 1.268
[05-30 16:48:27][flashy.solver][INFO] - Train | Epoch 85 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.070E+00 | ce 0.228 | ppl 1.257
[05-30 17:04:37][flashy.solver][INFO] - Train Summary | Epoch 85 | lr=1.00E-04 | grad_norm=1.003E+00 | ce=0.232 | ppl=1.264 | duration=9666.283
[05-30 17:05:14][flashy.solver][INFO] - Valid | Epoch 85 | 7/79 | 0.21 it/sec | ce 0.206 | ppl 1.230
[05-30 17:05:47][flashy.solver][INFO] - Valid | Epoch 85 | 14/79 | 0.21 it/sec | ce 0.190 | ppl 1.212
[05-30 17:06:19][flashy.solver][INFO] - Valid | Epoch 85 | 21/79 | 0.21 it/sec | ce 0.228 | ppl 1.257
[05-30 17:06:52][flashy.solver][INFO] - Valid | Epoch 85 | 28/79 | 0.21 it/sec | ce 0.195 | ppl 1.218
[05-30 17:07:24][flashy.solver][INFO] - Valid | Epoch 85 | 35/79 | 0.21 it/sec | ce 0.224 | ppl 1.253
[05-30 17:07:57][flashy.solver][INFO] - Valid | Epoch 85 | 42/79 | 0.21 it/sec | ce 0.193 | ppl 1.214
[05-30 17:08:30][flashy.solver][INFO] - Valid | Epoch 85 | 49/79 | 0.21 it/sec | ce 0.261 | ppl 1.301
[05-30 17:09:03][flashy.solver][INFO] - Valid | Epoch 85 | 56/79 | 0.21 it/sec | ce 0.196 | ppl 1.218
[05-30 17:09:36][flashy.solver][INFO] - Valid | Epoch 85 | 63/79 | 0.21 it/sec | ce 0.247 | ppl 1.282
[05-30 17:10:08][flashy.solver][INFO] - Valid | Epoch 85 | 70/79 | 0.21 it/sec | ce 0.229 | ppl 1.259
[05-30 17:10:41][flashy.solver][INFO] - Valid | Epoch 85 | 77/79 | 0.21 it/sec | ce 0.261 | ppl 1.299
[05-30 17:10:42][flashy.solver][INFO] - Valid Summary | Epoch 85 | ce=0.220 | ppl=1.248 | duration=365.405
[05-30 17:10:42][flashy.solver][INFO] - New best state with ce=0.220 (was 0.220)
[05-30 17:10:42][flashy.solver][INFO] - Model hash: 56425657e5fa50ac87abf7feb7bf90ba15429ca9
[05-30 17:10:42][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 17:26:54][flashy.solver][INFO] - Train | Epoch 86 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.416E-01 | ce 0.230 | ppl 1.260
[05-30 17:43:03][flashy.solver][INFO] - Train | Epoch 86 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.089E+00 | ce 0.234 | ppl 1.265
[05-30 17:59:13][flashy.solver][INFO] - Train | Epoch 86 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.265E-01 | ce 0.227 | ppl 1.257
[05-30 18:15:21][flashy.solver][INFO] - Train | Epoch 86 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.667E-01 | ce 0.230 | ppl 1.261
[05-30 18:31:28][flashy.solver][INFO] - Train | Epoch 86 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.044E+00 | ce 0.232 | ppl 1.262
[05-30 18:47:37][flashy.solver][INFO] - Train | Epoch 86 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.087E+00 | ce 0.227 | ppl 1.258
[05-30 19:03:47][flashy.solver][INFO] - Train | Epoch 86 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.682E-01 | ce 0.230 | ppl 1.260
[05-30 19:19:57][flashy.solver][INFO] - Train | Epoch 86 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.001E+00 | ce 0.232 | ppl 1.263
[05-30 19:36:01][flashy.solver][INFO] - Train | Epoch 86 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.101E+00 | ce 0.236 | ppl 1.269
[05-30 19:52:09][flashy.solver][INFO] - Train Summary | Epoch 86 | lr=1.00E-04 | grad_norm=1.012E+00 | ce=0.232 | ppl=1.263 | duration=9686.507
[05-30 19:52:46][flashy.solver][INFO] - Valid | Epoch 86 | 7/79 | 0.21 it/sec | ce 0.206 | ppl 1.230
[05-30 19:53:19][flashy.solver][INFO] - Valid | Epoch 86 | 14/79 | 0.21 it/sec | ce 0.190 | ppl 1.212
[05-30 19:53:52][flashy.solver][INFO] - Valid | Epoch 86 | 21/79 | 0.21 it/sec | ce 0.228 | ppl 1.257
[05-30 19:54:24][flashy.solver][INFO] - Valid | Epoch 86 | 28/79 | 0.21 it/sec | ce 0.195 | ppl 1.219
[05-30 19:54:57][flashy.solver][INFO] - Valid | Epoch 86 | 35/79 | 0.21 it/sec | ce 0.225 | ppl 1.253
[05-30 19:55:29][flashy.solver][INFO] - Valid | Epoch 86 | 42/79 | 0.21 it/sec | ce 0.193 | ppl 1.214
[05-30 19:56:02][flashy.solver][INFO] - Valid | Epoch 86 | 49/79 | 0.21 it/sec | ce 0.261 | ppl 1.302
[05-30 19:56:35][flashy.solver][INFO] - Valid | Epoch 86 | 56/79 | 0.21 it/sec | ce 0.196 | ppl 1.218
[05-30 19:57:08][flashy.solver][INFO] - Valid | Epoch 86 | 63/79 | 0.21 it/sec | ce 0.247 | ppl 1.281
[05-30 19:57:41][flashy.solver][INFO] - Valid | Epoch 86 | 70/79 | 0.21 it/sec | ce 0.229 | ppl 1.259
[05-30 19:58:14][flashy.solver][INFO] - Valid | Epoch 86 | 77/79 | 0.21 it/sec | ce 0.261 | ppl 1.299
[05-30 19:58:14][flashy.solver][INFO] - Valid Summary | Epoch 86 | ce=0.220 | ppl=1.248 | duration=365.868
[05-30 19:58:14][flashy.solver][INFO] - Model hash: 4f4c4f8639854c11b9e59f501454294aaef22b55
[05-30 19:58:15][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 20:14:27][flashy.solver][INFO] - Train | Epoch 87 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.315E-01 | ce 0.225 | ppl 1.254
[05-30 20:30:34][flashy.solver][INFO] - Train | Epoch 87 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.476E-01 | ce 0.228 | ppl 1.258
[05-30 20:46:42][flashy.solver][INFO] - Train | Epoch 87 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.498E-01 | ce 0.230 | ppl 1.260
[05-30 21:02:47][flashy.solver][INFO] - Train | Epoch 87 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.128E-01 | ce 0.231 | ppl 1.262
[05-30 21:18:51][flashy.solver][INFO] - Train | Epoch 87 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.281E-01 | ce 0.222 | ppl 1.250
[05-30 21:34:54][flashy.solver][INFO] - Train | Epoch 87 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.111E+00 | ce 0.236 | ppl 1.269
[05-30 21:50:52][flashy.solver][INFO] - Train | Epoch 87 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.712E-01 | ce 0.230 | ppl 1.261
[05-30 22:06:45][flashy.solver][INFO] - Train | Epoch 87 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.093E+00 | ce 0.226 | ppl 1.256
[05-30 22:22:42][flashy.solver][INFO] - Train | Epoch 87 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.395E-01 | ce 0.233 | ppl 1.265
[05-30 22:38:40][flashy.solver][INFO] - Train Summary | Epoch 87 | lr=1.00E-04 | grad_norm=9.838E-01 | ce=0.230 | ppl=1.261 | duration=9625.168
[05-30 22:39:17][flashy.solver][INFO] - Valid | Epoch 87 | 7/79 | 0.22 it/sec | ce 0.206 | ppl 1.230
[05-30 22:39:49][flashy.solver][INFO] - Valid | Epoch 87 | 14/79 | 0.22 it/sec | ce 0.190 | ppl 1.211
[05-30 22:40:21][flashy.solver][INFO] - Valid | Epoch 87 | 21/79 | 0.22 it/sec | ce 0.227 | ppl 1.257
[05-30 22:40:53][flashy.solver][INFO] - Valid | Epoch 87 | 28/79 | 0.22 it/sec | ce 0.195 | ppl 1.218
[05-30 22:41:25][flashy.solver][INFO] - Valid | Epoch 87 | 35/79 | 0.22 it/sec | ce 0.224 | ppl 1.252
[05-30 22:41:58][flashy.solver][INFO] - Valid | Epoch 87 | 42/79 | 0.22 it/sec | ce 0.192 | ppl 1.213
[05-30 22:42:31][flashy.solver][INFO] - Valid | Epoch 87 | 49/79 | 0.22 it/sec | ce 0.261 | ppl 1.301
[05-30 22:43:03][flashy.solver][INFO] - Valid | Epoch 87 | 56/79 | 0.22 it/sec | ce 0.196 | ppl 1.217
[05-30 22:43:36][flashy.solver][INFO] - Valid | Epoch 87 | 63/79 | 0.22 it/sec | ce 0.247 | ppl 1.281
[05-30 22:44:08][flashy.solver][INFO] - Valid | Epoch 87 | 70/79 | 0.22 it/sec | ce 0.229 | ppl 1.258
[05-30 22:44:41][flashy.solver][INFO] - Valid | Epoch 87 | 77/79 | 0.22 it/sec | ce 0.261 | ppl 1.299
[05-30 22:44:41][flashy.solver][INFO] - Valid Summary | Epoch 87 | ce=0.220 | ppl=1.248 | duration=361.445
[05-30 22:44:41][flashy.solver][INFO] - New best state with ce=0.220 (was 0.220)
[05-30 22:44:41][flashy.solver][INFO] - Model hash: 9fc53ab3acea2ca4b7d42b5791aa370820f9d6ee
[05-30 22:44:41][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-30 23:00:41][flashy.solver][INFO] - Train | Epoch 88 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.774E-01 | ce 0.225 | ppl 1.255
[05-30 23:16:38][flashy.solver][INFO] - Train | Epoch 88 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.013E+00 | ce 0.235 | ppl 1.267
[05-30 23:32:31][flashy.solver][INFO] - Train | Epoch 88 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.661E-01 | ce 0.225 | ppl 1.255
[05-30 23:48:25][flashy.solver][INFO] - Train | Epoch 88 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.099E+00 | ce 0.228 | ppl 1.258
[05-31 00:04:19][flashy.solver][INFO] - Train | Epoch 88 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.733E-01 | ce 0.227 | ppl 1.257
[05-31 00:20:17][flashy.solver][INFO] - Train | Epoch 88 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.123E+00 | ce 0.235 | ppl 1.267
[05-31 00:36:12][flashy.solver][INFO] - Train | Epoch 88 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.094E+00 | ce 0.231 | ppl 1.263
[05-31 00:52:08][flashy.solver][INFO] - Train | Epoch 88 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.649E-01 | ce 0.228 | ppl 1.258
[05-31 01:08:07][flashy.solver][INFO] - Train | Epoch 88 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.864E-01 | ce 0.225 | ppl 1.255
[05-31 01:24:00][flashy.solver][INFO] - Train Summary | Epoch 88 | lr=1.00E-04 | grad_norm=1.021E+00 | ce=0.229 | ppl=1.260 | duration=9559.164
[05-31 01:24:37][flashy.solver][INFO] - Valid | Epoch 88 | 7/79 | 0.22 it/sec | ce 0.205 | ppl 1.229
[05-31 01:25:09][flashy.solver][INFO] - Valid | Epoch 88 | 14/79 | 0.22 it/sec | ce 0.190 | ppl 1.211
[05-31 01:25:42][flashy.solver][INFO] - Valid | Epoch 88 | 21/79 | 0.22 it/sec | ce 0.227 | ppl 1.257
[05-31 01:26:14][flashy.solver][INFO] - Valid | Epoch 88 | 28/79 | 0.22 it/sec | ce 0.195 | ppl 1.218
[05-31 01:26:46][flashy.solver][INFO] - Valid | Epoch 88 | 35/79 | 0.22 it/sec | ce 0.224 | ppl 1.252
[05-31 01:27:19][flashy.solver][INFO] - Valid | Epoch 88 | 42/79 | 0.22 it/sec | ce 0.192 | ppl 1.213
[05-31 01:27:51][flashy.solver][INFO] - Valid | Epoch 88 | 49/79 | 0.22 it/sec | ce 0.260 | ppl 1.300
[05-31 01:28:24][flashy.solver][INFO] - Valid | Epoch 88 | 56/79 | 0.22 it/sec | ce 0.196 | ppl 1.217
[05-31 01:28:56][flashy.solver][INFO] - Valid | Epoch 88 | 63/79 | 0.22 it/sec | ce 0.247 | ppl 1.281
[05-31 01:29:28][flashy.solver][INFO] - Valid | Epoch 88 | 70/79 | 0.22 it/sec | ce 0.229 | ppl 1.258
[05-31 01:30:01][flashy.solver][INFO] - Valid | Epoch 88 | 77/79 | 0.22 it/sec | ce 0.261 | ppl 1.299
[05-31 01:30:02][flashy.solver][INFO] - Valid Summary | Epoch 88 | ce=0.220 | ppl=1.247 | duration=361.433
[05-31 01:30:02][flashy.solver][INFO] - New best state with ce=0.220 (was 0.220)
[05-31 01:30:02][flashy.solver][INFO] - Model hash: 7ecc210f88ace59de894d0eeb72ca56e71e1cecc
[05-31 01:30:02][audiocraft.utils.checkpoint][INFO] - Checkpoint saved to /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[05-31 01:46:04][flashy.solver][INFO] - Train | Epoch 89 | 200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.737E-01 | ce 0.226 | ppl 1.256
[05-31 02:02:09][flashy.solver][INFO] - Train | Epoch 89 | 400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.734E-01 | ce 0.231 | ppl 1.262
[05-31 02:18:19][flashy.solver][INFO] - Train | Epoch 89 | 600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.025E+00 | ce 0.235 | ppl 1.267
[05-31 02:34:24][flashy.solver][INFO] - Train | Epoch 89 | 800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.932E-01 | ce 0.224 | ppl 1.253
[05-31 02:50:31][flashy.solver][INFO] - Train | Epoch 89 | 1000/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 9.461E-01 | ce 0.219 | ppl 1.247
[05-31 03:06:35][flashy.solver][INFO] - Train | Epoch 89 | 1200/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.066E+00 | ce 0.224 | ppl 1.253
[05-31 03:22:42][flashy.solver][INFO] - Train | Epoch 89 | 1400/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.021E+00 | ce 0.231 | ppl 1.263
[05-31 03:38:49][flashy.solver][INFO] - Train | Epoch 89 | 1600/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.093E+00 | ce 0.231 | ppl 1.262
[05-31 03:54:58][flashy.solver][INFO] - Train | Epoch 89 | 1800/2000 | 0.21 it/sec | lr 1.00E-04 | grad_norm 1.078E+00 | ce 0.231 | ppl 1.262
[05-31 04:11:10][flashy.solver][INFO] - Train Summary | Epoch 89 | lr=1.00E-04 | grad_norm=1.019E+00 | ce=0.229 | ppl=1.259 | duration=9668.145
[05-31 04:11:48][flashy.solver][INFO] - Valid | Epoch 89 | 7/79 | 0.21 it/sec | ce 0.205 | ppl 1.229
[05-31 04:12:20][flashy.solver][INFO] - Valid | Epoch 89 | 14/79 | 0.21 it/sec | ce 0.190 | ppl 1.211
[05-31 04:12:54][flashy.solver][INFO] - Valid | Epoch 89 | 21/79 | 0.21 it/sec | ce 0.227 | ppl 1.256
[05-31 04:13:26][flashy.solver][INFO] - Valid | Epoch 89 | 28/79 | 0.21 it/sec | ce 0.195 | ppl 1.218
[05-31 04:13:59][flashy.solver][INFO] - Valid | Epoch 89 | 35/79 | 0.21 it/sec | ce 0.224 | ppl 1.252
[05-31 04:14:32][flashy.solver][INFO] - Valid | Epoch 89 | 42/79 | 0.21 it/sec | ce 0.192 | ppl 1.213
[05-31 04:15:05][flashy.solver][INFO] - Valid | Epoch 89 | 49/79 | 0.21 it/sec | ce 0.261 | ppl 1.301
[05-31 04:15:38][flashy.solver][INFO] - Valid | Epoch 89 | 56/79 | 0.21 it/sec | ce 0.195 | ppl 1.217
[05-31 04:16:11][flashy.solver][INFO] - Valid | Epoch 89 | 63/79 | 0.21 it/sec | ce 0.247 | ppl 1.281
[05-31 04:16:44][flashy.solver][INFO] - Valid | Epoch 89 | 70/79 | 0.21 it/sec | ce 0.229 | ppl 1.258
[05-31 04:17:17][flashy.solver][INFO] - Valid | Epoch 89 | 77/79 | 0.21 it/sec | ce 0.260 | ppl 1.299
[05-31 04:17:18][flashy.solver][INFO] - Valid Summary | Epoch 89 | ce=0.219 | ppl=1.247 | duration=368.031
[05-31 04:17:18][flashy.solver][INFO] - New best state with ce=0.219 (was 0.220)
[06-02 05:10:56][dora.distrib][INFO] - world_size is 1, skipping init.
[06-02 05:10:56][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[06-02 05:10:56][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[06-02 05:10:56][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjsung/project/audiocraft/egs/example
[06-02 05:10:56][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjsung/project/audiocraft/egs/example
[06-02 05:10:56][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjsung/project/audiocraft/egs/example
[06-02 05:10:56][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjsung/project/audiocraft/egs/example
[06-02 05:10:56][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[06-02 05:10:56][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[06-02 05:10:56][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[06-02 05:10:58][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[06-02 05:10:58][flashy.solver][INFO] - No EMA on the model.
[06-02 05:10:58][flashy.solver][INFO] - Model size: 1.21 M params
[06-02 05:10:58][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[06-02 05:10:58][flashy.solver][INFO] - Restoring weights and history.
[06-02 05:10:58][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[06-02 05:10:58][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[06-02 05:10:58][flashy.solver][INFO] - Model hash: 7ecc210f88ace59de894d0eeb72ca56e71e1cecc
[06-02 05:10:58][flashy.solver][INFO] - Replaying past metrics...
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 16 | lr=1.00E-04 | grad_norm=5.373E-01 | ce=0.274 | ppl=1.318 | duration=9552.291
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 16 | ce=0.265 | ppl=1.306 | duration=362.290
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 17 | lr=1.00E-04 | grad_norm=5.495E-01 | ce=0.274 | ppl=1.318 | duration=9593.252
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 17 | ce=0.263 | ppl=1.303 | duration=361.603
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 18 | lr=1.00E-04 | grad_norm=5.303E-01 | ce=0.270 | ppl=1.314 | duration=9601.733
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 18 | ce=0.261 | ppl=1.301 | duration=362.833
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 19 | lr=1.00E-04 | grad_norm=5.436E-01 | ce=0.271 | ppl=1.314 | duration=9603.793
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 19 | ce=0.259 | ppl=1.298 | duration=362.807
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 20 | lr=1.00E-04 | grad_norm=5.256E-01 | ce=0.267 | ppl=1.309 | duration=9513.591
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 20 | ce=0.256 | ppl=1.295 | duration=359.164
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 21 | lr=1.00E-04 | grad_norm=5.398E-01 | ce=0.267 | ppl=1.309 | duration=9521.433
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 21 | ce=0.255 | ppl=1.293 | duration=359.736
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 22 | lr=1.00E-04 | grad_norm=5.273E-01 | ce=0.264 | ppl=1.305 | duration=9522.932
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 22 | ce=0.254 | ppl=1.291 | duration=359.952
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 23 | lr=1.00E-04 | grad_norm=5.359E-01 | ce=0.263 | ppl=1.304 | duration=9520.848
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 23 | ce=0.252 | ppl=1.290 | duration=359.946
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 24 | lr=1.00E-04 | grad_norm=5.356E-01 | ce=0.260 | ppl=1.300 | duration=9517.164
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 24 | ce=0.250 | ppl=1.287 | duration=360.223
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 25 | lr=1.00E-04 | grad_norm=5.350E-01 | ce=0.258 | ppl=1.297 | duration=9526.316
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 25 | ce=0.249 | ppl=1.286 | duration=359.858
[06-02 05:10:58][flashy.solver][INFO] - Evaluate Summary | Epoch 25 | duration=0.000
[06-02 05:10:58][flashy.solver][INFO] - Generate Summary | Epoch 25 | rtf=0.143 | duration=7.398
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 26 | lr=1.00E-04 | grad_norm=5.344E-01 | ce=0.255 | ppl=1.294 | duration=9505.304
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 26 | ce=0.248 | ppl=1.284 | duration=359.841
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 27 | lr=1.00E-04 | grad_norm=5.476E-01 | ce=0.256 | ppl=1.294 | duration=9515.969
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 27 | ce=0.247 | ppl=1.282 | duration=359.513
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 28 | lr=1.00E-04 | grad_norm=5.444E-01 | ce=0.254 | ppl=1.292 | duration=9521.965
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 28 | ce=0.245 | ppl=1.281 | duration=360.116
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 29 | lr=1.00E-04 | grad_norm=5.705E-01 | ce=0.255 | ppl=1.293 | duration=9536.583
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 29 | ce=0.245 | ppl=1.280 | duration=361.077
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 30 | lr=1.00E-04 | grad_norm=5.520E-01 | ce=0.253 | ppl=1.291 | duration=9517.157
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 30 | ce=0.243 | ppl=1.278 | duration=360.175
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 31 | lr=1.00E-04 | grad_norm=5.536E-01 | ce=0.252 | ppl=1.290 | duration=9515.999
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 31 | ce=0.242 | ppl=1.277 | duration=359.757
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 32 | lr=1.00E-04 | grad_norm=5.734E-01 | ce=0.251 | ppl=1.288 | duration=9525.455
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 32 | ce=0.241 | ppl=1.275 | duration=359.850
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 33 | lr=1.00E-04 | grad_norm=5.841E-01 | ce=0.252 | ppl=1.289 | duration=9522.251
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 33 | ce=0.240 | ppl=1.274 | duration=360.855
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 34 | lr=1.00E-04 | grad_norm=5.932E-01 | ce=0.250 | ppl=1.287 | duration=9530.122
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 34 | ce=0.240 | ppl=1.273 | duration=360.152
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 35 | lr=1.00E-04 | grad_norm=5.957E-01 | ce=0.248 | ppl=1.284 | duration=9524.631
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 35 | ce=0.239 | ppl=1.272 | duration=359.587
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 36 | lr=1.00E-04 | grad_norm=6.259E-01 | ce=0.251 | ppl=1.288 | duration=9531.088
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 36 | ce=0.239 | ppl=1.272 | duration=360.096
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 37 | lr=1.00E-04 | grad_norm=6.270E-01 | ce=0.248 | ppl=1.284 | duration=9533.307
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 37 | ce=0.237 | ppl=1.270 | duration=360.546
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 38 | lr=1.00E-04 | grad_norm=6.272E-01 | ce=0.248 | ppl=1.284 | duration=9529.414
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 38 | ce=0.237 | ppl=1.270 | duration=360.767
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 39 | lr=1.00E-04 | grad_norm=6.386E-01 | ce=0.246 | ppl=1.281 | duration=9524.562
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 39 | ce=0.236 | ppl=1.268 | duration=360.385
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 40 | lr=1.00E-04 | grad_norm=6.360E-01 | ce=0.244 | ppl=1.279 | duration=9642.435
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 40 | ce=0.235 | ppl=1.267 | duration=385.528
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 41 | lr=1.00E-04 | grad_norm=6.410E-01 | ce=0.246 | ppl=1.281 | duration=9668.847
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 41 | ce=0.235 | ppl=1.267 | duration=363.285
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 42 | lr=1.00E-04 | grad_norm=6.642E-01 | ce=0.244 | ppl=1.278 | duration=9572.143
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 42 | ce=0.234 | ppl=1.266 | duration=362.071
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 43 | lr=1.00E-04 | grad_norm=6.578E-01 | ce=0.244 | ppl=1.278 | duration=9564.111
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 43 | ce=0.234 | ppl=1.265 | duration=362.399
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 44 | lr=1.00E-04 | grad_norm=6.778E-01 | ce=0.244 | ppl=1.279 | duration=9545.633
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 44 | ce=0.233 | ppl=1.264 | duration=360.226
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 45 | lr=1.00E-04 | grad_norm=6.707E-01 | ce=0.240 | ppl=1.273 | duration=9558.510
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 45 | ce=0.232 | ppl=1.263 | duration=361.292
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 46 | lr=1.00E-04 | grad_norm=7.081E-01 | ce=0.243 | ppl=1.277 | duration=9549.454
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 46 | ce=0.232 | ppl=1.263 | duration=361.476
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 47 | lr=1.00E-04 | grad_norm=6.983E-01 | ce=0.241 | ppl=1.275 | duration=9994.908
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 47 | ce=0.231 | ppl=1.262 | duration=380.415
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 48 | lr=1.00E-04 | grad_norm=7.072E-01 | ce=0.241 | ppl=1.274 | duration=9756.736
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 48 | ce=0.231 | ppl=1.262 | duration=368.529
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 49 | lr=1.00E-04 | grad_norm=7.122E-01 | ce=0.239 | ppl=1.273 | duration=9719.254
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 49 | ce=0.230 | ppl=1.261 | duration=360.200
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 50 | lr=1.00E-04 | grad_norm=7.181E-01 | ce=0.237 | ppl=1.270 | duration=9752.171
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 50 | ce=0.230 | ppl=1.261 | duration=359.721
[06-02 05:10:58][flashy.solver][INFO] - Evaluate Summary | Epoch 50 | duration=0.000
[06-02 05:10:58][flashy.solver][INFO] - Generate Summary | Epoch 50 | rtf=0.134 | duration=6.334
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 51 | lr=1.00E-04 | grad_norm=7.459E-01 | ce=0.243 | ppl=1.277 | duration=9700.451
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 51 | ce=0.230 | ppl=1.260 | duration=360.665
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 52 | lr=1.00E-04 | grad_norm=7.595E-01 | ce=0.241 | ppl=1.274 | duration=9529.466
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 52 | ce=0.229 | ppl=1.260 | duration=360.312
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 53 | lr=1.00E-04 | grad_norm=7.351E-01 | ce=0.239 | ppl=1.273 | duration=9539.188
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 53 | ce=0.229 | ppl=1.259 | duration=359.881
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 54 | lr=1.00E-04 | grad_norm=7.549E-01 | ce=0.239 | ppl=1.272 | duration=9542.972
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 54 | ce=0.228 | ppl=1.258 | duration=368.039
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 55 | lr=1.00E-04 | grad_norm=7.593E-01 | ce=0.237 | ppl=1.269 | duration=9580.192
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 55 | ce=0.228 | ppl=1.258 | duration=380.776
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 56 | lr=1.00E-04 | grad_norm=7.741E-01 | ce=0.237 | ppl=1.270 | duration=10097.090
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 56 | ce=0.228 | ppl=1.258 | duration=380.956
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 57 | lr=1.00E-04 | grad_norm=7.629E-01 | ce=0.238 | ppl=1.271 | duration=9941.644
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 57 | ce=0.227 | ppl=1.257 | duration=369.864
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 58 | lr=1.00E-04 | grad_norm=7.822E-01 | ce=0.237 | ppl=1.270 | duration=9753.884
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 58 | ce=0.227 | ppl=1.257 | duration=368.705
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 59 | lr=1.00E-04 | grad_norm=7.903E-01 | ce=0.237 | ppl=1.269 | duration=9762.634
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 59 | ce=0.226 | ppl=1.256 | duration=370.989
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 60 | lr=1.00E-04 | grad_norm=8.007E-01 | ce=0.237 | ppl=1.269 | duration=9874.322
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 60 | ce=0.227 | ppl=1.256 | duration=375.685
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 61 | lr=1.00E-04 | grad_norm=8.122E-01 | ce=0.238 | ppl=1.271 | duration=9884.318
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 61 | ce=0.226 | ppl=1.255 | duration=372.385
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 62 | lr=1.00E-04 | grad_norm=7.932E-01 | ce=0.234 | ppl=1.266 | duration=9840.462
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 62 | ce=0.225 | ppl=1.255 | duration=371.850
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 63 | lr=1.00E-04 | grad_norm=8.240E-01 | ce=0.233 | ppl=1.265 | duration=9806.020
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 63 | ce=0.226 | ppl=1.255 | duration=369.038
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 64 | lr=1.00E-04 | grad_norm=8.545E-01 | ce=0.237 | ppl=1.269 | duration=9878.421
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 64 | ce=0.225 | ppl=1.254 | duration=370.405
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 65 | lr=1.00E-04 | grad_norm=8.380E-01 | ce=0.235 | ppl=1.267 | duration=9768.115
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 65 | ce=0.225 | ppl=1.254 | duration=368.232
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 66 | lr=1.00E-04 | grad_norm=8.622E-01 | ce=0.234 | ppl=1.266 | duration=9705.978
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 66 | ce=0.225 | ppl=1.254 | duration=368.388
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 67 | lr=1.00E-04 | grad_norm=8.433E-01 | ce=0.235 | ppl=1.267 | duration=9708.521
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 67 | ce=0.224 | ppl=1.253 | duration=366.592
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 68 | lr=1.00E-04 | grad_norm=8.658E-01 | ce=0.233 | ppl=1.264 | duration=9681.177
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 68 | ce=0.224 | ppl=1.253 | duration=366.955
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 69 | lr=1.00E-04 | grad_norm=8.897E-01 | ce=0.234 | ppl=1.266 | duration=9676.317
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 69 | ce=0.224 | ppl=1.253 | duration=365.945
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 70 | lr=1.00E-04 | grad_norm=8.614E-01 | ce=0.231 | ppl=1.262 | duration=9693.598
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 70 | ce=0.223 | ppl=1.252 | duration=366.943
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 71 | lr=1.00E-04 | grad_norm=9.080E-01 | ce=0.235 | ppl=1.268 | duration=9683.285
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 71 | ce=0.223 | ppl=1.252 | duration=368.008
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 72 | lr=1.00E-04 | grad_norm=8.908E-01 | ce=0.233 | ppl=1.265 | duration=9695.132
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 72 | ce=0.222 | ppl=1.251 | duration=366.920
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 73 | lr=1.00E-04 | grad_norm=9.097E-01 | ce=0.235 | ppl=1.267 | duration=9707.760
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 73 | ce=0.223 | ppl=1.251 | duration=368.145
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 74 | lr=1.00E-04 | grad_norm=9.211E-01 | ce=0.234 | ppl=1.266 | duration=9650.362
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 74 | ce=0.222 | ppl=1.251 | duration=361.467
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 75 | lr=1.00E-04 | grad_norm=9.272E-01 | ce=0.233 | ppl=1.265 | duration=9633.944
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 75 | ce=0.222 | ppl=1.251 | duration=367.500
[06-02 05:10:58][flashy.solver][INFO] - Evaluate Summary | Epoch 75 | duration=0.000
[06-02 05:10:58][flashy.solver][INFO] - Generate Summary | Epoch 75 | rtf=0.144 | duration=6.500
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 76 | lr=1.00E-04 | grad_norm=9.470E-01 | ce=0.232 | ppl=1.263 | duration=9666.764
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 76 | ce=0.222 | ppl=1.250 | duration=365.508
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 77 | lr=1.00E-04 | grad_norm=9.267E-01 | ce=0.230 | ppl=1.261 | duration=9671.955
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 77 | ce=0.222 | ppl=1.250 | duration=365.621
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 78 | lr=1.00E-04 | grad_norm=9.488E-01 | ce=0.232 | ppl=1.263 | duration=9607.441
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 78 | ce=0.222 | ppl=1.250 | duration=361.455
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 79 | lr=1.00E-04 | grad_norm=9.638E-01 | ce=0.230 | ppl=1.261 | duration=9569.696
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 79 | ce=0.221 | ppl=1.250 | duration=361.342
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 80 | lr=1.00E-04 | grad_norm=9.486E-01 | ce=0.229 | ppl=1.260 | duration=9602.914
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 80 | ce=0.221 | ppl=1.249 | duration=365.637
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 81 | lr=1.00E-04 | grad_norm=9.651E-01 | ce=0.232 | ppl=1.263 | duration=9679.771
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 81 | ce=0.220 | ppl=1.249 | duration=366.636
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 82 | lr=1.00E-04 | grad_norm=9.775E-01 | ce=0.231 | ppl=1.262 | duration=9685.887
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 82 | ce=0.221 | ppl=1.249 | duration=365.850
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 83 | lr=1.00E-04 | grad_norm=1.019E+00 | ce=0.229 | ppl=1.260 | duration=9914.915
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 83 | ce=0.220 | ppl=1.249 | duration=387.122
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 84 | lr=1.00E-04 | grad_norm=9.639E-01 | ce=0.231 | ppl=1.262 | duration=9993.127
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 84 | ce=0.220 | ppl=1.248 | duration=365.554
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 85 | lr=1.00E-04 | grad_norm=1.003E+00 | ce=0.232 | ppl=1.264 | duration=9666.283
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 85 | ce=0.220 | ppl=1.248 | duration=365.405
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 86 | lr=1.00E-04 | grad_norm=1.012E+00 | ce=0.232 | ppl=1.263 | duration=9686.507
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 86 | ce=0.220 | ppl=1.248 | duration=365.868
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 87 | lr=1.00E-04 | grad_norm=9.838E-01 | ce=0.230 | ppl=1.261 | duration=9625.168
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 87 | ce=0.220 | ppl=1.248 | duration=361.445
[06-02 05:10:58][flashy.solver][INFO] - Train Summary | Epoch 88 | lr=1.00E-04 | grad_norm=1.021E+00 | ce=0.229 | ppl=1.260 | duration=9559.164
[06-02 05:10:58][flashy.solver][INFO] - Valid Summary | Epoch 88 | ce=0.220 | ppl=1.247 | duration=361.433
[06-02 05:11:03][audiocraft.modules.codebooks_patterns][INFO] - New pattern, time steps: 250, sequence steps: 251
[06-02 07:17:36][dora.distrib][INFO] - world_size is 1, skipping init.
[06-02 07:17:36][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[06-02 07:17:36][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[06-02 07:17:36][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjs/project/audiocraft/egs/example
[06-02 07:17:36][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjs/project/audiocraft/egs/example
[06-02 07:17:36][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjs/project/audiocraft/egs/example
[06-02 07:17:36][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjs/project/audiocraft/egs/example
[06-02 07:17:36][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[06-02 07:17:36][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[06-02 07:17:36][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[06-02 07:17:37][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[06-02 07:17:37][flashy.solver][INFO] - No EMA on the model.
[06-02 07:17:37][flashy.solver][INFO] - Model size: 1.21 M params
[06-02 07:17:37][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[06-02 07:17:37][flashy.solver][INFO] - Restoring weights and history.
[06-02 07:17:37][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[06-02 07:17:37][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[06-02 07:17:37][flashy.solver][INFO] - Model hash: 7ecc210f88ace59de894d0eeb72ca56e71e1cecc
[06-02 07:17:37][flashy.solver][INFO] - Replaying past metrics...
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 16 | lr=1.00E-04 | grad_norm=5.373E-01 | ce=0.274 | ppl=1.318 | duration=9552.291
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 16 | ce=0.265 | ppl=1.306 | duration=362.290
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 17 | lr=1.00E-04 | grad_norm=5.495E-01 | ce=0.274 | ppl=1.318 | duration=9593.252
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 17 | ce=0.263 | ppl=1.303 | duration=361.603
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 18 | lr=1.00E-04 | grad_norm=5.303E-01 | ce=0.270 | ppl=1.314 | duration=9601.733
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 18 | ce=0.261 | ppl=1.301 | duration=362.833
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 19 | lr=1.00E-04 | grad_norm=5.436E-01 | ce=0.271 | ppl=1.314 | duration=9603.793
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 19 | ce=0.259 | ppl=1.298 | duration=362.807
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 20 | lr=1.00E-04 | grad_norm=5.256E-01 | ce=0.267 | ppl=1.309 | duration=9513.591
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 20 | ce=0.256 | ppl=1.295 | duration=359.164
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 21 | lr=1.00E-04 | grad_norm=5.398E-01 | ce=0.267 | ppl=1.309 | duration=9521.433
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 21 | ce=0.255 | ppl=1.293 | duration=359.736
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 22 | lr=1.00E-04 | grad_norm=5.273E-01 | ce=0.264 | ppl=1.305 | duration=9522.932
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 22 | ce=0.254 | ppl=1.291 | duration=359.952
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 23 | lr=1.00E-04 | grad_norm=5.359E-01 | ce=0.263 | ppl=1.304 | duration=9520.848
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 23 | ce=0.252 | ppl=1.290 | duration=359.946
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 24 | lr=1.00E-04 | grad_norm=5.356E-01 | ce=0.260 | ppl=1.300 | duration=9517.164
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 24 | ce=0.250 | ppl=1.287 | duration=360.223
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 25 | lr=1.00E-04 | grad_norm=5.350E-01 | ce=0.258 | ppl=1.297 | duration=9526.316
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 25 | ce=0.249 | ppl=1.286 | duration=359.858
[06-02 07:17:37][flashy.solver][INFO] - Evaluate Summary | Epoch 25 | duration=0.000
[06-02 07:17:37][flashy.solver][INFO] - Generate Summary | Epoch 25 | rtf=0.143 | duration=7.398
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 26 | lr=1.00E-04 | grad_norm=5.344E-01 | ce=0.255 | ppl=1.294 | duration=9505.304
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 26 | ce=0.248 | ppl=1.284 | duration=359.841
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 27 | lr=1.00E-04 | grad_norm=5.476E-01 | ce=0.256 | ppl=1.294 | duration=9515.969
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 27 | ce=0.247 | ppl=1.282 | duration=359.513
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 28 | lr=1.00E-04 | grad_norm=5.444E-01 | ce=0.254 | ppl=1.292 | duration=9521.965
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 28 | ce=0.245 | ppl=1.281 | duration=360.116
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 29 | lr=1.00E-04 | grad_norm=5.705E-01 | ce=0.255 | ppl=1.293 | duration=9536.583
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 29 | ce=0.245 | ppl=1.280 | duration=361.077
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 30 | lr=1.00E-04 | grad_norm=5.520E-01 | ce=0.253 | ppl=1.291 | duration=9517.157
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 30 | ce=0.243 | ppl=1.278 | duration=360.175
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 31 | lr=1.00E-04 | grad_norm=5.536E-01 | ce=0.252 | ppl=1.290 | duration=9515.999
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 31 | ce=0.242 | ppl=1.277 | duration=359.757
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 32 | lr=1.00E-04 | grad_norm=5.734E-01 | ce=0.251 | ppl=1.288 | duration=9525.455
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 32 | ce=0.241 | ppl=1.275 | duration=359.850
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 33 | lr=1.00E-04 | grad_norm=5.841E-01 | ce=0.252 | ppl=1.289 | duration=9522.251
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 33 | ce=0.240 | ppl=1.274 | duration=360.855
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 34 | lr=1.00E-04 | grad_norm=5.932E-01 | ce=0.250 | ppl=1.287 | duration=9530.122
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 34 | ce=0.240 | ppl=1.273 | duration=360.152
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 35 | lr=1.00E-04 | grad_norm=5.957E-01 | ce=0.248 | ppl=1.284 | duration=9524.631
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 35 | ce=0.239 | ppl=1.272 | duration=359.587
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 36 | lr=1.00E-04 | grad_norm=6.259E-01 | ce=0.251 | ppl=1.288 | duration=9531.088
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 36 | ce=0.239 | ppl=1.272 | duration=360.096
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 37 | lr=1.00E-04 | grad_norm=6.270E-01 | ce=0.248 | ppl=1.284 | duration=9533.307
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 37 | ce=0.237 | ppl=1.270 | duration=360.546
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 38 | lr=1.00E-04 | grad_norm=6.272E-01 | ce=0.248 | ppl=1.284 | duration=9529.414
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 38 | ce=0.237 | ppl=1.270 | duration=360.767
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 39 | lr=1.00E-04 | grad_norm=6.386E-01 | ce=0.246 | ppl=1.281 | duration=9524.562
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 39 | ce=0.236 | ppl=1.268 | duration=360.385
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 40 | lr=1.00E-04 | grad_norm=6.360E-01 | ce=0.244 | ppl=1.279 | duration=9642.435
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 40 | ce=0.235 | ppl=1.267 | duration=385.528
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 41 | lr=1.00E-04 | grad_norm=6.410E-01 | ce=0.246 | ppl=1.281 | duration=9668.847
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 41 | ce=0.235 | ppl=1.267 | duration=363.285
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 42 | lr=1.00E-04 | grad_norm=6.642E-01 | ce=0.244 | ppl=1.278 | duration=9572.143
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 42 | ce=0.234 | ppl=1.266 | duration=362.071
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 43 | lr=1.00E-04 | grad_norm=6.578E-01 | ce=0.244 | ppl=1.278 | duration=9564.111
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 43 | ce=0.234 | ppl=1.265 | duration=362.399
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 44 | lr=1.00E-04 | grad_norm=6.778E-01 | ce=0.244 | ppl=1.279 | duration=9545.633
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 44 | ce=0.233 | ppl=1.264 | duration=360.226
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 45 | lr=1.00E-04 | grad_norm=6.707E-01 | ce=0.240 | ppl=1.273 | duration=9558.510
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 45 | ce=0.232 | ppl=1.263 | duration=361.292
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 46 | lr=1.00E-04 | grad_norm=7.081E-01 | ce=0.243 | ppl=1.277 | duration=9549.454
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 46 | ce=0.232 | ppl=1.263 | duration=361.476
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 47 | lr=1.00E-04 | grad_norm=6.983E-01 | ce=0.241 | ppl=1.275 | duration=9994.908
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 47 | ce=0.231 | ppl=1.262 | duration=380.415
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 48 | lr=1.00E-04 | grad_norm=7.072E-01 | ce=0.241 | ppl=1.274 | duration=9756.736
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 48 | ce=0.231 | ppl=1.262 | duration=368.529
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 49 | lr=1.00E-04 | grad_norm=7.122E-01 | ce=0.239 | ppl=1.273 | duration=9719.254
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 49 | ce=0.230 | ppl=1.261 | duration=360.200
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 50 | lr=1.00E-04 | grad_norm=7.181E-01 | ce=0.237 | ppl=1.270 | duration=9752.171
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 50 | ce=0.230 | ppl=1.261 | duration=359.721
[06-02 07:17:37][flashy.solver][INFO] - Evaluate Summary | Epoch 50 | duration=0.000
[06-02 07:17:37][flashy.solver][INFO] - Generate Summary | Epoch 50 | rtf=0.134 | duration=6.334
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 51 | lr=1.00E-04 | grad_norm=7.459E-01 | ce=0.243 | ppl=1.277 | duration=9700.451
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 51 | ce=0.230 | ppl=1.260 | duration=360.665
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 52 | lr=1.00E-04 | grad_norm=7.595E-01 | ce=0.241 | ppl=1.274 | duration=9529.466
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 52 | ce=0.229 | ppl=1.260 | duration=360.312
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 53 | lr=1.00E-04 | grad_norm=7.351E-01 | ce=0.239 | ppl=1.273 | duration=9539.188
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 53 | ce=0.229 | ppl=1.259 | duration=359.881
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 54 | lr=1.00E-04 | grad_norm=7.549E-01 | ce=0.239 | ppl=1.272 | duration=9542.972
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 54 | ce=0.228 | ppl=1.258 | duration=368.039
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 55 | lr=1.00E-04 | grad_norm=7.593E-01 | ce=0.237 | ppl=1.269 | duration=9580.192
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 55 | ce=0.228 | ppl=1.258 | duration=380.776
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 56 | lr=1.00E-04 | grad_norm=7.741E-01 | ce=0.237 | ppl=1.270 | duration=10097.090
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 56 | ce=0.228 | ppl=1.258 | duration=380.956
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 57 | lr=1.00E-04 | grad_norm=7.629E-01 | ce=0.238 | ppl=1.271 | duration=9941.644
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 57 | ce=0.227 | ppl=1.257 | duration=369.864
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 58 | lr=1.00E-04 | grad_norm=7.822E-01 | ce=0.237 | ppl=1.270 | duration=9753.884
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 58 | ce=0.227 | ppl=1.257 | duration=368.705
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 59 | lr=1.00E-04 | grad_norm=7.903E-01 | ce=0.237 | ppl=1.269 | duration=9762.634
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 59 | ce=0.226 | ppl=1.256 | duration=370.989
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 60 | lr=1.00E-04 | grad_norm=8.007E-01 | ce=0.237 | ppl=1.269 | duration=9874.322
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 60 | ce=0.227 | ppl=1.256 | duration=375.685
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 61 | lr=1.00E-04 | grad_norm=8.122E-01 | ce=0.238 | ppl=1.271 | duration=9884.318
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 61 | ce=0.226 | ppl=1.255 | duration=372.385
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 62 | lr=1.00E-04 | grad_norm=7.932E-01 | ce=0.234 | ppl=1.266 | duration=9840.462
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 62 | ce=0.225 | ppl=1.255 | duration=371.850
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 63 | lr=1.00E-04 | grad_norm=8.240E-01 | ce=0.233 | ppl=1.265 | duration=9806.020
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 63 | ce=0.226 | ppl=1.255 | duration=369.038
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 64 | lr=1.00E-04 | grad_norm=8.545E-01 | ce=0.237 | ppl=1.269 | duration=9878.421
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 64 | ce=0.225 | ppl=1.254 | duration=370.405
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 65 | lr=1.00E-04 | grad_norm=8.380E-01 | ce=0.235 | ppl=1.267 | duration=9768.115
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 65 | ce=0.225 | ppl=1.254 | duration=368.232
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 66 | lr=1.00E-04 | grad_norm=8.622E-01 | ce=0.234 | ppl=1.266 | duration=9705.978
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 66 | ce=0.225 | ppl=1.254 | duration=368.388
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 67 | lr=1.00E-04 | grad_norm=8.433E-01 | ce=0.235 | ppl=1.267 | duration=9708.521
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 67 | ce=0.224 | ppl=1.253 | duration=366.592
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 68 | lr=1.00E-04 | grad_norm=8.658E-01 | ce=0.233 | ppl=1.264 | duration=9681.177
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 68 | ce=0.224 | ppl=1.253 | duration=366.955
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 69 | lr=1.00E-04 | grad_norm=8.897E-01 | ce=0.234 | ppl=1.266 | duration=9676.317
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 69 | ce=0.224 | ppl=1.253 | duration=365.945
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 70 | lr=1.00E-04 | grad_norm=8.614E-01 | ce=0.231 | ppl=1.262 | duration=9693.598
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 70 | ce=0.223 | ppl=1.252 | duration=366.943
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 71 | lr=1.00E-04 | grad_norm=9.080E-01 | ce=0.235 | ppl=1.268 | duration=9683.285
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 71 | ce=0.223 | ppl=1.252 | duration=368.008
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 72 | lr=1.00E-04 | grad_norm=8.908E-01 | ce=0.233 | ppl=1.265 | duration=9695.132
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 72 | ce=0.222 | ppl=1.251 | duration=366.920
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 73 | lr=1.00E-04 | grad_norm=9.097E-01 | ce=0.235 | ppl=1.267 | duration=9707.760
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 73 | ce=0.223 | ppl=1.251 | duration=368.145
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 74 | lr=1.00E-04 | grad_norm=9.211E-01 | ce=0.234 | ppl=1.266 | duration=9650.362
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 74 | ce=0.222 | ppl=1.251 | duration=361.467
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 75 | lr=1.00E-04 | grad_norm=9.272E-01 | ce=0.233 | ppl=1.265 | duration=9633.944
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 75 | ce=0.222 | ppl=1.251 | duration=367.500
[06-02 07:17:37][flashy.solver][INFO] - Evaluate Summary | Epoch 75 | duration=0.000
[06-02 07:17:37][flashy.solver][INFO] - Generate Summary | Epoch 75 | rtf=0.144 | duration=6.500
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 76 | lr=1.00E-04 | grad_norm=9.470E-01 | ce=0.232 | ppl=1.263 | duration=9666.764
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 76 | ce=0.222 | ppl=1.250 | duration=365.508
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 77 | lr=1.00E-04 | grad_norm=9.267E-01 | ce=0.230 | ppl=1.261 | duration=9671.955
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 77 | ce=0.222 | ppl=1.250 | duration=365.621
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 78 | lr=1.00E-04 | grad_norm=9.488E-01 | ce=0.232 | ppl=1.263 | duration=9607.441
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 78 | ce=0.222 | ppl=1.250 | duration=361.455
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 79 | lr=1.00E-04 | grad_norm=9.638E-01 | ce=0.230 | ppl=1.261 | duration=9569.696
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 79 | ce=0.221 | ppl=1.250 | duration=361.342
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 80 | lr=1.00E-04 | grad_norm=9.486E-01 | ce=0.229 | ppl=1.260 | duration=9602.914
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 80 | ce=0.221 | ppl=1.249 | duration=365.637
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 81 | lr=1.00E-04 | grad_norm=9.651E-01 | ce=0.232 | ppl=1.263 | duration=9679.771
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 81 | ce=0.220 | ppl=1.249 | duration=366.636
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 82 | lr=1.00E-04 | grad_norm=9.775E-01 | ce=0.231 | ppl=1.262 | duration=9685.887
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 82 | ce=0.221 | ppl=1.249 | duration=365.850
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 83 | lr=1.00E-04 | grad_norm=1.019E+00 | ce=0.229 | ppl=1.260 | duration=9914.915
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 83 | ce=0.220 | ppl=1.249 | duration=387.122
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 84 | lr=1.00E-04 | grad_norm=9.639E-01 | ce=0.231 | ppl=1.262 | duration=9993.127
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 84 | ce=0.220 | ppl=1.248 | duration=365.554
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 85 | lr=1.00E-04 | grad_norm=1.003E+00 | ce=0.232 | ppl=1.264 | duration=9666.283
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 85 | ce=0.220 | ppl=1.248 | duration=365.405
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 86 | lr=1.00E-04 | grad_norm=1.012E+00 | ce=0.232 | ppl=1.263 | duration=9686.507
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 86 | ce=0.220 | ppl=1.248 | duration=365.868
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 87 | lr=1.00E-04 | grad_norm=9.838E-01 | ce=0.230 | ppl=1.261 | duration=9625.168
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 87 | ce=0.220 | ppl=1.248 | duration=361.445
[06-02 07:17:37][flashy.solver][INFO] - Train Summary | Epoch 88 | lr=1.00E-04 | grad_norm=1.021E+00 | ce=0.229 | ppl=1.260 | duration=9559.164
[06-02 07:17:37][flashy.solver][INFO] - Valid Summary | Epoch 88 | ce=0.220 | ppl=1.247 | duration=361.433
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-4-22.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-4-22.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-13.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-13.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-26.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-26.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-3-52.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-3-52.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-8-5.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-8-5.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-2-7.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-2-7.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-4-23.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-4-23.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-39.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-39.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-8-13.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-8-13.wav')
[06-02 07:17:37][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-4-15.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-4-15.wav')
[06-02 07:17:57][dora.distrib][INFO] - world_size is 1, skipping init.
[06-02 07:17:57][flashy.solver][INFO] - Instantiating solver AudioGenSolver for XP a4911d1c
[06-02 07:17:57][flashy.solver][INFO] - All XP logs are stored in /tmp/audiocraft_root/xps/a4911d1c
[06-02 07:17:57][audiocraft.solvers.builders][INFO] - Loading audio data split train: /home/yjs/project/audiocraft/egs/example
[06-02 07:17:57][audiocraft.solvers.builders][INFO] - Loading audio data split valid: /home/yjs/project/audiocraft/egs/example
[06-02 07:17:57][audiocraft.solvers.builders][INFO] - Loading audio data split evaluate: /home/yjs/project/audiocraft/egs/example
[06-02 07:17:57][audiocraft.solvers.builders][INFO] - Loading audio data split generate: /home/yjs/project/audiocraft/egs/example
[06-02 07:17:57][root][INFO] - Getting pretrained compression model from HF facebook/encodec_32khz
[06-02 07:17:58][flashy.solver][INFO] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50
[06-02 07:17:58][audiocraft.modules.conditioners][INFO] - T5 will be evaluated with autocast as float32
[06-02 07:17:59][flashy.solver][INFO] - Model hash: 949fc9e427fe11b80c38600e359ffd9a56d2b314
[06-02 07:17:59][flashy.solver][INFO] - No EMA on the model.
[06-02 07:17:59][flashy.solver][INFO] - Model size: 1.21 M params
[06-02 07:17:59][flashy.solver][INFO] - Base memory usage, with model, grad and optim: 0.02 GB
[06-02 07:17:59][flashy.solver][INFO] - Restoring weights and history.
[06-02 07:17:59][flashy.solver][INFO] - Loading existing checkpoint: /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[06-02 07:17:59][audiocraft.utils.checkpoint][INFO] - Checkpoint loaded from /tmp/audiocraft_root/xps/a4911d1c/checkpoint.th
[06-02 07:17:59][flashy.solver][INFO] - Model hash: 7ecc210f88ace59de894d0eeb72ca56e71e1cecc
[06-02 07:17:59][flashy.solver][INFO] - Replaying past metrics...
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 1 | lr=1.00E-04 | grad_norm=7.140E-01 | ce=2.174 | ppl=60.758 | duration=9990.136
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 1 | ce=1.058 | ppl=2.894 | duration=407.024
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 2 | lr=1.00E-04 | grad_norm=8.953E-01 | ce=0.762 | ppl=2.179 | duration=9709.127
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 2 | ce=0.530 | ppl=1.706 | duration=364.330
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 3 | lr=1.00E-04 | grad_norm=1.659E+00 | ce=0.459 | ppl=1.591 | duration=9646.101
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 3 | ce=0.388 | ppl=1.480 | duration=363.993
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 4 | lr=1.00E-04 | grad_norm=1.303E+00 | ce=0.378 | ppl=1.466 | duration=9648.555
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 4 | ce=0.347 | ppl=1.420 | duration=363.939
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 5 | lr=1.00E-04 | grad_norm=1.068E+00 | ce=0.351 | ppl=1.425 | duration=9649.993
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 5 | ce=0.327 | ppl=1.392 | duration=364.262
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 6 | lr=1.00E-04 | grad_norm=9.062E-01 | ce=0.333 | ppl=1.400 | duration=9648.130
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 6 | ce=0.315 | ppl=1.374 | duration=364.708
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 7 | lr=1.00E-04 | grad_norm=8.161E-01 | ce=0.322 | ppl=1.385 | duration=9692.625
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 7 | ce=0.304 | ppl=1.359 | duration=369.087
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 8 | lr=1.00E-04 | grad_norm=7.959E-01 | ce=0.317 | ppl=1.377 | duration=9762.893
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 8 | ce=0.297 | ppl=1.350 | duration=367.238
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 9 | lr=1.00E-04 | grad_norm=6.960E-01 | ce=0.304 | ppl=1.359 | duration=9715.030
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 9 | ce=0.291 | ppl=1.341 | duration=367.797
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 10 | lr=1.00E-04 | grad_norm=6.389E-01 | ce=0.300 | ppl=1.354 | duration=9720.476
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 10 | ce=0.286 | ppl=1.334 | duration=367.710
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 11 | lr=1.00E-04 | grad_norm=6.235E-01 | ce=0.292 | ppl=1.343 | duration=9698.741
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 11 | ce=0.281 | ppl=1.328 | duration=367.338
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 12 | lr=1.00E-04 | grad_norm=5.876E-01 | ce=0.289 | ppl=1.338 | duration=9717.985
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 12 | ce=0.279 | ppl=1.324 | duration=370.425
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 13 | lr=1.00E-04 | grad_norm=5.886E-01 | ce=0.286 | ppl=1.335 | duration=9734.229
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 13 | ce=0.274 | ppl=1.318 | duration=368.015
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 14 | lr=1.00E-04 | grad_norm=5.637E-01 | ce=0.281 | ppl=1.328 | duration=9727.160
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 14 | ce=0.271 | ppl=1.314 | duration=369.493
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 15 | lr=1.00E-04 | grad_norm=5.610E-01 | ce=0.279 | ppl=1.325 | duration=9589.747
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 15 | ce=0.268 | ppl=1.310 | duration=362.921
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 16 | lr=1.00E-04 | grad_norm=5.373E-01 | ce=0.274 | ppl=1.318 | duration=9552.291
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 16 | ce=0.265 | ppl=1.306 | duration=362.290
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 17 | lr=1.00E-04 | grad_norm=5.495E-01 | ce=0.274 | ppl=1.318 | duration=9593.252
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 17 | ce=0.263 | ppl=1.303 | duration=361.603
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 18 | lr=1.00E-04 | grad_norm=5.303E-01 | ce=0.270 | ppl=1.314 | duration=9601.733
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 18 | ce=0.261 | ppl=1.301 | duration=362.833
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 19 | lr=1.00E-04 | grad_norm=5.436E-01 | ce=0.271 | ppl=1.314 | duration=9603.793
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 19 | ce=0.259 | ppl=1.298 | duration=362.807
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 20 | lr=1.00E-04 | grad_norm=5.256E-01 | ce=0.267 | ppl=1.309 | duration=9513.591
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 20 | ce=0.256 | ppl=1.295 | duration=359.164
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 21 | lr=1.00E-04 | grad_norm=5.398E-01 | ce=0.267 | ppl=1.309 | duration=9521.433
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 21 | ce=0.255 | ppl=1.293 | duration=359.736
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 22 | lr=1.00E-04 | grad_norm=5.273E-01 | ce=0.264 | ppl=1.305 | duration=9522.932
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 22 | ce=0.254 | ppl=1.291 | duration=359.952
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 23 | lr=1.00E-04 | grad_norm=5.359E-01 | ce=0.263 | ppl=1.304 | duration=9520.848
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 23 | ce=0.252 | ppl=1.290 | duration=359.946
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 24 | lr=1.00E-04 | grad_norm=5.356E-01 | ce=0.260 | ppl=1.300 | duration=9517.164
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 24 | ce=0.250 | ppl=1.287 | duration=360.223
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 25 | lr=1.00E-04 | grad_norm=5.350E-01 | ce=0.258 | ppl=1.297 | duration=9526.316
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 25 | ce=0.249 | ppl=1.286 | duration=359.858
[06-02 07:17:59][flashy.solver][INFO] - Evaluate Summary | Epoch 25 | duration=0.000
[06-02 07:17:59][flashy.solver][INFO] - Generate Summary | Epoch 25 | rtf=0.143 | duration=7.398
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 26 | lr=1.00E-04 | grad_norm=5.344E-01 | ce=0.255 | ppl=1.294 | duration=9505.304
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 26 | ce=0.248 | ppl=1.284 | duration=359.841
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 27 | lr=1.00E-04 | grad_norm=5.476E-01 | ce=0.256 | ppl=1.294 | duration=9515.969
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 27 | ce=0.247 | ppl=1.282 | duration=359.513
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 28 | lr=1.00E-04 | grad_norm=5.444E-01 | ce=0.254 | ppl=1.292 | duration=9521.965
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 28 | ce=0.245 | ppl=1.281 | duration=360.116
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 29 | lr=1.00E-04 | grad_norm=5.705E-01 | ce=0.255 | ppl=1.293 | duration=9536.583
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 29 | ce=0.245 | ppl=1.280 | duration=361.077
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 30 | lr=1.00E-04 | grad_norm=5.520E-01 | ce=0.253 | ppl=1.291 | duration=9517.157
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 30 | ce=0.243 | ppl=1.278 | duration=360.175
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 31 | lr=1.00E-04 | grad_norm=5.536E-01 | ce=0.252 | ppl=1.290 | duration=9515.999
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 31 | ce=0.242 | ppl=1.277 | duration=359.757
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 32 | lr=1.00E-04 | grad_norm=5.734E-01 | ce=0.251 | ppl=1.288 | duration=9525.455
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 32 | ce=0.241 | ppl=1.275 | duration=359.850
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 33 | lr=1.00E-04 | grad_norm=5.841E-01 | ce=0.252 | ppl=1.289 | duration=9522.251
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 33 | ce=0.240 | ppl=1.274 | duration=360.855
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 34 | lr=1.00E-04 | grad_norm=5.932E-01 | ce=0.250 | ppl=1.287 | duration=9530.122
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 34 | ce=0.240 | ppl=1.273 | duration=360.152
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 35 | lr=1.00E-04 | grad_norm=5.957E-01 | ce=0.248 | ppl=1.284 | duration=9524.631
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 35 | ce=0.239 | ppl=1.272 | duration=359.587
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 36 | lr=1.00E-04 | grad_norm=6.259E-01 | ce=0.251 | ppl=1.288 | duration=9531.088
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 36 | ce=0.239 | ppl=1.272 | duration=360.096
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 37 | lr=1.00E-04 | grad_norm=6.270E-01 | ce=0.248 | ppl=1.284 | duration=9533.307
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 37 | ce=0.237 | ppl=1.270 | duration=360.546
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 38 | lr=1.00E-04 | grad_norm=6.272E-01 | ce=0.248 | ppl=1.284 | duration=9529.414
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 38 | ce=0.237 | ppl=1.270 | duration=360.767
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 39 | lr=1.00E-04 | grad_norm=6.386E-01 | ce=0.246 | ppl=1.281 | duration=9524.562
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 39 | ce=0.236 | ppl=1.268 | duration=360.385
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 40 | lr=1.00E-04 | grad_norm=6.360E-01 | ce=0.244 | ppl=1.279 | duration=9642.435
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 40 | ce=0.235 | ppl=1.267 | duration=385.528
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 41 | lr=1.00E-04 | grad_norm=6.410E-01 | ce=0.246 | ppl=1.281 | duration=9668.847
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 41 | ce=0.235 | ppl=1.267 | duration=363.285
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 42 | lr=1.00E-04 | grad_norm=6.642E-01 | ce=0.244 | ppl=1.278 | duration=9572.143
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 42 | ce=0.234 | ppl=1.266 | duration=362.071
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 43 | lr=1.00E-04 | grad_norm=6.578E-01 | ce=0.244 | ppl=1.278 | duration=9564.111
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 43 | ce=0.234 | ppl=1.265 | duration=362.399
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 44 | lr=1.00E-04 | grad_norm=6.778E-01 | ce=0.244 | ppl=1.279 | duration=9545.633
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 44 | ce=0.233 | ppl=1.264 | duration=360.226
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 45 | lr=1.00E-04 | grad_norm=6.707E-01 | ce=0.240 | ppl=1.273 | duration=9558.510
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 45 | ce=0.232 | ppl=1.263 | duration=361.292
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 46 | lr=1.00E-04 | grad_norm=7.081E-01 | ce=0.243 | ppl=1.277 | duration=9549.454
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 46 | ce=0.232 | ppl=1.263 | duration=361.476
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 47 | lr=1.00E-04 | grad_norm=6.983E-01 | ce=0.241 | ppl=1.275 | duration=9994.908
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 47 | ce=0.231 | ppl=1.262 | duration=380.415
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 48 | lr=1.00E-04 | grad_norm=7.072E-01 | ce=0.241 | ppl=1.274 | duration=9756.736
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 48 | ce=0.231 | ppl=1.262 | duration=368.529
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 49 | lr=1.00E-04 | grad_norm=7.122E-01 | ce=0.239 | ppl=1.273 | duration=9719.254
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 49 | ce=0.230 | ppl=1.261 | duration=360.200
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 50 | lr=1.00E-04 | grad_norm=7.181E-01 | ce=0.237 | ppl=1.270 | duration=9752.171
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 50 | ce=0.230 | ppl=1.261 | duration=359.721
[06-02 07:17:59][flashy.solver][INFO] - Evaluate Summary | Epoch 50 | duration=0.000
[06-02 07:17:59][flashy.solver][INFO] - Generate Summary | Epoch 50 | rtf=0.134 | duration=6.334
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 51 | lr=1.00E-04 | grad_norm=7.459E-01 | ce=0.243 | ppl=1.277 | duration=9700.451
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 51 | ce=0.230 | ppl=1.260 | duration=360.665
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 52 | lr=1.00E-04 | grad_norm=7.595E-01 | ce=0.241 | ppl=1.274 | duration=9529.466
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 52 | ce=0.229 | ppl=1.260 | duration=360.312
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 53 | lr=1.00E-04 | grad_norm=7.351E-01 | ce=0.239 | ppl=1.273 | duration=9539.188
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 53 | ce=0.229 | ppl=1.259 | duration=359.881
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 54 | lr=1.00E-04 | grad_norm=7.549E-01 | ce=0.239 | ppl=1.272 | duration=9542.972
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 54 | ce=0.228 | ppl=1.258 | duration=368.039
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 55 | lr=1.00E-04 | grad_norm=7.593E-01 | ce=0.237 | ppl=1.269 | duration=9580.192
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 55 | ce=0.228 | ppl=1.258 | duration=380.776
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 56 | lr=1.00E-04 | grad_norm=7.741E-01 | ce=0.237 | ppl=1.270 | duration=10097.090
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 56 | ce=0.228 | ppl=1.258 | duration=380.956
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 57 | lr=1.00E-04 | grad_norm=7.629E-01 | ce=0.238 | ppl=1.271 | duration=9941.644
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 57 | ce=0.227 | ppl=1.257 | duration=369.864
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 58 | lr=1.00E-04 | grad_norm=7.822E-01 | ce=0.237 | ppl=1.270 | duration=9753.884
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 58 | ce=0.227 | ppl=1.257 | duration=368.705
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 59 | lr=1.00E-04 | grad_norm=7.903E-01 | ce=0.237 | ppl=1.269 | duration=9762.634
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 59 | ce=0.226 | ppl=1.256 | duration=370.989
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 60 | lr=1.00E-04 | grad_norm=8.007E-01 | ce=0.237 | ppl=1.269 | duration=9874.322
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 60 | ce=0.227 | ppl=1.256 | duration=375.685
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 61 | lr=1.00E-04 | grad_norm=8.122E-01 | ce=0.238 | ppl=1.271 | duration=9884.318
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 61 | ce=0.226 | ppl=1.255 | duration=372.385
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 62 | lr=1.00E-04 | grad_norm=7.932E-01 | ce=0.234 | ppl=1.266 | duration=9840.462
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 62 | ce=0.225 | ppl=1.255 | duration=371.850
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 63 | lr=1.00E-04 | grad_norm=8.240E-01 | ce=0.233 | ppl=1.265 | duration=9806.020
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 63 | ce=0.226 | ppl=1.255 | duration=369.038
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 64 | lr=1.00E-04 | grad_norm=8.545E-01 | ce=0.237 | ppl=1.269 | duration=9878.421
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 64 | ce=0.225 | ppl=1.254 | duration=370.405
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 65 | lr=1.00E-04 | grad_norm=8.380E-01 | ce=0.235 | ppl=1.267 | duration=9768.115
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 65 | ce=0.225 | ppl=1.254 | duration=368.232
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 66 | lr=1.00E-04 | grad_norm=8.622E-01 | ce=0.234 | ppl=1.266 | duration=9705.978
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 66 | ce=0.225 | ppl=1.254 | duration=368.388
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 67 | lr=1.00E-04 | grad_norm=8.433E-01 | ce=0.235 | ppl=1.267 | duration=9708.521
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 67 | ce=0.224 | ppl=1.253 | duration=366.592
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 68 | lr=1.00E-04 | grad_norm=8.658E-01 | ce=0.233 | ppl=1.264 | duration=9681.177
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 68 | ce=0.224 | ppl=1.253 | duration=366.955
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 69 | lr=1.00E-04 | grad_norm=8.897E-01 | ce=0.234 | ppl=1.266 | duration=9676.317
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 69 | ce=0.224 | ppl=1.253 | duration=365.945
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 70 | lr=1.00E-04 | grad_norm=8.614E-01 | ce=0.231 | ppl=1.262 | duration=9693.598
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 70 | ce=0.223 | ppl=1.252 | duration=366.943
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 71 | lr=1.00E-04 | grad_norm=9.080E-01 | ce=0.235 | ppl=1.268 | duration=9683.285
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 71 | ce=0.223 | ppl=1.252 | duration=368.008
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 72 | lr=1.00E-04 | grad_norm=8.908E-01 | ce=0.233 | ppl=1.265 | duration=9695.132
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 72 | ce=0.222 | ppl=1.251 | duration=366.920
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 73 | lr=1.00E-04 | grad_norm=9.097E-01 | ce=0.235 | ppl=1.267 | duration=9707.760
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 73 | ce=0.223 | ppl=1.251 | duration=368.145
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 74 | lr=1.00E-04 | grad_norm=9.211E-01 | ce=0.234 | ppl=1.266 | duration=9650.362
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 74 | ce=0.222 | ppl=1.251 | duration=361.467
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 75 | lr=1.00E-04 | grad_norm=9.272E-01 | ce=0.233 | ppl=1.265 | duration=9633.944
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 75 | ce=0.222 | ppl=1.251 | duration=367.500
[06-02 07:17:59][flashy.solver][INFO] - Evaluate Summary | Epoch 75 | duration=0.000
[06-02 07:17:59][flashy.solver][INFO] - Generate Summary | Epoch 75 | rtf=0.144 | duration=6.500
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 76 | lr=1.00E-04 | grad_norm=9.470E-01 | ce=0.232 | ppl=1.263 | duration=9666.764
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 76 | ce=0.222 | ppl=1.250 | duration=365.508
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 77 | lr=1.00E-04 | grad_norm=9.267E-01 | ce=0.230 | ppl=1.261 | duration=9671.955
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 77 | ce=0.222 | ppl=1.250 | duration=365.621
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 78 | lr=1.00E-04 | grad_norm=9.488E-01 | ce=0.232 | ppl=1.263 | duration=9607.441
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 78 | ce=0.222 | ppl=1.250 | duration=361.455
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 79 | lr=1.00E-04 | grad_norm=9.638E-01 | ce=0.230 | ppl=1.261 | duration=9569.696
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 79 | ce=0.221 | ppl=1.250 | duration=361.342
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 80 | lr=1.00E-04 | grad_norm=9.486E-01 | ce=0.229 | ppl=1.260 | duration=9602.914
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 80 | ce=0.221 | ppl=1.249 | duration=365.637
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 81 | lr=1.00E-04 | grad_norm=9.651E-01 | ce=0.232 | ppl=1.263 | duration=9679.771
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 81 | ce=0.220 | ppl=1.249 | duration=366.636
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 82 | lr=1.00E-04 | grad_norm=9.775E-01 | ce=0.231 | ppl=1.262 | duration=9685.887
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 82 | ce=0.221 | ppl=1.249 | duration=365.850
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 83 | lr=1.00E-04 | grad_norm=1.019E+00 | ce=0.229 | ppl=1.260 | duration=9914.915
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 83 | ce=0.220 | ppl=1.249 | duration=387.122
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 84 | lr=1.00E-04 | grad_norm=9.639E-01 | ce=0.231 | ppl=1.262 | duration=9993.127
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 84 | ce=0.220 | ppl=1.248 | duration=365.554
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 85 | lr=1.00E-04 | grad_norm=1.003E+00 | ce=0.232 | ppl=1.264 | duration=9666.283
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 85 | ce=0.220 | ppl=1.248 | duration=365.405
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 86 | lr=1.00E-04 | grad_norm=1.012E+00 | ce=0.232 | ppl=1.263 | duration=9686.507
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 86 | ce=0.220 | ppl=1.248 | duration=365.868
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 87 | lr=1.00E-04 | grad_norm=9.838E-01 | ce=0.230 | ppl=1.261 | duration=9625.168
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 87 | ce=0.220 | ppl=1.248 | duration=361.445
[06-02 07:17:59][flashy.solver][INFO] - Train Summary | Epoch 88 | lr=1.00E-04 | grad_norm=1.021E+00 | ce=0.229 | ppl=1.260 | duration=9559.164
[06-02 07:17:59][flashy.solver][INFO] - Valid Summary | Epoch 88 | ce=0.220 | ppl=1.247 | duration=361.433
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-4-22.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-4-22.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-13.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-12-13.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-26.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-10-28-7-26.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-3-52.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-3-52.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-8-5.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-8-5.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-2-7.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-2-7.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-4-23.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-10-4-23.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-39.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-1-39.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-8-13.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-12-8-13.wav')
[06-02 07:17:59][audiocraft.data.audio_dataset][WARNING] - Error opening file /home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-4-15.wav: FileNotFoundError(2, 'No such file or directory', '/home/yjsung/VibViz-Dataset-master/Vibviz/v-09-18-4-15.wav')