Quantumbraid's picture
download
raw
247 kB
Running 1 job
{
"type": "diffusion_trainer",
"training_folder": "/app/ai-toolkit/output",
"sqlite_db_path": "/app/ai-toolkit/aitk_db.db",
"device": "cuda",
"trigger_word": null,
"performance_log_every": 10,
"network": {
"type": "lora",
"linear": 32,
"linear_alpha": 32,
"conv": 16,
"conv_alpha": 16,
"lokr_full_rank": true,
"lokr_factor": -1,
"network_kwargs": {
"ignore_if_contains": []
}
},
"save": {
"dtype": "bf16",
"save_every": 50,
"max_step_saves_to_keep": 300,
"save_format": "diffusers",
"push_to_hub": false
},
"datasets": [
{
"folder_path": "/app/ai-toolkit/datasets/jess",
"mask_path": null,
"mask_min_value": 0.1,
"default_caption": "jess",
"caption_ext": "txt",
"caption_dropout_rate": 0.05,
"cache_latents_to_disk": true,
"is_reg": false,
"network_weight": 1,
"resolution": [
512,
256
],
"controls": [],
"shrink_video_to_frames": true,
"num_frames": 1,
"flip_x": false,
"flip_y": false,
"num_repeats": 1
}
],
"train": {
"batch_size": 7,
"bypass_guidance_embedding": false,
"steps": 8000,
"gradient_accumulation": 1,
"train_unet": true,
"train_text_encoder": false,
"gradient_checkpointing": true,
"noise_scheduler": "ddpm",
"optimizer": "adamw8bit",
"timestep_type": "sigmoid",
"content_or_style": "balanced",
"optimizer_params": {
"weight_decay": 0.0001
},
"unload_text_encoder": false,
"cache_text_embeddings": false,
"lr": 6e-05,
"ema_config": {
"use_ema": false,
"ema_decay": 0.99
},
"skip_first_sample": false,
"force_first_sample": false,
"disable_sampling": false,
"dtype": "bf16",
"diff_output_preservation": false,
"diff_output_preservation_multiplier": 1,
"diff_output_preservation_class": "person",
"switch_boundary_every": 1,
"loss_type": "mse",
"do_differential_guidance": true,
"differential_guidance_scale": 3
},
"logging": {
"log_every": 1,
"use_ui_logger": true
},
"model": {
"name_or_path": "stable-diffusion-v1-5/stable-diffusion-v1-5",
"quantize": true,
"qtype": "qfloat8",
"quantize_te": true,
"qtype_te": "qfloat8",
"arch": "sd1",
"low_vram": false,
"model_kwargs": {},
"compile": false
},
"sample": {
"sampler": "ddpm",
"sample_every": 100,
"sample_start_step": 0,
"width": 512,
"height": 512,
"samples": [
{
"prompt": "jess standing nude in the kithen, having morning coffee"
},
{
"prompt": "jess standing in a hot steamy shower nude"
},
{
"prompt": "jess standing nude in the street, neon signs reflect, raining"
}
],
"neg": "",
"seed": 42,
"walk_seed": false,
"guidance_scale": 6,
"sample_steps": 30,
"num_frames": 1,
"fps": 1
}
}
Using SQLite database at /app/ai-toolkit/aitk_db.db
Job ID: "8a6f2513-8d3b-4309-9835-c12b94aafd6e"
#############################################
# Running job: jess
#############################################
Running 1 process
Loading pipeline components...: 0%| | 0/6 [00:00<?, ?it/s]
Loading weights: 0%| | 0/196 [00:00<?, ?it/s] Loading weights: 100%|██████████| 196/196 [00:00<00:00, 9994.57it/s]
Loading pipeline components...: 83%|████████▎ | 5/6 [00:00<00:00, 35.44it/s] Loading pipeline components...: 100%|██████████| 6/6 [00:00<00:00, 35.80it/s]
create LoRA network. base dim (rank): 32, alpha: 32
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
apply LoRA to Conv2d with kernel size (3,3). dim (rank): 16, alpha: 16
create LoRA for Text Encoder: 0 modules.
create LoRA for U-Net: 278 modules.
enable LoRA for U-Net
#### IMPORTANT RESUMING FROM /app/ai-toolkit/output/jess/jess_000003500.safetensors ####
Loading from /app/ai-toolkit/output/jess/jess_000003500.safetensors
Missing keys: []
Found step 3500 in metadata, starting from there
Loading optimizer state from /app/ai-toolkit/output/jess/optimizer.pt
Updating optimizer LR from params
Dataset: /app/ai-toolkit/datasets/jess
- Preprocessing image dimensions
0%| | 0/97 [00:00<?, ?it/s] 100%|██████████| 97/97 [00:00<00:00, 70328.00it/s]
- Found 97 images
Bucket sizes for /app/ai-toolkit/datasets/jess:
512x512: 19 files
416x624: 65 files
528x496: 2 files
624x416: 8 files
672x384: 1 files
448x576: 1 files
384x576: 1 files
7 buckets made
Caching latents for /app/ai-toolkit/datasets/jess
- Saving latents to disk
Caching latents to disk: 0%| | 0/97 [00:00<?, ?it/s] Caching latents to disk: 100%|██████████| 97/97 [00:00<00:00, 11589.45it/s]
Dataset: /app/ai-toolkit/datasets/jess
- Preprocessing image dimensions
0%| | 0/97 [00:00<?, ?it/s] 100%|██████████| 97/97 [00:00<00:00, 92845.16it/s]
- Found 97 images
Bucket sizes for /app/ai-toolkit/datasets/jess:
256x256: 21 files
208x304: 66 files
304x208: 8 files
336x192: 1 files
224x288: 1 files
5 buckets made
Caching latents for /app/ai-toolkit/datasets/jess
- Saving latents to disk
Caching latents to disk: 0%| | 0/97 [00:00<?, ?it/s] Caching latents to disk: 100%|██████████| 97/97 [00:00<00:00, 15793.77it/s]
jess: 44%|████▍ | 3500/8000 [00:00<?, ?it/s] jess: 44%|████▍ | 3500/8000 [00:00<?, ?it/s, lr: 6.0e-05 loss: 7.155e-03] jess: 44%|████▍ | 3501/8000 [00:00<1:01:35, 1.22it/s, lr: 6.0e-05 loss: 7.155e-03] jess: 44%|████▍ | 3501/8000 [00:01<1:01:35, 1.22it/s, lr: 6.0e-05 loss: 8.579e-02] jess: 44%|████▍ | 3502/8000 [00:01<30:20, 2.47it/s, lr: 6.0e-05 loss: 8.579e-02] jess: 44%|████▍ | 3502/8000 [00:01<30:20, 2.47it/s, lr: 6.0e-05 loss: 1.534e-01] jess: 44%|████▍ | 3503/8000 [00:01<21:47, 3.44it/s, lr: 6.0e-05 loss: 1.534e-01] jess: 44%|████▍ | 3503/8000 [00:01<21:47, 3.44it/s, lr: 6.0e-05 loss: 5.770e-02] jess: 44%|████▍ | 3504/8000 [00:01<14:13, 5.27it/s, lr: 6.0e-05 loss: 5.770e-02] jess: 44%|████▍ | 3504/8000 [00:01<14:13, 5.27it/s, lr: 6.0e-05 loss: 5.689e-02] jess: 44%|████▍ | 3505/8000 [00:01<20:24, 3.67it/s, lr: 6.0e-05 loss: 5.689e-02] jess: 44%|████▍ | 3505/8000 [00:02<20:24, 3.67it/s, lr: 6.0e-05 loss: 3.637e-02] jess: 44%|████▍ | 3506/8000 [00:02<20:27, 3.66it/s, lr: 6.0e-05 loss: 3.637e-02] jess: 44%|████▍ | 3506/8000 [00:02<20:27, 3.66it/s, lr: 6.0e-05 loss: 1.244e-01] jess: 44%|████▍ | 3507/8000 [00:02<20:31, 3.65it/s, lr: 6.0e-05 loss: 1.244e-01] jess: 44%|████▍ | 3507/8000 [00:02<20:31, 3.65it/s, lr: 6.0e-05 loss: 5.810e-02] jess: 44%|████▍ | 3508/8000 [00:02<14:35, 5.13it/s, lr: 6.0e-05 loss: 5.810e-02] jess: 44%|████▍ | 3508/8000 [00:02<14:35, 5.13it/s, lr: 6.0e-05 loss: 1.021e-01] jess: 44%|████▍ | 3509/8000 [00:02<12:44, 5.87it/s, lr: 6.0e-05 loss: 1.021e-01] jess: 44%|████▍ | 3509/8000 [00:02<12:44, 5.87it/s, lr: 6.0e-05 loss: 1.526e-01] jess: 44%|████▍ | 3510/8000 [00:02<18:07, 4.13it/s, lr: 6.0e-05 loss: 1.526e-01] jess: 44%|████▍ | 3510/8000 [00:03<18:07, 4.13it/s, lr: 6.0e-05 loss: 7.881e-02] jess: 44%|████▍ | 3511/8000 [00:03<13:51, 5.40it/s, lr: 6.0e-05 loss: 7.881e-02] jess: 44%|████▍ | 3511/8000 [00:03<13:51, 5.40it/s, lr: 6.0e-05 loss: 6.962e-02] jess: 44%|████▍ | 3512/8000 [00:03<18:28, 4.05it/s, lr: 6.0e-05 loss: 6.962e-02] jess: 44%|████▍ | 3512/8000 [00:03<18:28, 4.05it/s, lr: 6.0e-05 loss: 9.116e-02] jess: 44%|████▍ | 3513/8000 [00:03<14:01, 5.33it/s, lr: 6.0e-05 loss: 9.116e-02] jess: 44%|████▍ | 3513/8000 [00:03<14:01, 5.33it/s, lr: 6.0e-05 loss: 1.518e-01] jess: 44%|████▍ | 3514/8000 [00:03<18:55, 3.95it/s, lr: 6.0e-05 loss: 1.518e-01] jess: 44%|████▍ | 3514/8000 [00:04<18:55, 3.95it/s, lr: 6.0e-05 loss: 1.413e-01] jess: 44%|████▍ | 3515/8000 [00:04<19:58, 3.74it/s, lr: 6.0e-05 loss: 1.413e-01] jess: 44%|████▍ | 3515/8000 [00:04<19:58, 3.74it/s, lr: 6.0e-05 loss: 6.929e-02] jess: 44%|████▍ | 3516/8000 [00:04<13:43, 5.44it/s, lr: 6.0e-05 loss: 6.929e-02] jess: 44%|████▍ | 3516/8000 [00:04<13:43, 5.44it/s, lr: 6.0e-05 loss: 5.211e-02] jess: 44%|████▍ | 3517/8000 [00:04<11:49, 6.31it/s, lr: 6.0e-05 loss: 5.211e-02] jess: 44%|████▍ | 3517/8000 [00:04<11:49, 6.31it/s, lr: 6.0e-05 loss: 1.047e-01] jess: 44%|████▍ | 3518/8000 [00:04<11:12, 6.66it/s, lr: 6.0e-05 loss: 1.047e-01] jess: 44%|████▍ | 3518/8000 [00:04<11:12, 6.66it/s, lr: 6.0e-05 loss: 5.420e-02] jess: 44%|████▍ | 3519/8000 [00:04<18:22, 4.07it/s, lr: 6.0e-05 loss: 5.420e-02] jess: 44%|████▍ | 3519/8000 [00:04<18:22, 4.07it/s, lr: 6.0e-05 loss: 6.599e-02] jess: 44%|████▍ | 3520/8000 [00:04<13:21, 5.59it/s, lr: 6.0e-05 loss: 6.599e-02] jess: 44%|████▍ | 3520/8000 [00:05<13:21, 5.59it/s, lr: 6.0e-05 loss: 2.255e-01] jess: 44%|████▍ | 3521/8000 [00:05<18:56, 3.94it/s, lr: 6.0e-05 loss: 2.255e-01] jess: 44%|████▍ | 3521/8000 [00:05<18:56, 3.94it/s, lr: 6.0e-05 loss: 6.361e-02] jess: 44%|████▍ | 3522/8000 [00:05<21:02, 3.55it/s, lr: 6.0e-05 loss: 6.361e-02] jess: 44%|████▍ | 3522/8000 [00:05<21:02, 3.55it/s, lr: 6.0e-05 loss: 1.561e-01] jess: 44%|████▍ | 3523/8000 [00:05<20:32, 3.63it/s, lr: 6.0e-05 loss: 1.561e-01] jess: 44%|████▍ | 3523/8000 [00:06<20:32, 3.63it/s, lr: 6.0e-05 loss: 7.904e-02] jess: 44%|████▍ | 3524/8000 [00:06<34:39, 2.15it/s, lr: 6.0e-05 loss: 7.904e-02] jess: 44%|████▍ | 3524/8000 [00:06<34:39, 2.15it/s, lr: 6.0e-05 loss: 4.766e-02] jess: 44%|████▍ | 3525/8000 [00:06<18:14, 4.09it/s, lr: 6.0e-05 loss: 4.766e-02] jess: 44%|████▍ | 3525/8000 [00:06<18:14, 4.09it/s, lr: 6.0e-05 loss: 1.720e-01] jess: 44%|████▍ | 3526/8000 [00:06<13:11, 5.65it/s, lr: 6.0e-05 loss: 1.720e-01] jess: 44%|████▍ | 3526/8000 [00:06<13:11, 5.65it/s, lr: 6.0e-05 loss: 1.422e-01] jess: 44%|████▍ | 3527/8000 [00:06<18:00, 4.14it/s, lr: 6.0e-05 loss: 1.422e-01] jess: 44%|████▍ | 3527/8000 [00:07<18:00, 4.14it/s, lr: 6.0e-05 loss: 1.045e-01] jess: 44%|████▍ | 3528/8000 [00:07<19:35, 3.81it/s, lr: 6.0e-05 loss: 1.045e-01] jess: 44%|████▍ | 3528/8000 [00:07<19:35, 3.81it/s, lr: 6.0e-05 loss: 3.109e-02] jess: 44%|████▍ | 3529/8000 [00:07<20:24, 3.65it/s, lr: 6.0e-05 loss: 3.109e-02] jess: 44%|████▍ | 3529/8000 [00:07<20:24, 3.65it/s, lr: 6.0e-05 loss: 3.908e-02] jess: 44%|████▍ | 3530/8000 [00:07<19:47, 3.76it/s, lr: 6.0e-05 loss: 3.908e-02] jess: 44%|████▍ | 3530/8000 [00:07<19:47, 3.76it/s, lr: 6.0e-05 loss: 1.493e-01] jess: 44%|████▍ | 3531/8000 [00:07<14:01, 5.31it/s, lr: 6.0e-05 loss: 1.493e-01] jess: 44%|████▍ | 3531/8000 [00:08<14:01, 5.31it/s, lr: 6.0e-05 loss: 1.450e-01] jess: 44%|████▍ | 3532/8000 [00:08<18:24, 4.05it/s, lr: 6.0e-05 loss: 1.450e-01] jess: 44%|████▍ | 3532/8000 [00:08<18:24, 4.05it/s, lr: 6.0e-05 loss: 7.279e-02] jess: 44%|████▍ | 3533/8000 [00:08<19:32, 3.81it/s, lr: 6.0e-05 loss: 7.279e-02] jess: 44%|████▍ | 3533/8000 [00:08<19:32, 3.81it/s, lr: 6.0e-05 loss: 8.620e-02] jess: 44%|████▍ | 3534/8000 [00:08<19:57, 3.73it/s, lr: 6.0e-05 loss: 8.620e-02] jess: 44%|████▍ | 3534/8000 [00:08<19:57, 3.73it/s, lr: 6.0e-05 loss: 3.173e-02] jess: 44%|████▍ | 3535/8000 [00:08<20:30, 3.63it/s, lr: 6.0e-05 loss: 3.173e-02] jess: 44%|████▍ | 3535/8000 [00:09<20:30, 3.63it/s, lr: 6.0e-05 loss: 1.048e-01] jess: 44%|████▍ | 3536/8000 [00:09<20:19, 3.66it/s, lr: 6.0e-05 loss: 1.048e-01] jess: 44%|████▍ | 3536/8000 [00:09<20:19, 3.66it/s, lr: 6.0e-05 loss: 6.062e-02] jess: 44%|████▍ | 3537/8000 [00:09<14:16, 5.21it/s, lr: 6.0e-05 loss: 6.062e-02] jess: 44%|████▍ | 3537/8000 [00:09<14:16, 5.21it/s, lr: 6.0e-05 loss: 1.420e-01] jess: 44%|████▍ | 3538/8000 [00:09<18:15, 4.07it/s, lr: 6.0e-05 loss: 1.420e-01] jess: 44%|████▍ | 3538/8000 [00:09<18:15, 4.07it/s, lr: 6.0e-05 loss: 6.432e-02] jess: 44%|████▍ | 3539/8000 [00:09<19:43, 3.77it/s, lr: 6.0e-05 loss: 6.432e-02] jess: 44%|████▍ | 3539/8000 [00:10<19:43, 3.77it/s, lr: 6.0e-05 loss: 1.551e-01] jess: 44%|████▍ | 3540/8000 [00:10<20:01, 3.71it/s, lr: 6.0e-05 loss: 1.551e-01] jess: 44%|████▍ | 3540/8000 [00:10<20:01, 3.71it/s, lr: 6.0e-05 loss: 5.418e-02] jess: 44%|████▍ | 3541/8000 [00:10<13:44, 5.41it/s, lr: 6.0e-05 loss: 5.418e-02] jess: 44%|████▍ | 3541/8000 [00:10<13:44, 5.41it/s, lr: 6.0e-05 loss: 1.035e-01] jess: 44%|████▍ | 3542/8000 [00:10<11:43, 6.34it/s, lr: 6.0e-05 loss: 1.035e-01] jess: 44%|████▍ | 3542/8000 [00:10<11:43, 6.34it/s, lr: 6.0e-05 loss: 6.867e-02] jess: 44%|████▍ | 3543/8000 [00:10<11:06, 6.68it/s, lr: 6.0e-05 loss: 6.867e-02] jess: 44%|████▍ | 3543/8000 [00:10<11:06, 6.68it/s, lr: 6.0e-05 loss: 1.404e-01] jess: 44%|████▍ | 3544/8000 [00:10<17:18, 4.29it/s, lr: 6.0e-05 loss: 1.404e-01] jess: 44%|████▍ | 3544/8000 [00:11<17:18, 4.29it/s, lr: 6.0e-05 loss: 8.298e-02] jess: 44%|████▍ | 3545/8000 [00:11<19:18, 3.84it/s, lr: 6.0e-05 loss: 8.298e-02] jess: 44%|████▍ | 3545/8000 [00:11<19:18, 3.84it/s, lr: 6.0e-05 loss: 2.297e-01] jess: 44%|████▍ | 3546/8000 [00:11<19:56, 3.72it/s, lr: 6.0e-05 loss: 2.297e-01] jess: 44%|████▍ | 3546/8000 [00:11<19:56, 3.72it/s, lr: 6.0e-05 loss: 5.715e-02] jess: 44%|████▍ | 3547/8000 [00:11<20:11, 3.68it/s, lr: 6.0e-05 loss: 5.715e-02] jess: 44%|████▍ | 3547/8000 [00:12<20:11, 3.68it/s, lr: 6.0e-05 loss: 1.596e-01] jess: 44%|████▍ | 3548/8000 [00:12<33:20, 2.23it/s, lr: 6.0e-05 loss: 1.596e-01] jess: 44%|████▍ | 3548/8000 [00:12<33:20, 2.23it/s, lr: 6.0e-05 loss: 1.269e-01] jess: 44%|████▍ | 3549/8000 [00:12<17:45, 4.18it/s, lr: 6.0e-05 loss: 1.269e-01] jess: 44%|████▍ | 3549/8000 [00:12<17:45, 4.18it/s, lr: 6.0e-05 loss: 7.719e-02] jess: 44%|████▍ | 3550/8000 [00:12<19:16, 3.85it/s, lr: 6.0e-05 loss: 7.719e-02] jess: 44%|████▍ | 3550/8000 [00:12<19:16, 3.85it/s, lr: 6.0e-05 loss: 1.984e-01]
Saving at step 3550
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003550.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 44%|████▍ | 3551/8000 [00:13<19:16, 3.85it/s, lr: 6.0e-05 loss: 9.443e-02] jess: 44%|████▍ | 3552/8000 [00:13<16:27, 4.50it/s, lr: 6.0e-05 loss: 9.443e-02] jess: 44%|████▍ | 3552/8000 [00:13<16:27, 4.50it/s, lr: 6.0e-05 loss: 3.962e-02] jess: 44%|████▍ | 3553/8000 [00:13<17:24, 4.26it/s, lr: 6.0e-05 loss: 3.962e-02] jess: 44%|████▍ | 3553/8000 [00:13<17:24, 4.26it/s, lr: 6.0e-05 loss: 1.902e-02] jess: 44%|████▍ | 3554/8000 [00:13<18:54, 3.92it/s, lr: 6.0e-05 loss: 1.902e-02] jess: 44%|████▍ | 3554/8000 [00:14<18:54, 3.92it/s, lr: 6.0e-05 loss: 1.216e-02] jess: 44%|████▍ | 3555/8000 [00:14<13:22, 5.54it/s, lr: 6.0e-05 loss: 1.216e-02] jess: 44%|████▍ | 3555/8000 [00:14<13:22, 5.54it/s, lr: 6.0e-05 loss: 3.960e-02] jess: 44%|████▍ | 3556/8000 [00:14<11:38, 6.36it/s, lr: 6.0e-05 loss: 3.960e-02] jess: 44%|████▍ | 3556/8000 [00:14<11:38, 6.36it/s, lr: 6.0e-05 loss: 6.973e-02] jess: 44%|████▍ | 3557/8000 [00:14<17:27, 4.24it/s, lr: 6.0e-05 loss: 6.973e-02] jess: 44%|████▍ | 3557/8000 [00:14<17:27, 4.24it/s, lr: 6.0e-05 loss: 5.040e-02] jess: 44%|████▍ | 3558/8000 [00:14<19:45, 3.75it/s, lr: 6.0e-05 loss: 5.040e-02] jess: 44%|████▍ | 3558/8000 [00:14<19:45, 3.75it/s, lr: 6.0e-05 loss: 1.951e-01] jess: 44%|████▍ | 3559/8000 [00:14<13:30, 5.48it/s, lr: 6.0e-05 loss: 1.951e-01] jess: 44%|████▍ | 3559/8000 [00:15<13:30, 5.48it/s, lr: 6.0e-05 loss: 6.653e-03] jess: 44%|████▍ | 3560/8000 [00:15<11:43, 6.31it/s, lr: 6.0e-05 loss: 6.653e-03] jess: 44%|████▍ | 3560/8000 [00:15<11:43, 6.31it/s, lr: 6.0e-05 loss: 1.629e-01] jess: 45%|████▍ | 3561/8000 [00:15<17:39, 4.19it/s, lr: 6.0e-05 loss: 1.629e-01] jess: 45%|████▍ | 3561/8000 [00:15<17:39, 4.19it/s, lr: 6.0e-05 loss: 1.214e-01] jess: 45%|████▍ | 3562/8000 [00:15<12:50, 5.76it/s, lr: 6.0e-05 loss: 1.214e-01] jess: 45%|████▍ | 3562/8000 [00:15<12:50, 5.76it/s, lr: 6.0e-05 loss: 1.596e-02] jess: 45%|████▍ | 3563/8000 [00:15<17:47, 4.16it/s, lr: 6.0e-05 loss: 1.596e-02] jess: 45%|████▍ | 3563/8000 [00:15<17:47, 4.16it/s, lr: 6.0e-05 loss: 1.401e-01] jess: 45%|████▍ | 3564/8000 [00:15<12:52, 5.74it/s, lr: 6.0e-05 loss: 1.401e-01] jess: 45%|████▍ | 3564/8000 [00:16<12:52, 5.74it/s, lr: 6.0e-05 loss: 1.089e-01] jess: 45%|████▍ | 3565/8000 [00:16<11:22, 6.49it/s, lr: 6.0e-05 loss: 1.089e-01] jess: 45%|████▍ | 3565/8000 [00:16<11:22, 6.49it/s, lr: 6.0e-05 loss: 1.139e-01] jess: 45%|████▍ | 3566/8000 [00:16<10:53, 6.78it/s, lr: 6.0e-05 loss: 1.139e-01] jess: 45%|████▍ | 3566/8000 [00:16<10:53, 6.78it/s, lr: 6.0e-05 loss: 1.129e-01] jess: 45%|████▍ | 3567/8000 [00:16<17:17, 4.27it/s, lr: 6.0e-05 loss: 1.129e-01] jess: 45%|████▍ | 3567/8000 [00:16<17:17, 4.27it/s, lr: 6.0e-05 loss: 7.528e-02] jess: 45%|████▍ | 3568/8000 [00:16<12:43, 5.80it/s, lr: 6.0e-05 loss: 7.528e-02] jess: 45%|████▍ | 3568/8000 [00:16<12:43, 5.80it/s, lr: 6.0e-05 loss: 9.837e-02] jess: 45%|████▍ | 3569/8000 [00:16<11:21, 6.50it/s, lr: 6.0e-05 loss: 9.837e-02] jess: 45%|████▍ | 3569/8000 [00:16<11:21, 6.50it/s, lr: 6.0e-05 loss: 7.804e-02] jess: 45%|████▍ | 3570/8000 [00:16<10:53, 6.78it/s, lr: 6.0e-05 loss: 7.804e-02] jess: 45%|████▍ | 3570/8000 [00:17<10:53, 6.78it/s, lr: 6.0e-05 loss: 1.202e-01] jess: 45%|████▍ | 3571/8000 [00:17<17:13, 4.29it/s, lr: 6.0e-05 loss: 1.202e-01] jess: 45%|████▍ | 3571/8000 [00:17<17:13, 4.29it/s, lr: 6.0e-05 loss: 1.205e-01] jess: 45%|████▍ | 3572/8000 [00:17<19:10, 3.85it/s, lr: 6.0e-05 loss: 1.205e-01] jess: 45%|████▍ | 3572/8000 [00:17<19:10, 3.85it/s, lr: 6.0e-05 loss: 8.830e-02] jess: 45%|████▍ | 3573/8000 [00:17<13:45, 5.36it/s, lr: 6.0e-05 loss: 8.830e-02] jess: 45%|████▍ | 3573/8000 [00:17<13:45, 5.36it/s, lr: 6.0e-05 loss: 3.901e-02] jess: 45%|████▍ | 3574/8000 [00:17<11:36, 6.35it/s, lr: 6.0e-05 loss: 3.901e-02] jess: 45%|████▍ | 3574/8000 [00:18<11:36, 6.35it/s, lr: 6.0e-05 loss: 6.714e-02] jess: 45%|████▍ | 3575/8000 [00:18<17:32, 4.20it/s, lr: 6.0e-05 loss: 6.714e-02] jess: 45%|████▍ | 3575/8000 [00:18<17:32, 4.20it/s, lr: 6.0e-05 loss: 1.292e-01] jess: 45%|████▍ | 3576/8000 [00:18<27:23, 2.69it/s, lr: 6.0e-05 loss: 1.292e-01] jess: 45%|████▍ | 3576/8000 [00:18<27:23, 2.69it/s, lr: 6.0e-05 loss: 1.220e-01] jess: 45%|████▍ | 3577/8000 [00:18<22:10, 3.32it/s, lr: 6.0e-05 loss: 1.220e-01] jess: 45%|████▍ | 3577/8000 [00:18<22:10, 3.32it/s, lr: 6.0e-05 loss: 7.833e-02] jess: 45%|████▍ | 3578/8000 [00:18<20:48, 3.54it/s, lr: 6.0e-05 loss: 7.833e-02] jess: 45%|████▍ | 3578/8000 [00:19<20:48, 3.54it/s, lr: 6.0e-05 loss: 6.980e-02] jess: 45%|████▍ | 3579/8000 [00:19<20:14, 3.64it/s, lr: 6.0e-05 loss: 6.980e-02] jess: 45%|████▍ | 3579/8000 [00:19<20:14, 3.64it/s, lr: 6.0e-05 loss: 1.035e-02] jess: 45%|████▍ | 3580/8000 [00:19<13:33, 5.43it/s, lr: 6.0e-05 loss: 1.035e-02] jess: 45%|████▍ | 3580/8000 [00:19<13:33, 5.43it/s, lr: 6.0e-05 loss: 8.641e-02] jess: 45%|████▍ | 3581/8000 [00:19<18:07, 4.06it/s, lr: 6.0e-05 loss: 8.641e-02] jess: 45%|████▍ | 3581/8000 [00:19<18:07, 4.06it/s, lr: 6.0e-05 loss: 2.261e-01] jess: 45%|████▍ | 3582/8000 [00:19<12:56, 5.69it/s, lr: 6.0e-05 loss: 2.261e-01] jess: 45%|████▍ | 3582/8000 [00:19<12:56, 5.69it/s, lr: 6.0e-05 loss: 8.358e-02] jess: 45%|████▍ | 3583/8000 [00:19<11:23, 6.46it/s, lr: 6.0e-05 loss: 8.358e-02] jess: 45%|████▍ | 3583/8000 [00:20<11:23, 6.46it/s, lr: 6.0e-05 loss: 5.145e-02] jess: 45%|████▍ | 3584/8000 [00:20<17:27, 4.22it/s, lr: 6.0e-05 loss: 5.145e-02] jess: 45%|████▍ | 3584/8000 [00:20<17:27, 4.22it/s, lr: 6.0e-05 loss: 6.448e-02] jess: 45%|████▍ | 3585/8000 [00:20<12:48, 5.74it/s, lr: 6.0e-05 loss: 6.448e-02] jess: 45%|████▍ | 3585/8000 [00:20<12:48, 5.74it/s, lr: 6.0e-05 loss: 6.114e-02] jess: 45%|████▍ | 3586/8000 [00:20<17:50, 4.12it/s, lr: 6.0e-05 loss: 6.114e-02] jess: 45%|████▍ | 3586/8000 [00:20<17:50, 4.12it/s, lr: 6.0e-05 loss: 2.017e-01] jess: 45%|████▍ | 3587/8000 [00:20<12:53, 5.70it/s, lr: 6.0e-05 loss: 2.017e-01] jess: 45%|████▍ | 3587/8000 [00:20<12:53, 5.70it/s, lr: 6.0e-05 loss: 1.064e-01] jess: 45%|████▍ | 3588/8000 [00:20<11:19, 6.49it/s, lr: 6.0e-05 loss: 1.064e-01] jess: 45%|████▍ | 3588/8000 [00:21<11:19, 6.49it/s, lr: 6.0e-05 loss: 7.629e-03] jess: 45%|████▍ | 3589/8000 [00:21<10:55, 6.72it/s, lr: 6.0e-05 loss: 7.629e-03] jess: 45%|████▍ | 3589/8000 [00:21<10:55, 6.72it/s, lr: 6.0e-05 loss: 6.969e-02] jess: 45%|████▍ | 3590/8000 [00:21<17:07, 4.29it/s, lr: 6.0e-05 loss: 6.969e-02] jess: 45%|████▍ | 3590/8000 [00:21<17:07, 4.29it/s, lr: 6.0e-05 loss: 8.053e-02] jess: 45%|████▍ | 3591/8000 [00:21<12:41, 5.79it/s, lr: 6.0e-05 loss: 8.053e-02] jess: 45%|████▍ | 3591/8000 [00:21<12:41, 5.79it/s, lr: 6.0e-05 loss: 1.385e-01] jess: 45%|████▍ | 3592/8000 [00:21<17:36, 4.17it/s, lr: 6.0e-05 loss: 1.385e-01] jess: 45%|████▍ | 3592/8000 [00:22<17:36, 4.17it/s, lr: 6.0e-05 loss: 1.354e-01] jess: 45%|████▍ | 3593/8000 [00:22<19:14, 3.82it/s, lr: 6.0e-05 loss: 1.354e-01] jess: 45%|████▍ | 3593/8000 [00:22<19:14, 3.82it/s, lr: 6.0e-05 loss: 7.372e-02] jess: 45%|████▍ | 3594/8000 [00:22<13:19, 5.51it/s, lr: 6.0e-05 loss: 7.372e-02] jess: 45%|████▍ | 3594/8000 [00:22<13:19, 5.51it/s, lr: 6.0e-05 loss: 9.814e-02] jess: 45%|████▍ | 3595/8000 [00:22<17:55, 4.10it/s, lr: 6.0e-05 loss: 9.814e-02] jess: 45%|████▍ | 3595/8000 [00:22<17:55, 4.10it/s, lr: 6.0e-05 loss: 4.086e-02] jess: 45%|████▍ | 3596/8000 [00:22<19:13, 3.82it/s, lr: 6.0e-05 loss: 4.086e-02] jess: 45%|████▍ | 3596/8000 [00:23<19:13, 3.82it/s, lr: 6.0e-05 loss: 1.421e-01] jess: 45%|████▍ | 3597/8000 [00:23<19:46, 3.71it/s, lr: 6.0e-05 loss: 1.421e-01] jess: 45%|████▍ | 3597/8000 [00:23<19:46, 3.71it/s, lr: 6.0e-05 loss: 1.208e-01] jess: 45%|████▍ | 3598/8000 [00:23<19:46, 3.71it/s, lr: 6.0e-05 loss: 1.208e-01] jess: 45%|████▍ | 3598/8000 [00:23<19:46, 3.71it/s, lr: 6.0e-05 loss: 5.748e-02] jess: 45%|████▍ | 3599/8000 [00:23<13:27, 5.45it/s, lr: 6.0e-05 loss: 5.748e-02] jess: 45%|████▍ | 3599/8000 [00:23<13:27, 5.45it/s, lr: 6.0e-05 loss: 6.176e-02] jess: 45%|████▌ | 3600/8000 [00:23<26:25, 2.78it/s, lr: 6.0e-05 loss: 6.176e-02] jess: 45%|████▌ | 3600/8000 [00:24<26:25, 2.78it/s, lr: 6.0e-05 loss: 6.134e-02]
Saving at step 3600
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003600.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.49s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.40s/it]
Generating Images: 100%|██████████| 3/3 [00:04<00:00, 1.36s/it]

jess: 45%|████▌ | 3601/8000 [00:24<26:24, 2.78it/s, lr: 6.0e-05 loss: 1.136e-01] jess: 45%|████▌ | 3602/8000 [00:24<17:33, 4.17it/s, lr: 6.0e-05 loss: 1.136e-01] jess: 45%|████▌ | 3602/8000 [00:24<17:33, 4.17it/s, lr: 6.0e-05 loss: 7.440e-03] jess: 45%|████▌ | 3603/8000 [00:24<18:47, 3.90it/s, lr: 6.0e-05 loss: 7.440e-03] jess: 45%|████▌ | 3603/8000 [00:24<18:47, 3.90it/s, lr: 6.0e-05 loss: 8.557e-02] jess: 45%|████▌ | 3604/8000 [00:24<13:28, 5.43it/s, lr: 6.0e-05 loss: 8.557e-02] jess: 45%|████▌ | 3604/8000 [00:25<13:28, 5.43it/s, lr: 6.0e-05 loss: 3.883e-02] jess: 45%|████▌ | 3605/8000 [00:25<16:28, 4.45it/s, lr: 6.0e-05 loss: 3.883e-02] jess: 45%|████▌ | 3605/8000 [00:25<16:28, 4.45it/s, lr: 6.0e-05 loss: 1.541e-02] jess: 45%|████▌ | 3606/8000 [00:25<18:31, 3.95it/s, lr: 6.0e-05 loss: 1.541e-02] jess: 45%|████▌ | 3606/8000 [00:25<18:31, 3.95it/s, lr: 6.0e-05 loss: 1.118e-01] jess: 45%|████▌ | 3607/8000 [00:25<19:27, 3.76it/s, lr: 6.0e-05 loss: 1.118e-01] jess: 45%|████▌ | 3607/8000 [00:25<19:27, 3.76it/s, lr: 6.0e-05 loss: 1.358e-01] jess: 45%|████▌ | 3608/8000 [00:25<19:45, 3.71it/s, lr: 6.0e-05 loss: 1.358e-01] jess: 45%|████▌ | 3608/8000 [00:26<19:45, 3.71it/s, lr: 6.0e-05 loss: 1.483e-01] jess: 45%|████▌ | 3609/8000 [00:26<20:24, 3.59it/s, lr: 6.0e-05 loss: 1.483e-01] jess: 45%|████▌ | 3609/8000 [00:26<20:24, 3.59it/s, lr: 6.0e-05 loss: 8.279e-02] jess: 45%|████▌ | 3610/8000 [00:26<13:31, 5.41it/s, lr: 6.0e-05 loss: 8.279e-02] jess: 45%|████▌ | 3610/8000 [00:26<13:31, 5.41it/s, lr: 6.0e-05 loss: 1.175e-01] jess: 45%|████▌ | 3611/8000 [00:26<11:29, 6.37it/s, lr: 6.0e-05 loss: 1.175e-01] jess: 45%|████▌ | 3611/8000 [00:26<11:29, 6.37it/s, lr: 6.0e-05 loss: 1.378e-01] jess: 45%|████▌ | 3612/8000 [00:26<17:18, 4.23it/s, lr: 6.0e-05 loss: 1.378e-01] jess: 45%|████▌ | 3612/8000 [00:27<17:18, 4.23it/s, lr: 6.0e-05 loss: 7.743e-02] jess: 45%|████▌ | 3613/8000 [00:27<19:06, 3.83it/s, lr: 6.0e-05 loss: 7.743e-02] jess: 45%|████▌ | 3613/8000 [00:27<19:06, 3.83it/s, lr: 6.0e-05 loss: 1.482e-01] jess: 45%|████▌ | 3614/8000 [00:27<19:29, 3.75it/s, lr: 6.0e-05 loss: 1.482e-01] jess: 45%|████▌ | 3614/8000 [00:27<19:29, 3.75it/s, lr: 6.0e-05 loss: 6.582e-02] jess: 45%|████▌ | 3615/8000 [00:27<13:17, 5.50it/s, lr: 6.0e-05 loss: 6.582e-02] jess: 45%|████▌ | 3615/8000 [00:27<13:17, 5.50it/s, lr: 6.0e-05 loss: 5.044e-02] jess: 45%|████▌ | 3616/8000 [00:27<17:52, 4.09it/s, lr: 6.0e-05 loss: 5.044e-02] jess: 45%|████▌ | 3616/8000 [00:27<17:52, 4.09it/s, lr: 6.0e-05 loss: 1.090e-01] jess: 45%|████▌ | 3617/8000 [00:27<19:15, 3.79it/s, lr: 6.0e-05 loss: 1.090e-01] jess: 45%|████▌ | 3617/8000 [00:28<19:15, 3.79it/s, lr: 6.0e-05 loss: 1.254e-01] jess: 45%|████▌ | 3618/8000 [00:28<13:13, 5.52it/s, lr: 6.0e-05 loss: 1.254e-01] jess: 45%|████▌ | 3618/8000 [00:28<13:13, 5.52it/s, lr: 6.0e-05 loss: 9.780e-03] jess: 45%|████▌ | 3619/8000 [00:28<17:41, 4.13it/s, lr: 6.0e-05 loss: 9.780e-03] jess: 45%|████▌ | 3619/8000 [00:28<17:41, 4.13it/s, lr: 6.0e-05 loss: 8.022e-02] jess: 45%|████▌ | 3620/8000 [00:28<19:15, 3.79it/s, lr: 6.0e-05 loss: 8.022e-02] jess: 45%|████▌ | 3620/8000 [00:28<19:15, 3.79it/s, lr: 6.0e-05 loss: 8.318e-02] jess: 45%|████▌ | 3621/8000 [00:28<19:39, 3.71it/s, lr: 6.0e-05 loss: 8.318e-02] jess: 45%|████▌ | 3621/8000 [00:29<19:39, 3.71it/s, lr: 6.0e-05 loss: 1.628e-01] jess: 45%|████▌ | 3622/8000 [00:29<13:31, 5.40it/s, lr: 6.0e-05 loss: 1.628e-01] jess: 45%|████▌ | 3622/8000 [00:29<13:31, 5.40it/s, lr: 6.0e-05 loss: 7.257e-02] jess: 45%|████▌ | 3623/8000 [00:29<11:32, 6.32it/s, lr: 6.0e-05 loss: 7.257e-02] jess: 45%|████▌ | 3623/8000 [00:29<11:32, 6.32it/s, lr: 6.0e-05 loss: 5.999e-02] jess: 45%|████▌ | 3624/8000 [00:29<17:00, 4.29it/s, lr: 6.0e-05 loss: 5.999e-02] jess: 45%|████▌ | 3624/8000 [00:29<17:00, 4.29it/s, lr: 6.0e-05 loss: 7.356e-02] jess: 45%|████▌ | 3625/8000 [00:29<12:40, 5.75it/s, lr: 6.0e-05 loss: 7.356e-02] jess: 45%|████▌ | 3625/8000 [00:30<12:40, 5.75it/s, lr: 6.0e-05 loss: 5.448e-02] jess: 45%|████▌ | 3626/8000 [00:30<26:05, 2.79it/s, lr: 6.0e-05 loss: 5.448e-02] jess: 45%|████▌ | 3626/8000 [00:30<26:05, 2.79it/s, lr: 6.0e-05 loss: 9.173e-02] jess: 45%|████▌ | 3627/8000 [00:30<15:20, 4.75it/s, lr: 6.0e-05 loss: 9.173e-02] jess: 45%|████▌ | 3627/8000 [00:30<15:20, 4.75it/s, lr: 6.0e-05 loss: 2.877e-02] jess: 45%|████▌ | 3628/8000 [00:30<11:59, 6.08it/s, lr: 6.0e-05 loss: 2.877e-02] jess: 45%|████▌ | 3628/8000 [00:30<11:59, 6.08it/s, lr: 6.0e-05 loss: 2.059e-01] jess: 45%|████▌ | 3629/8000 [00:30<10:56, 6.65it/s, lr: 6.0e-05 loss: 2.059e-01] jess: 45%|████▌ | 3629/8000 [00:30<10:56, 6.65it/s, lr: 6.0e-05 loss: 9.011e-02] jess: 45%|████▌ | 3630/8000 [00:30<10:38, 6.85it/s, lr: 6.0e-05 loss: 9.011e-02] jess: 45%|████▌ | 3630/8000 [00:30<10:38, 6.85it/s, lr: 6.0e-05 loss: 1.359e-01] jess: 45%|████▌ | 3631/8000 [00:30<16:54, 4.31it/s, lr: 6.0e-05 loss: 1.359e-01] jess: 45%|████▌ | 3631/8000 [00:31<16:54, 4.31it/s, lr: 6.0e-05 loss: 1.795e-02] jess: 45%|████▌ | 3632/8000 [00:31<17:24, 4.18it/s, lr: 6.0e-05 loss: 1.795e-02] jess: 45%|████▌ | 3632/8000 [00:31<17:24, 4.18it/s, lr: 6.0e-05 loss: 1.920e-01] jess: 45%|████▌ | 3633/8000 [00:31<18:55, 3.85it/s, lr: 6.0e-05 loss: 1.920e-01] jess: 45%|████▌ | 3633/8000 [00:31<18:55, 3.85it/s, lr: 6.0e-05 loss: 1.128e-01] jess: 45%|████▌ | 3634/8000 [00:31<13:01, 5.58it/s, lr: 6.0e-05 loss: 1.128e-01] jess: 45%|████▌ | 3634/8000 [00:31<13:01, 5.58it/s, lr: 6.0e-05 loss: 1.004e-01] jess: 45%|████▌ | 3635/8000 [00:31<17:42, 4.11it/s, lr: 6.0e-05 loss: 1.004e-01] jess: 45%|████▌ | 3635/8000 [00:32<17:42, 4.11it/s, lr: 6.0e-05 loss: 1.176e-01] jess: 45%|████▌ | 3636/8000 [00:32<19:06, 3.81it/s, lr: 6.0e-05 loss: 1.176e-01] jess: 45%|████▌ | 3636/8000 [00:32<19:06, 3.81it/s, lr: 6.0e-05 loss: 6.179e-02] jess: 45%|████▌ | 3637/8000 [00:32<13:03, 5.57it/s, lr: 6.0e-05 loss: 6.179e-02] jess: 45%|████▌ | 3637/8000 [00:32<13:03, 5.57it/s, lr: 6.0e-05 loss: 6.117e-02] jess: 45%|████▌ | 3638/8000 [00:32<17:33, 4.14it/s, lr: 6.0e-05 loss: 6.117e-02] jess: 45%|████▌ | 3638/8000 [00:32<17:33, 4.14it/s, lr: 6.0e-05 loss: 8.355e-02] jess: 45%|████▌ | 3639/8000 [00:32<19:01, 3.82it/s, lr: 6.0e-05 loss: 8.355e-02] jess: 45%|████▌ | 3639/8000 [00:32<19:01, 3.82it/s, lr: 6.0e-05 loss: 7.313e-03] jess: 46%|████▌ | 3640/8000 [00:32<13:06, 5.54it/s, lr: 6.0e-05 loss: 7.313e-03] jess: 46%|████▌ | 3640/8000 [00:33<13:06, 5.54it/s, lr: 6.0e-05 loss: 2.023e-01] jess: 46%|████▌ | 3641/8000 [00:33<17:52, 4.07it/s, lr: 6.0e-05 loss: 2.023e-01] jess: 46%|████▌ | 3641/8000 [00:33<17:52, 4.07it/s, lr: 6.0e-05 loss: 1.367e-01] jess: 46%|████▌ | 3642/8000 [00:33<12:42, 5.72it/s, lr: 6.0e-05 loss: 1.367e-01] jess: 46%|████▌ | 3642/8000 [00:33<12:42, 5.72it/s, lr: 6.0e-05 loss: 1.026e-01] jess: 46%|████▌ | 3643/8000 [00:33<17:26, 4.16it/s, lr: 6.0e-05 loss: 1.026e-01] jess: 46%|████▌ | 3643/8000 [00:33<17:26, 4.16it/s, lr: 6.0e-05 loss: 5.749e-03] jess: 46%|████▌ | 3644/8000 [00:33<12:38, 5.75it/s, lr: 6.0e-05 loss: 5.749e-03] jess: 46%|████▌ | 3644/8000 [00:33<12:38, 5.75it/s, lr: 6.0e-05 loss: 8.298e-02] jess: 46%|████▌ | 3645/8000 [00:33<11:40, 6.22it/s, lr: 6.0e-05 loss: 8.298e-02] jess: 46%|████▌ | 3645/8000 [00:34<11:40, 6.22it/s, lr: 6.0e-05 loss: 2.261e-02] jess: 46%|████▌ | 3646/8000 [00:34<15:48, 4.59it/s, lr: 6.0e-05 loss: 2.261e-02] jess: 46%|████▌ | 3646/8000 [00:34<15:48, 4.59it/s, lr: 6.0e-05 loss: 3.393e-02] jess: 46%|████▌ | 3647/8000 [00:34<12:13, 5.94it/s, lr: 6.0e-05 loss: 3.393e-02] jess: 46%|████▌ | 3647/8000 [00:34<12:13, 5.94it/s, lr: 6.0e-05 loss: 6.136e-02] jess: 46%|████▌ | 3648/8000 [00:34<17:06, 4.24it/s, lr: 6.0e-05 loss: 6.136e-02] jess: 46%|████▌ | 3648/8000 [00:34<17:06, 4.24it/s, lr: 6.0e-05 loss: 8.644e-03] jess: 46%|████▌ | 3649/8000 [00:34<18:44, 3.87it/s, lr: 6.0e-05 loss: 8.644e-03] jess: 46%|████▌ | 3649/8000 [00:35<18:44, 3.87it/s, lr: 6.0e-05 loss: 2.162e-01] jess: 46%|████▌ | 3650/8000 [00:35<27:58, 2.59it/s, lr: 6.0e-05 loss: 2.162e-01] jess: 46%|████▌ | 3650/8000 [00:35<27:58, 2.59it/s, lr: 6.0e-05 loss: 1.253e-01]
Saving at step 3650
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003650.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 46%|████▌ | 3651/8000 [00:35<27:58, 2.59it/s, lr: 6.0e-05 loss: 1.233e-01] jess: 46%|████▌ | 3652/8000 [00:35<21:16, 3.41it/s, lr: 6.0e-05 loss: 1.233e-01] jess: 46%|████▌ | 3652/8000 [00:36<21:16, 3.41it/s, lr: 6.0e-05 loss: 3.297e-02] jess: 46%|████▌ | 3653/8000 [00:36<15:07, 4.79it/s, lr: 6.0e-05 loss: 3.297e-02] jess: 46%|████▌ | 3653/8000 [00:36<15:07, 4.79it/s, lr: 6.0e-05 loss: 3.676e-01] jess: 46%|████▌ | 3654/8000 [00:36<18:07, 4.00it/s, lr: 6.0e-05 loss: 3.676e-01] jess: 46%|████▌ | 3654/8000 [00:36<18:07, 4.00it/s, lr: 6.0e-05 loss: 2.418e-01] jess: 46%|████▌ | 3655/8000 [00:36<19:07, 3.79it/s, lr: 6.0e-05 loss: 2.418e-01] jess: 46%|████▌ | 3655/8000 [00:36<19:07, 3.79it/s, lr: 6.0e-05 loss: 1.201e-01] jess: 46%|████▌ | 3656/8000 [00:36<13:14, 5.47it/s, lr: 6.0e-05 loss: 1.201e-01] jess: 46%|████▌ | 3656/8000 [00:37<13:14, 5.47it/s, lr: 6.0e-05 loss: 2.339e-01] jess: 46%|████▌ | 3657/8000 [00:37<17:33, 4.12it/s, lr: 6.0e-05 loss: 2.339e-01] jess: 46%|████▌ | 3657/8000 [00:37<17:33, 4.12it/s, lr: 6.0e-05 loss: 1.111e-01] jess: 46%|████▌ | 3658/8000 [00:37<18:59, 3.81it/s, lr: 6.0e-05 loss: 1.111e-01] jess: 46%|████▌ | 3658/8000 [00:37<18:59, 3.81it/s, lr: 6.0e-05 loss: 1.926e-01] jess: 46%|████▌ | 3659/8000 [00:37<13:03, 5.54it/s, lr: 6.0e-05 loss: 1.926e-01] jess: 46%|████▌ | 3659/8000 [00:37<13:03, 5.54it/s, lr: 6.0e-05 loss: 1.122e-01] jess: 46%|████▌ | 3660/8000 [00:37<17:40, 4.09it/s, lr: 6.0e-05 loss: 1.122e-01] jess: 46%|████▌ | 3660/8000 [00:37<17:40, 4.09it/s, lr: 6.0e-05 loss: 3.202e-02] jess: 46%|████▌ | 3661/8000 [00:37<12:44, 5.67it/s, lr: 6.0e-05 loss: 3.202e-02] jess: 46%|████▌ | 3661/8000 [00:38<12:44, 5.67it/s, lr: 6.0e-05 loss: 1.690e-01] jess: 46%|████▌ | 3662/8000 [00:38<11:14, 6.43it/s, lr: 6.0e-05 loss: 1.690e-01] jess: 46%|████▌ | 3662/8000 [00:38<11:14, 6.43it/s, lr: 6.0e-05 loss: 5.447e-02] jess: 46%|████▌ | 3663/8000 [00:38<17:06, 4.23it/s, lr: 6.0e-05 loss: 5.447e-02] jess: 46%|████▌ | 3663/8000 [00:38<17:06, 4.23it/s, lr: 6.0e-05 loss: 9.358e-03] jess: 46%|████▌ | 3664/8000 [00:38<12:32, 5.76it/s, lr: 6.0e-05 loss: 9.358e-03] jess: 46%|████▌ | 3664/8000 [00:38<12:32, 5.76it/s, lr: 6.0e-05 loss: 3.269e-02] jess: 46%|████▌ | 3665/8000 [00:38<11:15, 6.42it/s, lr: 6.0e-05 loss: 3.269e-02] jess: 46%|████▌ | 3665/8000 [00:38<11:15, 6.42it/s, lr: 6.0e-05 loss: 5.898e-02] jess: 46%|████▌ | 3666/8000 [00:38<17:03, 4.23it/s, lr: 6.0e-05 loss: 5.898e-02] jess: 46%|████▌ | 3666/8000 [00:39<17:03, 4.23it/s, lr: 6.0e-05 loss: 7.315e-02] jess: 46%|████▌ | 3667/8000 [00:39<12:26, 5.80it/s, lr: 6.0e-05 loss: 7.315e-02] jess: 46%|████▌ | 3667/8000 [00:39<12:26, 5.80it/s, lr: 6.0e-05 loss: 2.796e-01] jess: 46%|████▌ | 3668/8000 [00:39<17:18, 4.17it/s, lr: 6.0e-05 loss: 2.796e-01] jess: 46%|████▌ | 3668/8000 [00:39<17:18, 4.17it/s, lr: 6.0e-05 loss: 1.819e-01] jess: 46%|████▌ | 3669/8000 [00:39<18:53, 3.82it/s, lr: 6.0e-05 loss: 1.819e-01] jess: 46%|████▌ | 3669/8000 [00:39<18:53, 3.82it/s, lr: 6.0e-05 loss: 2.006e-01] jess: 46%|████▌ | 3670/8000 [00:39<13:01, 5.54it/s, lr: 6.0e-05 loss: 2.006e-01] jess: 46%|████▌ | 3670/8000 [00:39<13:01, 5.54it/s, lr: 6.0e-05 loss: 2.035e-01] jess: 46%|████▌ | 3671/8000 [00:39<11:19, 6.37it/s, lr: 6.0e-05 loss: 2.035e-01] jess: 46%|████▌ | 3671/8000 [00:40<11:19, 6.37it/s, lr: 6.0e-05 loss: 1.393e-01] jess: 46%|████▌ | 3672/8000 [00:40<16:59, 4.25it/s, lr: 6.0e-05 loss: 1.393e-01] jess: 46%|████▌ | 3672/8000 [00:40<16:59, 4.25it/s, lr: 6.0e-05 loss: 1.593e-01] jess: 46%|████▌ | 3673/8000 [00:40<12:26, 5.79it/s, lr: 6.0e-05 loss: 1.593e-01] jess: 46%|████▌ | 3673/8000 [00:40<12:26, 5.79it/s, lr: 6.0e-05 loss: 1.171e-01] jess: 46%|████▌ | 3674/8000 [00:40<11:04, 6.51it/s, lr: 6.0e-05 loss: 1.171e-01] jess: 46%|████▌ | 3674/8000 [00:40<11:04, 6.51it/s, lr: 6.0e-05 loss: 5.174e-02] jess: 46%|████▌ | 3675/8000 [00:40<17:04, 4.22it/s, lr: 6.0e-05 loss: 5.174e-02] jess: 46%|████▌ | 3675/8000 [00:41<17:04, 4.22it/s, lr: 6.0e-05 loss: 3.294e-01] jess: 46%|████▌ | 3676/8000 [00:41<32:38, 2.21it/s, lr: 6.0e-05 loss: 3.294e-01] jess: 46%|████▌ | 3676/8000 [00:41<32:38, 2.21it/s, lr: 6.0e-05 loss: 4.290e-02] jess: 46%|████▌ | 3677/8000 [00:41<23:39, 3.04it/s, lr: 6.0e-05 loss: 4.290e-02] jess: 46%|████▌ | 3677/8000 [00:41<23:39, 3.04it/s, lr: 6.0e-05 loss: 6.654e-02] jess: 46%|████▌ | 3678/8000 [00:41<20:50, 3.46it/s, lr: 6.0e-05 loss: 6.654e-02] jess: 46%|████▌ | 3678/8000 [00:41<20:50, 3.46it/s, lr: 6.0e-05 loss: 1.061e-01] jess: 46%|████▌ | 3679/8000 [00:41<13:42, 5.25it/s, lr: 6.0e-05 loss: 1.061e-01] jess: 46%|████▌ | 3679/8000 [00:42<13:42, 5.25it/s, lr: 6.0e-05 loss: 1.885e-01] jess: 46%|████▌ | 3680/8000 [00:42<17:44, 4.06it/s, lr: 6.0e-05 loss: 1.885e-01] jess: 46%|████▌ | 3680/8000 [00:42<17:44, 4.06it/s, lr: 6.0e-05 loss: 1.312e-01] jess: 46%|████▌ | 3681/8000 [00:42<13:26, 5.35it/s, lr: 6.0e-05 loss: 1.312e-01] jess: 46%|████▌ | 3681/8000 [00:42<13:26, 5.35it/s, lr: 6.0e-05 loss: 1.427e-01] jess: 46%|████▌ | 3682/8000 [00:42<11:14, 6.40it/s, lr: 6.0e-05 loss: 1.427e-01] jess: 46%|████▌ | 3682/8000 [00:42<11:14, 6.40it/s, lr: 6.0e-05 loss: 6.023e-02] jess: 46%|████▌ | 3683/8000 [00:42<10:42, 6.72it/s, lr: 6.0e-05 loss: 6.023e-02] jess: 46%|████▌ | 3683/8000 [00:42<10:42, 6.72it/s, lr: 6.0e-05 loss: 2.742e-02] jess: 46%|████▌ | 3684/8000 [00:42<10:34, 6.80it/s, lr: 6.0e-05 loss: 2.742e-02] jess: 46%|████▌ | 3684/8000 [00:42<10:34, 6.80it/s, lr: 6.0e-05 loss: 1.202e-01] jess: 46%|████▌ | 3685/8000 [00:42<10:30, 6.84it/s, lr: 6.0e-05 loss: 1.202e-01] jess: 46%|████▌ | 3685/8000 [00:43<10:30, 6.84it/s, lr: 6.0e-05 loss: 9.568e-02] jess: 46%|████▌ | 3686/8000 [00:43<10:27, 6.88it/s, lr: 6.0e-05 loss: 9.568e-02] jess: 46%|████▌ | 3686/8000 [00:43<10:27, 6.88it/s, lr: 6.0e-05 loss: 1.227e-01] jess: 46%|████▌ | 3687/8000 [00:43<10:28, 6.86it/s, lr: 6.0e-05 loss: 1.227e-01] jess: 46%|████▌ | 3687/8000 [00:43<10:28, 6.86it/s, lr: 6.0e-05 loss: 2.617e-01] jess: 46%|████▌ | 3688/8000 [00:43<16:46, 4.28it/s, lr: 6.0e-05 loss: 2.617e-01] jess: 46%|████▌ | 3688/8000 [00:43<16:46, 4.28it/s, lr: 6.0e-05 loss: 1.709e-01] jess: 46%|████▌ | 3689/8000 [00:43<18:40, 3.85it/s, lr: 6.0e-05 loss: 1.709e-01] jess: 46%|████▌ | 3689/8000 [00:43<18:40, 3.85it/s, lr: 6.0e-05 loss: 4.848e-02] jess: 46%|████▌ | 3690/8000 [00:43<12:56, 5.55it/s, lr: 6.0e-05 loss: 4.848e-02] jess: 46%|████▌ | 3690/8000 [00:44<12:56, 5.55it/s, lr: 6.0e-05 loss: 5.296e-03] jess: 46%|████▌ | 3691/8000 [00:44<11:09, 6.43it/s, lr: 6.0e-05 loss: 5.296e-03] jess: 46%|████▌ | 3691/8000 [00:44<11:09, 6.43it/s, lr: 6.0e-05 loss: 3.401e-01] jess: 46%|████▌ | 3692/8000 [00:44<10:39, 6.74it/s, lr: 6.0e-05 loss: 3.401e-01] jess: 46%|████▌ | 3692/8000 [00:44<10:39, 6.74it/s, lr: 6.0e-05 loss: 1.819e-01] jess: 46%|████▌ | 3693/8000 [00:44<10:48, 6.64it/s, lr: 6.0e-05 loss: 1.819e-01] jess: 46%|████▌ | 3693/8000 [00:44<10:48, 6.64it/s, lr: 6.0e-05 loss: 6.768e-02] jess: 46%|████▌ | 3694/8000 [00:44<16:54, 4.24it/s, lr: 6.0e-05 loss: 6.768e-02] jess: 46%|████▌ | 3694/8000 [00:44<16:54, 4.24it/s, lr: 6.0e-05 loss: 7.352e-02] jess: 46%|████▌ | 3695/8000 [00:44<12:24, 5.79it/s, lr: 6.0e-05 loss: 7.352e-02] jess: 46%|████▌ | 3695/8000 [00:45<12:24, 5.79it/s, lr: 6.0e-05 loss: 1.410e-01] jess: 46%|████▌ | 3696/8000 [00:45<17:20, 4.14it/s, lr: 6.0e-05 loss: 1.410e-01] jess: 46%|████▌ | 3696/8000 [00:45<17:20, 4.14it/s, lr: 6.0e-05 loss: 1.070e-02] jess: 46%|████▌ | 3697/8000 [00:45<17:27, 4.11it/s, lr: 6.0e-05 loss: 1.070e-02] jess: 46%|████▌ | 3697/8000 [00:45<17:27, 4.11it/s, lr: 6.0e-05 loss: 1.047e-01] jess: 46%|████▌ | 3698/8000 [00:45<12:27, 5.76it/s, lr: 6.0e-05 loss: 1.047e-01] jess: 46%|████▌ | 3698/8000 [00:45<12:27, 5.76it/s, lr: 6.0e-05 loss: 5.136e-02] jess: 46%|████▌ | 3699/8000 [00:45<17:15, 4.16it/s, lr: 6.0e-05 loss: 5.136e-02] jess: 46%|████▌ | 3699/8000 [00:46<17:15, 4.16it/s, lr: 6.0e-05 loss: 1.587e-01] jess: 46%|████▋ | 3700/8000 [00:46<27:32, 2.60it/s, lr: 6.0e-05 loss: 1.587e-01] jess: 46%|████▋ | 3700/8000 [00:46<27:32, 2.60it/s, lr: 6.0e-05 loss: 1.329e-01]
Saving at step 3700
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003700.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.35s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.34s/it]
Generating Images: 100%|██████████| 3/3 [00:04<00:00, 1.34s/it]

jess: 46%|████▋ | 3701/8000 [00:46<27:31, 2.60it/s, lr: 6.0e-05 loss: 5.475e-02] jess: 46%|████▋ | 3702/8000 [00:46<17:40, 4.05it/s, lr: 6.0e-05 loss: 5.475e-02] jess: 46%|████▋ | 3702/8000 [00:47<17:40, 4.05it/s, lr: 6.0e-05 loss: 7.261e-02] jess: 46%|████▋ | 3703/8000 [00:47<18:36, 3.85it/s, lr: 6.0e-05 loss: 7.261e-02] jess: 46%|████▋ | 3703/8000 [00:47<18:36, 3.85it/s, lr: 6.0e-05 loss: 2.613e-01] jess: 46%|████▋ | 3704/8000 [00:47<19:08, 3.74it/s, lr: 6.0e-05 loss: 2.613e-01] jess: 46%|████▋ | 3704/8000 [00:47<19:08, 3.74it/s, lr: 6.0e-05 loss: 9.806e-02] jess: 46%|████▋ | 3705/8000 [00:47<19:20, 3.70it/s, lr: 6.0e-05 loss: 9.806e-02] jess: 46%|████▋ | 3705/8000 [00:47<19:20, 3.70it/s, lr: 6.0e-05 loss: 1.115e-01] jess: 46%|████▋ | 3706/8000 [00:47<19:19, 3.70it/s, lr: 6.0e-05 loss: 1.115e-01] jess: 46%|████▋ | 3706/8000 [00:48<19:19, 3.70it/s, lr: 6.0e-05 loss: 2.499e-01] jess: 46%|████▋ | 3707/8000 [00:48<19:12, 3.72it/s, lr: 6.0e-05 loss: 2.499e-01] jess: 46%|████▋ | 3707/8000 [00:48<19:12, 3.72it/s, lr: 6.0e-05 loss: 4.008e-02] jess: 46%|████▋ | 3708/8000 [00:48<19:16, 3.71it/s, lr: 6.0e-05 loss: 4.008e-02] jess: 46%|████▋ | 3708/8000 [00:48<19:16, 3.71it/s, lr: 6.0e-05 loss: 1.393e-01] jess: 46%|████▋ | 3709/8000 [00:48<19:26, 3.68it/s, lr: 6.0e-05 loss: 1.393e-01] jess: 46%|████▋ | 3709/8000 [00:49<19:26, 3.68it/s, lr: 6.0e-05 loss: 1.416e-01] jess: 46%|████▋ | 3710/8000 [00:49<19:26, 3.68it/s, lr: 6.0e-05 loss: 1.416e-01] jess: 46%|████▋ | 3710/8000 [00:49<19:26, 3.68it/s, lr: 6.0e-05 loss: 2.065e-02] jess: 46%|████▋ | 3711/8000 [00:49<13:07, 5.45it/s, lr: 6.0e-05 loss: 2.065e-02] jess: 46%|████▋ | 3711/8000 [00:49<13:07, 5.45it/s, lr: 6.0e-05 loss: 1.217e-01] jess: 46%|████▋ | 3712/8000 [00:49<17:27, 4.09it/s, lr: 6.0e-05 loss: 1.217e-01] jess: 46%|████▋ | 3712/8000 [00:49<17:27, 4.09it/s, lr: 6.0e-05 loss: 1.015e-02] jess: 46%|████▋ | 3713/8000 [00:49<18:29, 3.86it/s, lr: 6.0e-05 loss: 1.015e-02] jess: 46%|████▋ | 3713/8000 [00:49<18:29, 3.86it/s, lr: 6.0e-05 loss: 1.718e-01] jess: 46%|████▋ | 3714/8000 [00:49<12:51, 5.56it/s, lr: 6.0e-05 loss: 1.718e-01] jess: 46%|████▋ | 3714/8000 [00:50<12:51, 5.56it/s, lr: 6.0e-05 loss: 8.474e-02] jess: 46%|████▋ | 3715/8000 [00:50<17:23, 4.11it/s, lr: 6.0e-05 loss: 8.474e-02] jess: 46%|████▋ | 3715/8000 [00:50<17:23, 4.11it/s, lr: 6.0e-05 loss: 1.014e-01] jess: 46%|████▋ | 3716/8000 [00:50<18:43, 3.81it/s, lr: 6.0e-05 loss: 1.014e-01] jess: 46%|████▋ | 3716/8000 [00:50<18:43, 3.81it/s, lr: 6.0e-05 loss: 9.235e-02] jess: 46%|████▋ | 3717/8000 [00:50<13:24, 5.33it/s, lr: 6.0e-05 loss: 9.235e-02] jess: 46%|████▋ | 3717/8000 [00:50<13:24, 5.33it/s, lr: 6.0e-05 loss: 1.664e-01] jess: 46%|████▋ | 3718/8000 [00:50<17:35, 4.06it/s, lr: 6.0e-05 loss: 1.664e-01] jess: 46%|████▋ | 3718/8000 [00:51<17:35, 4.06it/s, lr: 6.0e-05 loss: 9.369e-02] jess: 46%|████▋ | 3719/8000 [00:51<18:53, 3.78it/s, lr: 6.0e-05 loss: 9.369e-02] jess: 46%|████▋ | 3719/8000 [00:51<18:53, 3.78it/s, lr: 6.0e-05 loss: 9.048e-02] jess: 46%|████▋ | 3720/8000 [00:51<19:11, 3.72it/s, lr: 6.0e-05 loss: 9.048e-02] jess: 46%|████▋ | 3720/8000 [00:51<19:11, 3.72it/s, lr: 6.0e-05 loss: 1.781e-01] jess: 47%|████▋ | 3721/8000 [00:51<19:12, 3.71it/s, lr: 6.0e-05 loss: 1.781e-01] jess: 47%|████▋ | 3721/8000 [00:51<19:12, 3.71it/s, lr: 6.0e-05 loss: 4.009e-02] jess: 47%|████▋ | 3722/8000 [00:51<19:11, 3.71it/s, lr: 6.0e-05 loss: 4.009e-02] jess: 47%|████▋ | 3722/8000 [00:52<19:11, 3.71it/s, lr: 6.0e-05 loss: 1.637e-02] jess: 47%|████▋ | 3723/8000 [00:52<13:01, 5.47it/s, lr: 6.0e-05 loss: 1.637e-02] jess: 47%|████▋ | 3723/8000 [00:52<13:01, 5.47it/s, lr: 6.0e-05 loss: 1.916e-02] jess: 47%|████▋ | 3724/8000 [00:52<17:18, 4.12it/s, lr: 6.0e-05 loss: 1.916e-02] jess: 47%|████▋ | 3724/8000 [00:52<17:18, 4.12it/s, lr: 6.0e-05 loss: 7.457e-02] jess: 47%|████▋ | 3725/8000 [00:52<12:25, 5.74it/s, lr: 6.0e-05 loss: 7.457e-02] jess: 47%|████▋ | 3725/8000 [00:52<12:25, 5.74it/s, lr: 6.0e-05 loss: 6.227e-02] jess: 47%|████▋ | 3726/8000 [00:52<25:27, 2.80it/s, lr: 6.0e-05 loss: 6.227e-02] jess: 47%|████▋ | 3726/8000 [00:53<25:27, 2.80it/s, lr: 6.0e-05 loss: 8.677e-02] jess: 47%|████▋ | 3727/8000 [00:53<15:00, 4.74it/s, lr: 6.0e-05 loss: 8.677e-02] jess: 47%|████▋ | 3727/8000 [00:53<15:00, 4.74it/s, lr: 6.0e-05 loss: 7.506e-02] jess: 47%|████▋ | 3728/8000 [00:53<11:45, 6.06it/s, lr: 6.0e-05 loss: 7.506e-02] jess: 47%|████▋ | 3728/8000 [00:53<11:45, 6.06it/s, lr: 6.0e-05 loss: 1.883e-01] jess: 47%|████▋ | 3729/8000 [00:53<10:47, 6.60it/s, lr: 6.0e-05 loss: 1.883e-01] jess: 47%|████▋ | 3729/8000 [00:53<10:47, 6.60it/s, lr: 6.0e-05 loss: 1.093e-01] jess: 47%|████▋ | 3730/8000 [00:53<10:23, 6.85it/s, lr: 6.0e-05 loss: 1.093e-01] jess: 47%|████▋ | 3730/8000 [00:53<10:23, 6.85it/s, lr: 6.0e-05 loss: 1.494e-02] jess: 47%|████▋ | 3731/8000 [00:53<15:17, 4.65it/s, lr: 6.0e-05 loss: 1.494e-02] jess: 47%|████▋ | 3731/8000 [00:54<15:17, 4.65it/s, lr: 6.0e-05 loss: 6.289e-02] jess: 47%|████▋ | 3732/8000 [00:54<17:46, 4.00it/s, lr: 6.0e-05 loss: 6.289e-02] jess: 47%|████▋ | 3732/8000 [00:54<17:46, 4.00it/s, lr: 6.0e-05 loss: 3.802e-02] jess: 47%|████▋ | 3733/8000 [00:54<18:53, 3.76it/s, lr: 6.0e-05 loss: 3.802e-02] jess: 47%|████▋ | 3733/8000 [00:54<18:53, 3.76it/s, lr: 6.0e-05 loss: 2.898e-02] jess: 47%|████▋ | 3734/8000 [00:54<19:12, 3.70it/s, lr: 6.0e-05 loss: 2.898e-02] jess: 47%|████▋ | 3734/8000 [00:54<19:12, 3.70it/s, lr: 6.0e-05 loss: 2.147e-01] jess: 47%|████▋ | 3735/8000 [00:54<19:18, 3.68it/s, lr: 6.0e-05 loss: 2.147e-01] jess: 47%|████▋ | 3735/8000 [00:54<19:18, 3.68it/s, lr: 6.0e-05 loss: 2.147e-01] jess: 47%|████▋ | 3736/8000 [00:54<13:06, 5.42it/s, lr: 6.0e-05 loss: 2.147e-01] jess: 47%|████▋ | 3736/8000 [00:55<13:06, 5.42it/s, lr: 6.0e-05 loss: 1.308e-01] jess: 47%|████▋ | 3737/8000 [00:55<17:26, 4.07it/s, lr: 6.0e-05 loss: 1.308e-01] jess: 47%|████▋ | 3737/8000 [00:55<17:26, 4.07it/s, lr: 6.0e-05 loss: 1.109e-01] jess: 47%|████▋ | 3738/8000 [00:55<18:44, 3.79it/s, lr: 6.0e-05 loss: 1.109e-01] jess: 47%|████▋ | 3738/8000 [00:55<18:44, 3.79it/s, lr: 6.0e-05 loss: 2.589e-02] jess: 47%|████▋ | 3739/8000 [00:55<19:01, 3.73it/s, lr: 6.0e-05 loss: 2.589e-02] jess: 47%|████▋ | 3739/8000 [00:56<19:01, 3.73it/s, lr: 6.0e-05 loss: 1.241e-01] jess: 47%|████▋ | 3740/8000 [00:56<19:39, 3.61it/s, lr: 6.0e-05 loss: 1.241e-01] jess: 47%|████▋ | 3740/8000 [00:56<19:39, 3.61it/s, lr: 6.0e-05 loss: 3.478e-02] jess: 47%|████▋ | 3741/8000 [00:56<19:25, 3.65it/s, lr: 6.0e-05 loss: 3.478e-02] jess: 47%|████▋ | 3741/8000 [00:56<19:25, 3.65it/s, lr: 6.0e-05 loss: 6.541e-02] jess: 47%|████▋ | 3742/8000 [00:56<13:06, 5.41it/s, lr: 6.0e-05 loss: 6.541e-02] jess: 47%|████▋ | 3742/8000 [00:56<13:06, 5.41it/s, lr: 6.0e-05 loss: 9.112e-02] jess: 47%|████▋ | 3743/8000 [00:56<11:11, 6.34it/s, lr: 6.0e-05 loss: 9.112e-02] jess: 47%|████▋ | 3743/8000 [00:56<11:11, 6.34it/s, lr: 6.0e-05 loss: 4.715e-02] jess: 47%|████▋ | 3744/8000 [00:56<10:38, 6.67it/s, lr: 6.0e-05 loss: 4.715e-02] jess: 47%|████▋ | 3744/8000 [00:57<10:38, 6.67it/s, lr: 6.0e-05 loss: 1.431e-01] jess: 47%|████▋ | 3745/8000 [00:57<16:35, 4.27it/s, lr: 6.0e-05 loss: 1.431e-01] jess: 47%|████▋ | 3745/8000 [00:57<16:35, 4.27it/s, lr: 6.0e-05 loss: 1.653e-02] jess: 47%|████▋ | 3746/8000 [00:57<12:13, 5.80it/s, lr: 6.0e-05 loss: 1.653e-02] jess: 47%|████▋ | 3746/8000 [00:57<12:13, 5.80it/s, lr: 6.0e-05 loss: 2.250e-01] jess: 47%|████▋ | 3747/8000 [00:57<17:08, 4.13it/s, lr: 6.0e-05 loss: 2.250e-01] jess: 47%|████▋ | 3747/8000 [00:57<17:08, 4.13it/s, lr: 6.0e-05 loss: 1.040e-01] jess: 47%|████▋ | 3748/8000 [00:57<12:27, 5.69it/s, lr: 6.0e-05 loss: 1.040e-01] jess: 47%|████▋ | 3748/8000 [00:57<12:27, 5.69it/s, lr: 6.0e-05 loss: 1.281e-01] jess: 47%|████▋ | 3749/8000 [00:57<10:59, 6.45it/s, lr: 6.0e-05 loss: 1.281e-01] jess: 47%|████▋ | 3749/8000 [00:58<10:59, 6.45it/s, lr: 6.0e-05 loss: 1.690e-01] jess: 47%|████▋ | 3750/8000 [00:58<24:42, 2.87it/s, lr: 6.0e-05 loss: 1.690e-01] jess: 47%|████▋ | 3750/8000 [00:58<24:42, 2.87it/s, lr: 6.0e-05 loss: 5.243e-02]
Saving at step 3750
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003750.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 47%|████▋ | 3751/8000 [00:58<24:42, 2.87it/s, lr: 6.0e-05 loss: 1.298e-01] jess: 47%|████▋ | 3752/8000 [00:58<20:09, 3.51it/s, lr: 6.0e-05 loss: 1.298e-01] jess: 47%|████▋ | 3752/8000 [00:59<20:09, 3.51it/s, lr: 6.0e-05 loss: 1.563e-01] jess: 47%|████▋ | 3753/8000 [00:59<20:04, 3.53it/s, lr: 6.0e-05 loss: 1.563e-01] jess: 47%|████▋ | 3753/8000 [00:59<20:04, 3.53it/s, lr: 6.0e-05 loss: 7.994e-02] jess: 47%|████▋ | 3754/8000 [00:59<13:33, 5.22it/s, lr: 6.0e-05 loss: 7.994e-02] jess: 47%|████▋ | 3754/8000 [00:59<13:33, 5.22it/s, lr: 6.0e-05 loss: 1.356e-02] jess: 47%|████▋ | 3755/8000 [00:59<11:26, 6.18it/s, lr: 6.0e-05 loss: 1.356e-02] jess: 47%|████▋ | 3755/8000 [00:59<11:26, 6.18it/s, lr: 6.0e-05 loss: 6.810e-02] jess: 47%|████▋ | 3756/8000 [00:59<10:37, 6.66it/s, lr: 6.0e-05 loss: 6.810e-02] jess: 47%|████▋ | 3756/8000 [00:59<10:37, 6.66it/s, lr: 6.0e-05 loss: 1.953e-01] jess: 47%|████▋ | 3757/8000 [00:59<10:22, 6.81it/s, lr: 6.0e-05 loss: 1.953e-01] jess: 47%|████▋ | 3757/8000 [00:59<10:22, 6.81it/s, lr: 6.0e-05 loss: 7.122e-02] jess: 47%|████▋ | 3758/8000 [00:59<10:18, 6.86it/s, lr: 6.0e-05 loss: 7.122e-02] jess: 47%|████▋ | 3758/8000 [01:00<10:18, 6.86it/s, lr: 6.0e-05 loss: 2.765e-01] jess: 47%|████▋ | 3759/8000 [01:00<16:21, 4.32it/s, lr: 6.0e-05 loss: 2.765e-01] jess: 47%|████▋ | 3759/8000 [01:00<16:21, 4.32it/s, lr: 6.0e-05 loss: 6.900e-02] jess: 47%|████▋ | 3760/8000 [01:00<12:04, 5.85it/s, lr: 6.0e-05 loss: 6.900e-02] jess: 47%|████▋ | 3760/8000 [01:00<12:04, 5.85it/s, lr: 6.0e-05 loss: 9.361e-02] jess: 47%|████▋ | 3761/8000 [01:00<16:58, 4.16it/s, lr: 6.0e-05 loss: 9.361e-02] jess: 47%|████▋ | 3761/8000 [01:00<16:58, 4.16it/s, lr: 6.0e-05 loss: 7.402e-02] jess: 47%|████▋ | 3762/8000 [01:00<12:14, 5.77it/s, lr: 6.0e-05 loss: 7.402e-02] jess: 47%|████▋ | 3762/8000 [01:00<12:14, 5.77it/s, lr: 6.0e-05 loss: 1.140e-01] jess: 47%|████▋ | 3763/8000 [01:00<10:50, 6.51it/s, lr: 6.0e-05 loss: 1.140e-01] jess: 47%|████▋ | 3763/8000 [01:01<10:50, 6.51it/s, lr: 6.0e-05 loss: 1.672e-01] jess: 47%|████▋ | 3764/8000 [01:01<16:35, 4.26it/s, lr: 6.0e-05 loss: 1.672e-01] jess: 47%|████▋ | 3764/8000 [01:01<16:35, 4.26it/s, lr: 6.0e-05 loss: 1.224e-01] jess: 47%|████▋ | 3765/8000 [01:01<18:21, 3.84it/s, lr: 6.0e-05 loss: 1.224e-01] jess: 47%|████▋ | 3765/8000 [01:01<18:21, 3.84it/s, lr: 6.0e-05 loss: 7.707e-02] jess: 47%|████▋ | 3766/8000 [01:01<12:41, 5.56it/s, lr: 6.0e-05 loss: 7.707e-02] jess: 47%|████▋ | 3766/8000 [01:01<12:41, 5.56it/s, lr: 6.0e-05 loss: 7.812e-02] jess: 47%|████▋ | 3767/8000 [01:01<10:56, 6.44it/s, lr: 6.0e-05 loss: 7.812e-02] jess: 47%|████▋ | 3767/8000 [01:01<10:56, 6.44it/s, lr: 6.0e-05 loss: 6.839e-02] jess: 47%|████▋ | 3768/8000 [01:01<16:32, 4.26it/s, lr: 6.0e-05 loss: 6.839e-02] jess: 47%|████▋ | 3768/8000 [01:02<16:32, 4.26it/s, lr: 6.0e-05 loss: 1.470e-01] jess: 47%|████▋ | 3769/8000 [01:02<18:21, 3.84it/s, lr: 6.0e-05 loss: 1.470e-01] jess: 47%|████▋ | 3769/8000 [01:02<18:21, 3.84it/s, lr: 6.0e-05 loss: 1.101e-01] jess: 47%|████▋ | 3770/8000 [01:02<19:09, 3.68it/s, lr: 6.0e-05 loss: 1.101e-01] jess: 47%|████▋ | 3770/8000 [01:02<19:09, 3.68it/s, lr: 6.0e-05 loss: 1.109e-01] jess: 47%|████▋ | 3771/8000 [01:02<12:55, 5.45it/s, lr: 6.0e-05 loss: 1.109e-01] jess: 47%|████▋ | 3771/8000 [01:02<12:55, 5.45it/s, lr: 6.0e-05 loss: 4.544e-02] jess: 47%|████▋ | 3772/8000 [01:02<17:05, 4.12it/s, lr: 6.0e-05 loss: 4.544e-02] jess: 47%|████▋ | 3772/8000 [01:03<17:05, 4.12it/s, lr: 6.0e-05 loss: 8.125e-02] jess: 47%|████▋ | 3773/8000 [01:03<12:16, 5.74it/s, lr: 6.0e-05 loss: 8.125e-02] jess: 47%|████▋ | 3773/8000 [01:03<12:16, 5.74it/s, lr: 6.0e-05 loss: 1.494e-01] jess: 47%|████▋ | 3774/8000 [01:03<10:47, 6.53it/s, lr: 6.0e-05 loss: 1.494e-01] jess: 47%|████▋ | 3774/8000 [01:03<10:47, 6.53it/s, lr: 6.0e-05 loss: 1.618e-01] jess: 47%|████▋ | 3775/8000 [01:03<10:25, 6.76it/s, lr: 6.0e-05 loss: 1.618e-01] jess: 47%|████▋ | 3775/8000 [01:03<10:25, 6.76it/s, lr: 6.0e-05 loss: 7.034e-02] jess: 47%|████▋ | 3776/8000 [01:03<29:17, 2.40it/s, lr: 6.0e-05 loss: 7.034e-02] jess: 47%|████▋ | 3776/8000 [01:04<29:17, 2.40it/s, lr: 6.0e-05 loss: 7.121e-02] jess: 47%|████▋ | 3777/8000 [01:04<22:13, 3.17it/s, lr: 6.0e-05 loss: 7.121e-02] jess: 47%|████▋ | 3777/8000 [01:04<22:13, 3.17it/s, lr: 6.0e-05 loss: 1.223e-01] jess: 47%|████▋ | 3778/8000 [01:04<20:02, 3.51it/s, lr: 6.0e-05 loss: 1.223e-01] jess: 47%|████▋ | 3778/8000 [01:04<20:02, 3.51it/s, lr: 6.0e-05 loss: 8.235e-02] jess: 47%|████▋ | 3779/8000 [01:04<19:20, 3.64it/s, lr: 6.0e-05 loss: 8.235e-02] jess: 47%|████▋ | 3779/8000 [01:04<19:20, 3.64it/s, lr: 6.0e-05 loss: 6.533e-02] jess: 47%|████▋ | 3780/8000 [01:04<13:00, 5.41it/s, lr: 6.0e-05 loss: 6.533e-02] jess: 47%|████▋ | 3780/8000 [01:05<13:00, 5.41it/s, lr: 6.0e-05 loss: 1.661e-02] jess: 47%|████▋ | 3781/8000 [01:05<11:05, 6.34it/s, lr: 6.0e-05 loss: 1.661e-02] jess: 47%|████▋ | 3781/8000 [01:05<11:05, 6.34it/s, lr: 6.0e-05 loss: 3.391e-02] jess: 47%|████▋ | 3782/8000 [01:05<16:37, 4.23it/s, lr: 6.0e-05 loss: 3.391e-02] jess: 47%|████▋ | 3782/8000 [01:05<16:37, 4.23it/s, lr: 6.0e-05 loss: 6.658e-02] jess: 47%|████▋ | 3783/8000 [01:05<18:10, 3.87it/s, lr: 6.0e-05 loss: 6.658e-02] jess: 47%|████▋ | 3783/8000 [01:05<18:10, 3.87it/s, lr: 6.0e-05 loss: 3.977e-02] jess: 47%|████▋ | 3784/8000 [01:05<17:25, 4.03it/s, lr: 6.0e-05 loss: 3.977e-02] jess: 47%|████▋ | 3784/8000 [01:06<17:25, 4.03it/s, lr: 6.0e-05 loss: 1.324e-01] jess: 47%|████▋ | 3785/8000 [01:06<18:26, 3.81it/s, lr: 6.0e-05 loss: 1.324e-01] jess: 47%|████▋ | 3785/8000 [01:06<18:26, 3.81it/s, lr: 6.0e-05 loss: 1.423e-01] jess: 47%|████▋ | 3786/8000 [01:06<18:51, 3.72it/s, lr: 6.0e-05 loss: 1.423e-01] jess: 47%|████▋ | 3786/8000 [01:06<18:51, 3.72it/s, lr: 6.0e-05 loss: 1.130e-01] jess: 47%|████▋ | 3787/8000 [01:06<12:53, 5.45it/s, lr: 6.0e-05 loss: 1.130e-01] jess: 47%|████▋ | 3787/8000 [01:06<12:53, 5.45it/s, lr: 6.0e-05 loss: 7.564e-02] jess: 47%|████▋ | 3788/8000 [01:06<17:10, 4.09it/s, lr: 6.0e-05 loss: 7.564e-02] jess: 47%|████▋ | 3788/8000 [01:07<17:10, 4.09it/s, lr: 6.0e-05 loss: 1.773e-01] jess: 47%|████▋ | 3789/8000 [01:07<18:56, 3.71it/s, lr: 6.0e-05 loss: 1.773e-01] jess: 47%|████▋ | 3789/8000 [01:07<18:56, 3.71it/s, lr: 6.0e-05 loss: 1.107e-01] jess: 47%|████▋ | 3790/8000 [01:07<18:54, 3.71it/s, lr: 6.0e-05 loss: 1.107e-01] jess: 47%|████▋ | 3790/8000 [01:07<18:54, 3.71it/s, lr: 6.0e-05 loss: 1.339e-01] jess: 47%|████▋ | 3791/8000 [01:07<12:57, 5.41it/s, lr: 6.0e-05 loss: 1.339e-01] jess: 47%|████▋ | 3791/8000 [01:07<12:57, 5.41it/s, lr: 6.0e-05 loss: 3.581e-03] jess: 47%|████▋ | 3792/8000 [01:07<10:59, 6.38it/s, lr: 6.0e-05 loss: 3.581e-03] jess: 47%|████▋ | 3792/8000 [01:07<10:59, 6.38it/s, lr: 6.0e-05 loss: 7.291e-02] jess: 47%|████▋ | 3793/8000 [01:07<16:37, 4.22it/s, lr: 6.0e-05 loss: 7.291e-02] jess: 47%|████▋ | 3793/8000 [01:08<16:37, 4.22it/s, lr: 6.0e-05 loss: 2.525e-01] jess: 47%|████▋ | 3794/8000 [01:08<18:08, 3.86it/s, lr: 6.0e-05 loss: 2.525e-01] jess: 47%|████▋ | 3794/8000 [01:08<18:08, 3.86it/s, lr: 6.0e-05 loss: 1.207e-01] jess: 47%|████▋ | 3795/8000 [01:08<12:39, 5.54it/s, lr: 6.0e-05 loss: 1.207e-01] jess: 47%|████▋ | 3795/8000 [01:08<12:39, 5.54it/s, lr: 6.0e-05 loss: 1.323e-01] jess: 47%|████▋ | 3796/8000 [01:08<16:59, 4.13it/s, lr: 6.0e-05 loss: 1.323e-01] jess: 47%|████▋ | 3796/8000 [01:08<16:59, 4.13it/s, lr: 6.0e-05 loss: 1.976e-02] jess: 47%|████▋ | 3797/8000 [01:08<18:14, 3.84it/s, lr: 6.0e-05 loss: 1.976e-02] jess: 47%|████▋ | 3797/8000 [01:09<18:14, 3.84it/s, lr: 6.0e-05 loss: 5.695e-02] jess: 47%|████▋ | 3798/8000 [01:09<18:47, 3.73it/s, lr: 6.0e-05 loss: 5.695e-02] jess: 47%|████▋ | 3798/8000 [01:09<18:47, 3.73it/s, lr: 6.0e-05 loss: 1.719e-01] jess: 47%|████▋ | 3799/8000 [01:09<12:47, 5.48it/s, lr: 6.0e-05 loss: 1.719e-01] jess: 47%|████▋ | 3799/8000 [01:09<12:47, 5.48it/s, lr: 6.0e-05 loss: 2.974e-02] jess: 48%|████▊ | 3800/8000 [01:09<24:30, 2.86it/s, lr: 6.0e-05 loss: 2.974e-02] jess: 48%|████▊ | 3800/8000 [01:09<24:30, 2.86it/s, lr: 6.0e-05 loss: 1.418e-01]
Saving at step 3800
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003800.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.33s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.33s/it]
Generating Images: 100%|██████████| 3/3 [00:03<00:00, 1.33s/it]

jess: 48%|████▊ | 3801/8000 [01:10<24:30, 2.86it/s, lr: 6.0e-05 loss: 1.695e-01] jess: 48%|████▊ | 3802/8000 [01:10<20:11, 3.46it/s, lr: 6.0e-05 loss: 1.695e-01] jess: 48%|████▊ | 3802/8000 [01:10<20:11, 3.46it/s, lr: 6.0e-05 loss: 3.914e-02] jess: 48%|████▊ | 3803/8000 [01:10<14:32, 4.81it/s, lr: 6.0e-05 loss: 3.914e-02] jess: 48%|████▊ | 3803/8000 [01:10<14:32, 4.81it/s, lr: 6.0e-05 loss: 7.638e-02] jess: 48%|████▊ | 3804/8000 [01:10<17:26, 4.01it/s, lr: 6.0e-05 loss: 7.638e-02] jess: 48%|████▊ | 3804/8000 [01:10<17:26, 4.01it/s, lr: 6.0e-05 loss: 8.707e-02] jess: 48%|████▊ | 3805/8000 [01:10<12:27, 5.62it/s, lr: 6.0e-05 loss: 8.707e-02] jess: 48%|████▊ | 3805/8000 [01:11<12:27, 5.62it/s, lr: 6.0e-05 loss: 1.352e-01] jess: 48%|████▊ | 3806/8000 [01:11<10:51, 6.44it/s, lr: 6.0e-05 loss: 1.352e-01] jess: 48%|████▊ | 3806/8000 [01:11<10:51, 6.44it/s, lr: 6.0e-05 loss: 3.240e-02] jess: 48%|████▊ | 3807/8000 [01:11<10:20, 6.76it/s, lr: 6.0e-05 loss: 3.240e-02] jess: 48%|████▊ | 3807/8000 [01:11<10:20, 6.76it/s, lr: 6.0e-05 loss: 1.083e-01] jess: 48%|████▊ | 3808/8000 [01:11<16:13, 4.31it/s, lr: 6.0e-05 loss: 1.083e-01] jess: 48%|████▊ | 3808/8000 [01:11<16:13, 4.31it/s, lr: 6.0e-05 loss: 1.113e-02] jess: 48%|████▊ | 3809/8000 [01:11<17:48, 3.92it/s, lr: 6.0e-05 loss: 1.113e-02] jess: 48%|████▊ | 3809/8000 [01:11<17:48, 3.92it/s, lr: 6.0e-05 loss: 1.586e-01] jess: 48%|████▊ | 3810/8000 [01:11<12:25, 5.62it/s, lr: 6.0e-05 loss: 1.586e-01] jess: 48%|████▊ | 3810/8000 [01:12<12:25, 5.62it/s, lr: 6.0e-05 loss: 1.243e-01] jess: 48%|████▊ | 3811/8000 [01:12<10:46, 6.48it/s, lr: 6.0e-05 loss: 1.243e-01] jess: 48%|████▊ | 3811/8000 [01:12<10:46, 6.48it/s, lr: 6.0e-05 loss: 1.558e-01] jess: 48%|████▊ | 3812/8000 [01:12<10:20, 6.75it/s, lr: 6.0e-05 loss: 1.558e-01] jess: 48%|████▊ | 3812/8000 [01:12<10:20, 6.75it/s, lr: 6.0e-05 loss: 1.022e-01] jess: 48%|████▊ | 3813/8000 [01:12<16:14, 4.30it/s, lr: 6.0e-05 loss: 1.022e-01] jess: 48%|████▊ | 3813/8000 [01:12<16:14, 4.30it/s, lr: 6.0e-05 loss: 4.560e-03] jess: 48%|████▊ | 3814/8000 [01:12<11:59, 5.82it/s, lr: 6.0e-05 loss: 4.560e-03] jess: 48%|████▊ | 3814/8000 [01:12<11:59, 5.82it/s, lr: 6.0e-05 loss: 1.185e-01] jess: 48%|████▊ | 3815/8000 [01:12<16:50, 4.14it/s, lr: 6.0e-05 loss: 1.185e-01] jess: 48%|████▊ | 3815/8000 [01:13<16:50, 4.14it/s, lr: 6.0e-05 loss: 7.487e-02] jess: 48%|████▊ | 3816/8000 [01:13<12:09, 5.74it/s, lr: 6.0e-05 loss: 7.487e-02] jess: 48%|████▊ | 3816/8000 [01:13<12:09, 5.74it/s, lr: 6.0e-05 loss: 1.344e-01] jess: 48%|████▊ | 3817/8000 [01:13<16:52, 4.13it/s, lr: 6.0e-05 loss: 1.344e-01] jess: 48%|████▊ | 3817/8000 [01:13<16:52, 4.13it/s, lr: 6.0e-05 loss: 1.014e-02] jess: 48%|████▊ | 3818/8000 [01:13<16:58, 4.10it/s, lr: 6.0e-05 loss: 1.014e-02] jess: 48%|████▊ | 3818/8000 [01:13<16:58, 4.10it/s, lr: 6.0e-05 loss: 5.161e-02] jess: 48%|████▊ | 3819/8000 [01:13<12:07, 5.75it/s, lr: 6.0e-05 loss: 5.161e-02] jess: 48%|████▊ | 3819/8000 [01:13<12:07, 5.75it/s, lr: 6.0e-05 loss: 1.805e-01] jess: 48%|████▊ | 3820/8000 [01:13<10:37, 6.56it/s, lr: 6.0e-05 loss: 1.805e-01] jess: 48%|████▊ | 3820/8000 [01:14<10:37, 6.56it/s, lr: 6.0e-05 loss: 1.410e-01] jess: 48%|████▊ | 3821/8000 [01:14<16:28, 4.23it/s, lr: 6.0e-05 loss: 1.410e-01] jess: 48%|████▊ | 3821/8000 [01:14<16:28, 4.23it/s, lr: 6.0e-05 loss: 3.481e-02] jess: 48%|████▊ | 3822/8000 [01:14<18:07, 3.84it/s, lr: 6.0e-05 loss: 3.481e-02] jess: 48%|████▊ | 3822/8000 [01:14<18:07, 3.84it/s, lr: 6.0e-05 loss: 6.289e-02] jess: 48%|████▊ | 3823/8000 [01:14<18:36, 3.74it/s, lr: 6.0e-05 loss: 6.289e-02] jess: 48%|████▊ | 3823/8000 [01:14<18:36, 3.74it/s, lr: 6.0e-05 loss: 1.474e-01] jess: 48%|████▊ | 3824/8000 [01:14<18:46, 3.71it/s, lr: 6.0e-05 loss: 1.474e-01] jess: 48%|████▊ | 3824/8000 [01:15<18:46, 3.71it/s, lr: 6.0e-05 loss: 3.631e-02] jess: 48%|████▊ | 3825/8000 [01:15<13:05, 5.32it/s, lr: 6.0e-05 loss: 3.631e-02] jess: 48%|████▊ | 3825/8000 [01:15<13:05, 5.32it/s, lr: 6.0e-05 loss: 1.120e-01] jess: 48%|████▊ | 3826/8000 [01:15<30:08, 2.31it/s, lr: 6.0e-05 loss: 1.120e-01] jess: 48%|████▊ | 3826/8000 [01:15<30:08, 2.31it/s, lr: 6.0e-05 loss: 1.022e-01] jess: 48%|████▊ | 3827/8000 [01:15<16:21, 4.25it/s, lr: 6.0e-05 loss: 1.022e-01] jess: 48%|████▊ | 3827/8000 [01:15<16:21, 4.25it/s, lr: 6.0e-05 loss: 1.753e-01] jess: 48%|████▊ | 3828/8000 [01:15<12:00, 5.79it/s, lr: 6.0e-05 loss: 1.753e-01] jess: 48%|████▊ | 3828/8000 [01:16<12:00, 5.79it/s, lr: 6.0e-05 loss: 8.512e-02] jess: 48%|████▊ | 3829/8000 [01:16<16:47, 4.14it/s, lr: 6.0e-05 loss: 8.512e-02] jess: 48%|████▊ | 3829/8000 [01:16<16:47, 4.14it/s, lr: 6.0e-05 loss: 2.178e-01] jess: 48%|████▊ | 3830/8000 [01:16<18:12, 3.82it/s, lr: 6.0e-05 loss: 2.178e-01] jess: 48%|████▊ | 3830/8000 [01:16<18:12, 3.82it/s, lr: 6.0e-05 loss: 1.383e-01] jess: 48%|████▊ | 3831/8000 [01:16<18:40, 3.72it/s, lr: 6.0e-05 loss: 1.383e-01] jess: 48%|████▊ | 3831/8000 [01:16<18:40, 3.72it/s, lr: 6.0e-05 loss: 1.649e-01] jess: 48%|████▊ | 3832/8000 [01:16<18:40, 3.72it/s, lr: 6.0e-05 loss: 1.649e-01] jess: 48%|████▊ | 3832/8000 [01:17<18:40, 3.72it/s, lr: 6.0e-05 loss: 5.621e-02] jess: 48%|████▊ | 3833/8000 [01:17<12:41, 5.48it/s, lr: 6.0e-05 loss: 5.621e-02] jess: 48%|████▊ | 3833/8000 [01:17<12:41, 5.48it/s, lr: 6.0e-05 loss: 2.139e-02] jess: 48%|████▊ | 3834/8000 [01:17<10:55, 6.35it/s, lr: 6.0e-05 loss: 2.139e-02] jess: 48%|████▊ | 3834/8000 [01:17<10:55, 6.35it/s, lr: 6.0e-05 loss: 1.035e-01] jess: 48%|████▊ | 3835/8000 [01:17<10:17, 6.75it/s, lr: 6.0e-05 loss: 1.035e-01] jess: 48%|████▊ | 3835/8000 [01:17<10:17, 6.75it/s, lr: 6.0e-05 loss: 9.050e-03] jess: 48%|████▊ | 3836/8000 [01:17<14:52, 4.67it/s, lr: 6.0e-05 loss: 9.050e-03] jess: 48%|████▊ | 3836/8000 [01:17<14:52, 4.67it/s, lr: 6.0e-05 loss: 8.861e-03] jess: 48%|████▊ | 3837/8000 [01:17<17:28, 3.97it/s, lr: 6.0e-05 loss: 8.861e-03] jess: 48%|████▊ | 3837/8000 [01:18<17:28, 3.97it/s, lr: 6.0e-05 loss: 6.048e-02] jess: 48%|████▊ | 3838/8000 [01:18<12:21, 5.61it/s, lr: 6.0e-05 loss: 6.048e-02] jess: 48%|████▊ | 3838/8000 [01:18<12:21, 5.61it/s, lr: 6.0e-05 loss: 1.768e-01] jess: 48%|████▊ | 3839/8000 [01:18<16:50, 4.12it/s, lr: 6.0e-05 loss: 1.768e-01] jess: 48%|████▊ | 3839/8000 [01:18<16:50, 4.12it/s, lr: 6.0e-05 loss: 7.779e-02] jess: 48%|████▊ | 3840/8000 [01:18<18:05, 3.83it/s, lr: 6.0e-05 loss: 7.779e-02] jess: 48%|████▊ | 3840/8000 [01:18<18:05, 3.83it/s, lr: 6.0e-05 loss: 2.016e-01] jess: 48%|████▊ | 3841/8000 [01:18<18:40, 3.71it/s, lr: 6.0e-05 loss: 2.016e-01] jess: 48%|████▊ | 3841/8000 [01:19<18:40, 3.71it/s, lr: 6.0e-05 loss: 9.759e-02] jess: 48%|████▊ | 3842/8000 [01:19<12:37, 5.49it/s, lr: 6.0e-05 loss: 9.759e-02] jess: 48%|████▊ | 3842/8000 [01:19<12:37, 5.49it/s, lr: 6.0e-05 loss: 1.379e-01] jess: 48%|████▊ | 3843/8000 [01:19<10:51, 6.38it/s, lr: 6.0e-05 loss: 1.379e-01] jess: 48%|████▊ | 3843/8000 [01:19<10:51, 6.38it/s, lr: 6.0e-05 loss: 2.328e-01] jess: 48%|████▊ | 3844/8000 [01:19<16:16, 4.25it/s, lr: 6.0e-05 loss: 2.328e-01] jess: 48%|████▊ | 3844/8000 [01:19<16:16, 4.25it/s, lr: 6.0e-05 loss: 2.513e-02] jess: 48%|████▊ | 3845/8000 [01:19<12:03, 5.74it/s, lr: 6.0e-05 loss: 2.513e-02] jess: 48%|████▊ | 3845/8000 [01:19<12:03, 5.74it/s, lr: 6.0e-05 loss: 7.669e-02] jess: 48%|████▊ | 3846/8000 [01:19<10:37, 6.51it/s, lr: 6.0e-05 loss: 7.669e-02] jess: 48%|████▊ | 3846/8000 [01:20<10:37, 6.51it/s, lr: 6.0e-05 loss: 4.939e-02] jess: 48%|████▊ | 3847/8000 [01:20<16:22, 4.23it/s, lr: 6.0e-05 loss: 4.939e-02] jess: 48%|████▊ | 3847/8000 [01:20<16:22, 4.23it/s, lr: 6.0e-05 loss: 2.334e-01] jess: 48%|████▊ | 3848/8000 [01:20<17:59, 3.85it/s, lr: 6.0e-05 loss: 2.334e-01] jess: 48%|████▊ | 3848/8000 [01:20<17:59, 3.85it/s, lr: 6.0e-05 loss: 9.386e-02] jess: 48%|████▊ | 3849/8000 [01:20<12:34, 5.50it/s, lr: 6.0e-05 loss: 9.386e-02] jess: 48%|████▊ | 3849/8000 [01:20<12:34, 5.50it/s, lr: 6.0e-05 loss: 5.742e-02] jess: 48%|████▊ | 3850/8000 [01:20<29:52, 2.32it/s, lr: 6.0e-05 loss: 5.742e-02] jess: 48%|████▊ | 3850/8000 [01:21<29:52, 2.32it/s, lr: 6.0e-05 loss: 1.288e-01]
Saving at step 3850
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003850.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 48%|████▊ | 3851/8000 [01:21<29:51, 2.32it/s, lr: 6.0e-05 loss: 6.251e-03] jess: 48%|████▊ | 3852/8000 [01:21<18:09, 3.81it/s, lr: 6.0e-05 loss: 6.251e-03] jess: 48%|████▊ | 3852/8000 [01:21<18:09, 3.81it/s, lr: 6.0e-05 loss: 2.443e-02] jess: 48%|████▊ | 3853/8000 [01:21<18:18, 3.78it/s, lr: 6.0e-05 loss: 2.443e-02] jess: 48%|████▊ | 3853/8000 [01:21<18:18, 3.78it/s, lr: 6.0e-05 loss: 1.458e-01] jess: 48%|████▊ | 3854/8000 [01:21<12:54, 5.35it/s, lr: 6.0e-05 loss: 1.458e-01] jess: 48%|████▊ | 3854/8000 [01:22<12:54, 5.35it/s, lr: 6.0e-05 loss: 1.624e-01] jess: 48%|████▊ | 3855/8000 [01:22<16:47, 4.11it/s, lr: 6.0e-05 loss: 1.624e-01] jess: 48%|████▊ | 3855/8000 [01:22<16:47, 4.11it/s, lr: 6.0e-05 loss: 6.811e-02] jess: 48%|████▊ | 3856/8000 [01:22<18:11, 3.80it/s, lr: 6.0e-05 loss: 6.811e-02] jess: 48%|████▊ | 3856/8000 [01:22<18:11, 3.80it/s, lr: 6.0e-05 loss: 1.562e-01] jess: 48%|████▊ | 3857/8000 [01:22<18:29, 3.73it/s, lr: 6.0e-05 loss: 1.562e-01] jess: 48%|████▊ | 3857/8000 [01:23<18:29, 3.73it/s, lr: 6.0e-05 loss: 1.231e-01] jess: 48%|████▊ | 3858/8000 [01:23<18:40, 3.70it/s, lr: 6.0e-05 loss: 1.231e-01] jess: 48%|████▊ | 3858/8000 [01:23<18:40, 3.70it/s, lr: 6.0e-05 loss: 8.877e-02] jess: 48%|████▊ | 3859/8000 [01:23<12:37, 5.47it/s, lr: 6.0e-05 loss: 8.877e-02] jess: 48%|████▊ | 3859/8000 [01:23<12:37, 5.47it/s, lr: 6.0e-05 loss: 1.415e-01] jess: 48%|████▊ | 3860/8000 [01:23<10:47, 6.40it/s, lr: 6.0e-05 loss: 1.415e-01] jess: 48%|████▊ | 3860/8000 [01:23<10:47, 6.40it/s, lr: 6.0e-05 loss: 1.370e-01] jess: 48%|████▊ | 3861/8000 [01:23<16:43, 4.12it/s, lr: 6.0e-05 loss: 1.370e-01] jess: 48%|████▊ | 3861/8000 [01:23<16:43, 4.12it/s, lr: 6.0e-05 loss: 8.788e-02] jess: 48%|████▊ | 3862/8000 [01:23<18:07, 3.81it/s, lr: 6.0e-05 loss: 8.788e-02] jess: 48%|████▊ | 3862/8000 [01:24<18:07, 3.81it/s, lr: 6.0e-05 loss: 9.929e-02] jess: 48%|████▊ | 3863/8000 [01:24<12:32, 5.50it/s, lr: 6.0e-05 loss: 9.929e-02] jess: 48%|████▊ | 3863/8000 [01:24<12:32, 5.50it/s, lr: 6.0e-05 loss: 2.941e-02] jess: 48%|████▊ | 3864/8000 [01:24<16:38, 4.14it/s, lr: 6.0e-05 loss: 2.941e-02] jess: 48%|████▊ | 3864/8000 [01:24<16:38, 4.14it/s, lr: 6.0e-05 loss: 6.130e-02] jess: 48%|████▊ | 3865/8000 [01:24<17:45, 3.88it/s, lr: 6.0e-05 loss: 6.130e-02] jess: 48%|████▊ | 3865/8000 [01:24<17:45, 3.88it/s, lr: 6.0e-05 loss: 2.158e-02] jess: 48%|████▊ | 3866/8000 [01:24<17:06, 4.03it/s, lr: 6.0e-05 loss: 2.158e-02] jess: 48%|████▊ | 3866/8000 [01:25<17:06, 4.03it/s, lr: 6.0e-05 loss: 1.976e-02] jess: 48%|████▊ | 3867/8000 [01:25<18:11, 3.79it/s, lr: 6.0e-05 loss: 1.976e-02] jess: 48%|████▊ | 3867/8000 [01:25<18:11, 3.79it/s, lr: 6.0e-05 loss: 8.692e-02] jess: 48%|████▊ | 3868/8000 [01:25<18:34, 3.71it/s, lr: 6.0e-05 loss: 8.692e-02] jess: 48%|████▊ | 3868/8000 [01:25<18:34, 3.71it/s, lr: 6.0e-05 loss: 7.195e-02] jess: 48%|████▊ | 3869/8000 [01:25<12:44, 5.40it/s, lr: 6.0e-05 loss: 7.195e-02] jess: 48%|████▊ | 3869/8000 [01:25<12:44, 5.40it/s, lr: 6.0e-05 loss: 1.820e-01] jess: 48%|████▊ | 3870/8000 [01:25<10:53, 6.32it/s, lr: 6.0e-05 loss: 1.820e-01] jess: 48%|████▊ | 3870/8000 [01:25<10:53, 6.32it/s, lr: 6.0e-05 loss: 6.832e-02] jess: 48%|████▊ | 3871/8000 [01:25<10:20, 6.65it/s, lr: 6.0e-05 loss: 6.832e-02] jess: 48%|████▊ | 3871/8000 [01:26<10:20, 6.65it/s, lr: 6.0e-05 loss: 1.117e-01] jess: 48%|████▊ | 3872/8000 [01:26<16:02, 4.29it/s, lr: 6.0e-05 loss: 1.117e-01] jess: 48%|████▊ | 3872/8000 [01:26<16:02, 4.29it/s, lr: 6.0e-05 loss: 5.599e-02] jess: 48%|████▊ | 3873/8000 [01:26<11:54, 5.78it/s, lr: 6.0e-05 loss: 5.599e-02] jess: 48%|████▊ | 3873/8000 [01:26<11:54, 5.78it/s, lr: 6.0e-05 loss: 1.208e-01] jess: 48%|████▊ | 3874/8000 [01:26<10:39, 6.45it/s, lr: 6.0e-05 loss: 1.208e-01] jess: 48%|████▊ | 3874/8000 [01:26<10:39, 6.45it/s, lr: 6.0e-05 loss: 1.467e-02] jess: 48%|████▊ | 3875/8000 [01:26<10:15, 6.70it/s, lr: 6.0e-05 loss: 1.467e-02] jess: 48%|████▊ | 3875/8000 [01:26<10:15, 6.70it/s, lr: 6.0e-05 loss: 1.172e-01] jess: 48%|████▊ | 3876/8000 [01:26<24:15, 2.83it/s, lr: 6.0e-05 loss: 1.172e-01] jess: 48%|████▊ | 3876/8000 [01:27<24:15, 2.83it/s, lr: 6.0e-05 loss: 4.406e-02] jess: 48%|████▊ | 3877/8000 [01:27<14:26, 4.76it/s, lr: 6.0e-05 loss: 4.406e-02] jess: 48%|████▊ | 3877/8000 [01:27<14:26, 4.76it/s, lr: 6.0e-05 loss: 8.140e-02] jess: 48%|████▊ | 3878/8000 [01:27<17:17, 3.97it/s, lr: 6.0e-05 loss: 8.140e-02] jess: 48%|████▊ | 3878/8000 [01:27<17:17, 3.97it/s, lr: 6.0e-05 loss: 3.336e-02] jess: 48%|████▊ | 3879/8000 [01:27<12:17, 5.59it/s, lr: 6.0e-05 loss: 3.336e-02] jess: 48%|████▊ | 3879/8000 [01:27<12:17, 5.59it/s, lr: 6.0e-05 loss: 1.204e-01] jess: 48%|████▊ | 3880/8000 [01:27<16:45, 4.10it/s, lr: 6.0e-05 loss: 1.204e-01] jess: 48%|████▊ | 3880/8000 [01:28<16:45, 4.10it/s, lr: 6.0e-05 loss: 4.382e-02] jess: 49%|████▊ | 3881/8000 [01:28<18:01, 3.81it/s, lr: 6.0e-05 loss: 4.382e-02] jess: 49%|████▊ | 3881/8000 [01:28<18:01, 3.81it/s, lr: 6.0e-05 loss: 2.196e-01] jess: 49%|████▊ | 3882/8000 [01:28<18:26, 3.72it/s, lr: 6.0e-05 loss: 2.196e-01] jess: 49%|████▊ | 3882/8000 [01:28<18:26, 3.72it/s, lr: 6.0e-05 loss: 1.431e-01] jess: 49%|████▊ | 3883/8000 [01:28<18:33, 3.70it/s, lr: 6.0e-05 loss: 1.431e-01] jess: 49%|████▊ | 3883/8000 [01:28<18:33, 3.70it/s, lr: 6.0e-05 loss: 9.428e-02] jess: 49%|████▊ | 3884/8000 [01:28<12:32, 5.47it/s, lr: 6.0e-05 loss: 9.428e-02] jess: 49%|████▊ | 3884/8000 [01:28<12:32, 5.47it/s, lr: 6.0e-05 loss: 1.103e-01] jess: 49%|████▊ | 3885/8000 [01:28<10:47, 6.36it/s, lr: 6.0e-05 loss: 1.103e-01] jess: 49%|████▊ | 3885/8000 [01:29<10:47, 6.36it/s, lr: 6.0e-05 loss: 1.367e-01] jess: 49%|████▊ | 3886/8000 [01:29<16:18, 4.21it/s, lr: 6.0e-05 loss: 1.367e-01] jess: 49%|████▊ | 3886/8000 [01:29<16:18, 4.21it/s, lr: 6.0e-05 loss: 1.337e-01] jess: 49%|████▊ | 3887/8000 [01:29<11:54, 5.76it/s, lr: 6.0e-05 loss: 1.337e-01] jess: 49%|████▊ | 3887/8000 [01:29<11:54, 5.76it/s, lr: 6.0e-05 loss: 7.000e-02] jess: 49%|████▊ | 3888/8000 [01:29<10:37, 6.45it/s, lr: 6.0e-05 loss: 7.000e-02] jess: 49%|████▊ | 3888/8000 [01:29<10:37, 6.45it/s, lr: 6.0e-05 loss: 1.043e-01] jess: 49%|████▊ | 3889/8000 [01:29<16:05, 4.26it/s, lr: 6.0e-05 loss: 1.043e-01] jess: 49%|████▊ | 3889/8000 [01:30<16:05, 4.26it/s, lr: 6.0e-05 loss: 7.003e-02] jess: 49%|████▊ | 3890/8000 [01:30<17:52, 3.83it/s, lr: 6.0e-05 loss: 7.003e-02] jess: 49%|████▊ | 3890/8000 [01:30<17:52, 3.83it/s, lr: 6.0e-05 loss: 1.292e-01] jess: 49%|████▊ | 3891/8000 [01:30<12:35, 5.44it/s, lr: 6.0e-05 loss: 1.292e-01] jess: 49%|████▊ | 3891/8000 [01:30<12:35, 5.44it/s, lr: 6.0e-05 loss: 1.308e-02] jess: 49%|████▊ | 3892/8000 [01:30<10:47, 6.35it/s, lr: 6.0e-05 loss: 1.308e-02] jess: 49%|████▊ | 3892/8000 [01:30<10:47, 6.35it/s, lr: 6.0e-05 loss: 1.937e-01] jess: 49%|████▊ | 3893/8000 [01:30<16:15, 4.21it/s, lr: 6.0e-05 loss: 1.937e-01] jess: 49%|████▊ | 3893/8000 [01:30<16:15, 4.21it/s, lr: 6.0e-05 loss: 1.096e-01] jess: 49%|████▊ | 3894/8000 [01:30<17:58, 3.81it/s, lr: 6.0e-05 loss: 1.096e-01] jess: 49%|████▊ | 3894/8000 [01:31<17:58, 3.81it/s, lr: 6.0e-05 loss: 7.812e-02] jess: 49%|████▊ | 3895/8000 [01:31<18:27, 3.71it/s, lr: 6.0e-05 loss: 7.812e-02] jess: 49%|████▊ | 3895/8000 [01:31<18:27, 3.71it/s, lr: 6.0e-05 loss: 7.856e-02] jess: 49%|████▊ | 3896/8000 [01:31<12:40, 5.40it/s, lr: 6.0e-05 loss: 7.856e-02] jess: 49%|████▊ | 3896/8000 [01:31<12:40, 5.40it/s, lr: 6.0e-05 loss: 2.708e-02] jess: 49%|████▊ | 3897/8000 [01:31<12:14, 5.59it/s, lr: 6.0e-05 loss: 2.708e-02] jess: 49%|████▊ | 3897/8000 [01:31<12:14, 5.59it/s, lr: 6.0e-05 loss: 1.499e-01] jess: 49%|████▊ | 3898/8000 [01:31<10:41, 6.39it/s, lr: 6.0e-05 loss: 1.499e-01] jess: 49%|████▊ | 3898/8000 [01:31<10:41, 6.39it/s, lr: 6.0e-05 loss: 2.489e-03] jess: 49%|████▊ | 3899/8000 [01:31<10:14, 6.68it/s, lr: 6.0e-05 loss: 2.489e-03] jess: 49%|████▊ | 3899/8000 [01:32<10:14, 6.68it/s, lr: 6.0e-05 loss: 1.282e-01] jess: 49%|████▉ | 3900/8000 [01:32<29:08, 2.35it/s, lr: 6.0e-05 loss: 1.282e-01] jess: 49%|████▉ | 3900/8000 [01:32<29:08, 2.35it/s, lr: 6.0e-05 loss: 1.021e-01]
Saving at step 3900
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003900.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.34s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.33s/it]
Generating Images: 100%|██████████| 3/3 [00:03<00:00, 1.33s/it]

jess: 49%|████▉ | 3901/8000 [01:32<29:07, 2.35it/s, lr: 6.0e-05 loss: 1.959e-01] jess: 49%|████▉ | 3902/8000 [01:32<20:39, 3.31it/s, lr: 6.0e-05 loss: 1.959e-01] jess: 49%|████▉ | 3902/8000 [01:33<20:39, 3.31it/s, lr: 6.0e-05 loss: 1.478e-01] jess: 49%|████▉ | 3903/8000 [01:33<19:31, 3.50it/s, lr: 6.0e-05 loss: 1.478e-01] jess: 49%|████▉ | 3903/8000 [01:33<19:31, 3.50it/s, lr: 6.0e-05 loss: 9.218e-03] jess: 49%|████▉ | 3904/8000 [01:33<13:17, 5.14it/s, lr: 6.0e-05 loss: 9.218e-03] jess: 49%|████▉ | 3904/8000 [01:33<13:17, 5.14it/s, lr: 6.0e-05 loss: 2.410e-01] jess: 49%|████▉ | 3905/8000 [01:33<16:46, 4.07it/s, lr: 6.0e-05 loss: 2.410e-01] jess: 49%|████▉ | 3905/8000 [01:33<16:46, 4.07it/s, lr: 6.0e-05 loss: 1.567e-02] jess: 49%|████▉ | 3906/8000 [01:33<17:42, 3.85it/s, lr: 6.0e-05 loss: 1.567e-02] jess: 49%|████▉ | 3906/8000 [01:34<17:42, 3.85it/s, lr: 6.0e-05 loss: 9.651e-02] jess: 49%|████▉ | 3907/8000 [01:34<18:19, 3.72it/s, lr: 6.0e-05 loss: 9.651e-02] jess: 49%|████▉ | 3907/8000 [01:34<18:19, 3.72it/s, lr: 6.0e-05 loss: 2.377e-02] jess: 49%|████▉ | 3908/8000 [01:34<18:32, 3.68it/s, lr: 6.0e-05 loss: 2.377e-02] jess: 49%|████▉ | 3908/8000 [01:34<18:32, 3.68it/s, lr: 6.0e-05 loss: 4.228e-02] jess: 49%|████▉ | 3909/8000 [01:34<12:36, 5.41it/s, lr: 6.0e-05 loss: 4.228e-02] jess: 49%|████▉ | 3909/8000 [01:34<12:36, 5.41it/s, lr: 6.0e-05 loss: 9.808e-02] jess: 49%|████▉ | 3910/8000 [01:34<10:44, 6.35it/s, lr: 6.0e-05 loss: 9.808e-02] jess: 49%|████▉ | 3910/8000 [01:35<10:44, 6.35it/s, lr: 6.0e-05 loss: 1.172e-01] jess: 49%|████▉ | 3911/8000 [01:35<16:06, 4.23it/s, lr: 6.0e-05 loss: 1.172e-01] jess: 49%|████▉ | 3911/8000 [01:35<16:06, 4.23it/s, lr: 6.0e-05 loss: 4.103e-02] jess: 49%|████▉ | 3912/8000 [01:35<17:37, 3.87it/s, lr: 6.0e-05 loss: 4.103e-02] jess: 49%|████▉ | 3912/8000 [01:35<17:37, 3.87it/s, lr: 6.0e-05 loss: 1.146e-01] jess: 49%|████▉ | 3913/8000 [01:35<18:15, 3.73it/s, lr: 6.0e-05 loss: 1.146e-01] jess: 49%|████▉ | 3913/8000 [01:35<18:15, 3.73it/s, lr: 6.0e-05 loss: 1.968e-01] jess: 49%|████▉ | 3914/8000 [01:35<12:47, 5.32it/s, lr: 6.0e-05 loss: 1.968e-01] jess: 49%|████▉ | 3914/8000 [01:35<12:47, 5.32it/s, lr: 6.0e-05 loss: 1.020e-02] jess: 49%|████▉ | 3915/8000 [01:35<15:29, 4.40it/s, lr: 6.0e-05 loss: 1.020e-02] jess: 49%|████▉ | 3915/8000 [01:36<15:29, 4.40it/s, lr: 6.0e-05 loss: 9.952e-02] jess: 49%|████▉ | 3916/8000 [01:36<11:36, 5.86it/s, lr: 6.0e-05 loss: 9.952e-02] jess: 49%|████▉ | 3916/8000 [01:36<11:36, 5.86it/s, lr: 6.0e-05 loss: 1.253e-01] jess: 49%|████▉ | 3917/8000 [01:36<16:23, 4.15it/s, lr: 6.0e-05 loss: 1.253e-01] jess: 49%|████▉ | 3917/8000 [01:36<16:23, 4.15it/s, lr: 6.0e-05 loss: 9.645e-02] jess: 49%|████▉ | 3918/8000 [01:36<17:51, 3.81it/s, lr: 6.0e-05 loss: 9.645e-02] jess: 49%|████▉ | 3918/8000 [01:36<17:51, 3.81it/s, lr: 6.0e-05 loss: 5.941e-02] jess: 49%|████▉ | 3919/8000 [01:36<18:20, 3.71it/s, lr: 6.0e-05 loss: 5.941e-02] jess: 49%|████▉ | 3919/8000 [01:37<18:20, 3.71it/s, lr: 6.0e-05 loss: 7.162e-02] jess: 49%|████▉ | 3920/8000 [01:37<12:30, 5.44it/s, lr: 6.0e-05 loss: 7.162e-02] jess: 49%|████▉ | 3920/8000 [01:37<12:30, 5.44it/s, lr: 6.0e-05 loss: 1.095e-01] jess: 49%|████▉ | 3921/8000 [01:37<10:46, 6.31it/s, lr: 6.0e-05 loss: 1.095e-01] jess: 49%|████▉ | 3921/8000 [01:37<10:46, 6.31it/s, lr: 6.0e-05 loss: 5.620e-02] jess: 49%|████▉ | 3922/8000 [01:37<16:08, 4.21it/s, lr: 6.0e-05 loss: 5.620e-02] jess: 49%|████▉ | 3922/8000 [01:37<16:08, 4.21it/s, lr: 6.0e-05 loss: 8.341e-02] jess: 49%|████▉ | 3923/8000 [01:37<17:48, 3.82it/s, lr: 6.0e-05 loss: 8.341e-02] jess: 49%|████▉ | 3923/8000 [01:38<17:48, 3.82it/s, lr: 6.0e-05 loss: 2.044e-01] jess: 49%|████▉ | 3924/8000 [01:38<18:18, 3.71it/s, lr: 6.0e-05 loss: 2.044e-01] jess: 49%|████▉ | 3924/8000 [01:38<18:18, 3.71it/s, lr: 6.0e-05 loss: 1.584e-01] jess: 49%|████▉ | 3925/8000 [01:38<12:29, 5.44it/s, lr: 6.0e-05 loss: 1.584e-01] jess: 49%|████▉ | 3925/8000 [01:38<12:29, 5.44it/s, lr: 6.0e-05 loss: 1.470e-01] jess: 49%|████▉ | 3926/8000 [01:38<24:14, 2.80it/s, lr: 6.0e-05 loss: 1.470e-01] jess: 49%|████▉ | 3926/8000 [01:38<24:14, 2.80it/s, lr: 6.0e-05 loss: 2.217e-01] jess: 49%|████▉ | 3927/8000 [01:38<20:16, 3.35it/s, lr: 6.0e-05 loss: 2.217e-01] jess: 49%|████▉ | 3927/8000 [01:39<20:16, 3.35it/s, lr: 6.0e-05 loss: 3.025e-02] jess: 49%|████▉ | 3928/8000 [01:39<13:02, 5.20it/s, lr: 6.0e-05 loss: 3.025e-02] jess: 49%|████▉ | 3928/8000 [01:39<13:02, 5.20it/s, lr: 6.0e-05 loss: 1.462e-01] jess: 49%|████▉ | 3929/8000 [01:39<10:53, 6.23it/s, lr: 6.0e-05 loss: 1.462e-01] jess: 49%|████▉ | 3929/8000 [01:39<10:53, 6.23it/s, lr: 6.0e-05 loss: 4.295e-02] jess: 49%|████▉ | 3930/8000 [01:39<10:13, 6.63it/s, lr: 6.0e-05 loss: 4.295e-02] jess: 49%|████▉ | 3930/8000 [01:39<10:13, 6.63it/s, lr: 6.0e-05 loss: 5.748e-02] jess: 49%|████▉ | 3931/8000 [01:39<15:53, 4.27it/s, lr: 6.0e-05 loss: 5.748e-02] jess: 49%|████▉ | 3931/8000 [01:39<15:53, 4.27it/s, lr: 6.0e-05 loss: 7.925e-02] jess: 49%|████▉ | 3932/8000 [01:39<11:40, 5.81it/s, lr: 6.0e-05 loss: 7.925e-02] jess: 49%|████▉ | 3932/8000 [01:39<11:40, 5.81it/s, lr: 6.0e-05 loss: 1.619e-01] jess: 49%|████▉ | 3933/8000 [01:39<10:49, 6.26it/s, lr: 6.0e-05 loss: 1.619e-01] jess: 49%|████▉ | 3933/8000 [01:40<10:49, 6.26it/s, lr: 6.0e-05 loss: 5.641e-02] jess: 49%|████▉ | 3934/8000 [01:40<14:51, 4.56it/s, lr: 6.0e-05 loss: 5.641e-02] jess: 49%|████▉ | 3934/8000 [01:40<14:51, 4.56it/s, lr: 6.0e-05 loss: 4.946e-02] jess: 49%|████▉ | 3935/8000 [01:40<11:29, 5.90it/s, lr: 6.0e-05 loss: 4.946e-02] jess: 49%|████▉ | 3935/8000 [01:40<11:29, 5.90it/s, lr: 6.0e-05 loss: 5.683e-02] jess: 49%|████▉ | 3936/8000 [01:40<16:18, 4.15it/s, lr: 6.0e-05 loss: 5.683e-02] jess: 49%|████▉ | 3936/8000 [01:40<16:18, 4.15it/s, lr: 6.0e-05 loss: 7.163e-02] jess: 49%|████▉ | 3937/8000 [01:40<17:43, 3.82it/s, lr: 6.0e-05 loss: 7.163e-02] jess: 49%|████▉ | 3937/8000 [01:41<17:43, 3.82it/s, lr: 6.0e-05 loss: 3.039e-02] jess: 49%|████▉ | 3938/8000 [01:41<12:21, 5.48it/s, lr: 6.0e-05 loss: 3.039e-02] jess: 49%|████▉ | 3938/8000 [01:41<12:21, 5.48it/s, lr: 6.0e-05 loss: 1.184e-01] jess: 49%|████▉ | 3939/8000 [01:41<16:29, 4.10it/s, lr: 6.0e-05 loss: 1.184e-01] jess: 49%|████▉ | 3939/8000 [01:41<16:29, 4.10it/s, lr: 6.0e-05 loss: 1.038e-02] jess: 49%|████▉ | 3940/8000 [01:41<17:33, 3.86it/s, lr: 6.0e-05 loss: 1.038e-02] jess: 49%|████▉ | 3940/8000 [01:41<17:33, 3.86it/s, lr: 6.0e-05 loss: 5.363e-02] jess: 49%|████▉ | 3941/8000 [01:41<18:14, 3.71it/s, lr: 6.0e-05 loss: 5.363e-02] jess: 49%|████▉ | 3941/8000 [01:42<18:14, 3.71it/s, lr: 6.0e-05 loss: 7.487e-02] jess: 49%|████▉ | 3942/8000 [01:42<18:19, 3.69it/s, lr: 6.0e-05 loss: 7.487e-02] jess: 49%|████▉ | 3942/8000 [01:42<18:19, 3.69it/s, lr: 6.0e-05 loss: 1.891e-02] jess: 49%|████▉ | 3943/8000 [01:42<12:27, 5.43it/s, lr: 6.0e-05 loss: 1.891e-02] jess: 49%|████▉ | 3943/8000 [01:42<12:27, 5.43it/s, lr: 6.0e-05 loss: 4.178e-02] jess: 49%|████▉ | 3944/8000 [01:42<16:27, 4.11it/s, lr: 6.0e-05 loss: 4.178e-02] jess: 49%|████▉ | 3944/8000 [01:42<16:27, 4.11it/s, lr: 6.0e-05 loss: 2.502e-01] jess: 49%|████▉ | 3945/8000 [01:42<17:48, 3.80it/s, lr: 6.0e-05 loss: 2.502e-01] jess: 49%|████▉ | 3945/8000 [01:42<17:48, 3.80it/s, lr: 6.0e-05 loss: 4.166e-02] jess: 49%|████▉ | 3946/8000 [01:42<12:17, 5.50it/s, lr: 6.0e-05 loss: 4.166e-02] jess: 49%|████▉ | 3946/8000 [01:43<12:17, 5.50it/s, lr: 6.0e-05 loss: 9.147e-03] jess: 49%|████▉ | 3947/8000 [01:43<10:38, 6.35it/s, lr: 6.0e-05 loss: 9.147e-03] jess: 49%|████▉ | 3947/8000 [01:43<10:38, 6.35it/s, lr: 6.0e-05 loss: 1.616e-01] jess: 49%|████▉ | 3948/8000 [01:43<15:52, 4.25it/s, lr: 6.0e-05 loss: 1.616e-01] jess: 49%|████▉ | 3948/8000 [01:43<15:52, 4.25it/s, lr: 6.0e-05 loss: 5.522e-02] jess: 49%|████▉ | 3949/8000 [01:43<11:42, 5.77it/s, lr: 6.0e-05 loss: 5.522e-02] jess: 49%|████▉ | 3949/8000 [01:43<11:42, 5.77it/s, lr: 6.0e-05 loss: 7.241e-02] jess: 49%|████▉ | 3950/8000 [01:43<24:27, 2.76it/s, lr: 6.0e-05 loss: 7.241e-02] jess: 49%|████▉ | 3950/8000 [01:44<24:27, 2.76it/s, lr: 6.0e-05 loss: 8.201e-02]
Saving at step 3950
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000003950.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 49%|████▉ | 3951/8000 [01:44<24:27, 2.76it/s, lr: 6.0e-05 loss: 5.939e-02] jess: 49%|████▉ | 3952/8000 [01:44<19:21, 3.49it/s, lr: 6.0e-05 loss: 5.939e-02] jess: 49%|████▉ | 3952/8000 [01:44<19:21, 3.49it/s, lr: 6.0e-05 loss: 1.083e-01] jess: 49%|████▉ | 3953/8000 [01:44<13:55, 4.84it/s, lr: 6.0e-05 loss: 1.083e-01] jess: 49%|████▉ | 3953/8000 [01:45<13:55, 4.84it/s, lr: 6.0e-05 loss: 1.601e-01] jess: 49%|████▉ | 3954/8000 [01:45<16:41, 4.04it/s, lr: 6.0e-05 loss: 1.601e-01] jess: 49%|████▉ | 3954/8000 [01:45<16:41, 4.04it/s, lr: 6.0e-05 loss: 1.148e-01] jess: 49%|████▉ | 3955/8000 [01:45<12:02, 5.60it/s, lr: 6.0e-05 loss: 1.148e-01] jess: 49%|████▉ | 3955/8000 [01:45<12:02, 5.60it/s, lr: 6.0e-05 loss: 8.746e-02] jess: 49%|████▉ | 3956/8000 [01:45<10:29, 6.42it/s, lr: 6.0e-05 loss: 8.746e-02] jess: 49%|████▉ | 3956/8000 [01:45<10:29, 6.42it/s, lr: 6.0e-05 loss: 1.881e-01] jess: 49%|████▉ | 3957/8000 [01:45<10:04, 6.69it/s, lr: 6.0e-05 loss: 1.881e-01] jess: 49%|████▉ | 3957/8000 [01:45<10:04, 6.69it/s, lr: 6.0e-05 loss: 9.265e-02] jess: 49%|████▉ | 3958/8000 [01:45<09:55, 6.79it/s, lr: 6.0e-05 loss: 9.265e-02] jess: 49%|████▉ | 3958/8000 [01:46<09:55, 6.79it/s, lr: 6.0e-05 loss: 6.333e-02] jess: 49%|████▉ | 3959/8000 [01:46<15:45, 4.27it/s, lr: 6.0e-05 loss: 6.333e-02] jess: 49%|████▉ | 3959/8000 [01:46<15:45, 4.27it/s, lr: 6.0e-05 loss: 2.091e-01] jess: 50%|████▉ | 3960/8000 [01:46<11:36, 5.80it/s, lr: 6.0e-05 loss: 2.091e-01] jess: 50%|████▉ | 3960/8000 [01:46<11:36, 5.80it/s, lr: 6.0e-05 loss: 4.458e-02] jess: 50%|████▉ | 3961/8000 [01:46<16:18, 4.13it/s, lr: 6.0e-05 loss: 4.458e-02] jess: 50%|████▉ | 3961/8000 [01:46<16:18, 4.13it/s, lr: 6.0e-05 loss: 1.243e-01] jess: 50%|████▉ | 3962/8000 [01:46<17:38, 3.81it/s, lr: 6.0e-05 loss: 1.243e-01] jess: 50%|████▉ | 3962/8000 [01:46<17:38, 3.81it/s, lr: 6.0e-05 loss: 3.048e-01] jess: 50%|████▉ | 3963/8000 [01:46<18:04, 3.72it/s, lr: 6.0e-05 loss: 3.048e-01] jess: 50%|████▉ | 3963/8000 [01:47<18:04, 3.72it/s, lr: 6.0e-05 loss: 1.730e-01] jess: 50%|████▉ | 3964/8000 [01:47<18:22, 3.66it/s, lr: 6.0e-05 loss: 1.730e-01] jess: 50%|████▉ | 3964/8000 [01:47<18:22, 3.66it/s, lr: 6.0e-05 loss: 7.973e-02] jess: 50%|████▉ | 3965/8000 [01:47<12:25, 5.41it/s, lr: 6.0e-05 loss: 7.973e-02] jess: 50%|████▉ | 3965/8000 [01:47<12:25, 5.41it/s, lr: 6.0e-05 loss: 1.178e-01] jess: 50%|████▉ | 3966/8000 [01:47<10:35, 6.35it/s, lr: 6.0e-05 loss: 1.178e-01] jess: 50%|████▉ | 3966/8000 [01:47<10:35, 6.35it/s, lr: 6.0e-05 loss: 5.947e-02] jess: 50%|████▉ | 3967/8000 [01:47<15:55, 4.22it/s, lr: 6.0e-05 loss: 5.947e-02] jess: 50%|████▉ | 3967/8000 [01:47<15:55, 4.22it/s, lr: 6.0e-05 loss: 1.459e-01] jess: 50%|████▉ | 3968/8000 [01:47<11:39, 5.76it/s, lr: 6.0e-05 loss: 1.459e-01] jess: 50%|████▉ | 3968/8000 [01:48<11:39, 5.76it/s, lr: 6.0e-05 loss: 1.866e-02] jess: 50%|████▉ | 3969/8000 [01:48<15:32, 4.32it/s, lr: 6.0e-05 loss: 1.866e-02] jess: 50%|████▉ | 3969/8000 [01:48<15:32, 4.32it/s, lr: 6.0e-05 loss: 3.566e-02] jess: 50%|████▉ | 3970/8000 [01:48<11:31, 5.82it/s, lr: 6.0e-05 loss: 3.566e-02] jess: 50%|████▉ | 3970/8000 [01:48<11:31, 5.82it/s, lr: 6.0e-05 loss: 1.840e-01] jess: 50%|████▉ | 3971/8000 [01:48<16:15, 4.13it/s, lr: 6.0e-05 loss: 1.840e-01] jess: 50%|████▉ | 3971/8000 [01:48<16:15, 4.13it/s, lr: 6.0e-05 loss: 2.010e-02] jess: 50%|████▉ | 3972/8000 [01:48<11:46, 5.70it/s, lr: 6.0e-05 loss: 2.010e-02] jess: 50%|████▉ | 3972/8000 [01:49<11:46, 5.70it/s, lr: 6.0e-05 loss: 6.601e-02] jess: 50%|████▉ | 3973/8000 [01:49<16:21, 4.10it/s, lr: 6.0e-05 loss: 6.601e-02] jess: 50%|████▉ | 3973/8000 [01:49<16:21, 4.10it/s, lr: 6.0e-05 loss: 2.319e-01] jess: 50%|████▉ | 3974/8000 [01:49<17:33, 3.82it/s, lr: 6.0e-05 loss: 2.319e-01] jess: 50%|████▉ | 3974/8000 [01:49<17:33, 3.82it/s, lr: 6.0e-05 loss: 2.572e-01] jess: 50%|████▉ | 3975/8000 [01:49<17:59, 3.73it/s, lr: 6.0e-05 loss: 2.572e-01] jess: 50%|████▉ | 3975/8000 [01:50<17:59, 3.73it/s, lr: 6.0e-05 loss: 6.344e-02] jess: 50%|████▉ | 3976/8000 [01:50<30:36, 2.19it/s, lr: 6.0e-05 loss: 6.344e-02] jess: 50%|████▉ | 3976/8000 [01:50<30:36, 2.19it/s, lr: 6.0e-05 loss: 1.708e-02] jess: 50%|████▉ | 3977/8000 [01:50<16:11, 4.14it/s, lr: 6.0e-05 loss: 1.708e-02] jess: 50%|████▉ | 3977/8000 [01:50<16:11, 4.14it/s, lr: 6.0e-05 loss: 1.122e-01] jess: 50%|████▉ | 3978/8000 [01:50<17:36, 3.81it/s, lr: 6.0e-05 loss: 1.122e-01] jess: 50%|████▉ | 3978/8000 [01:50<17:36, 3.81it/s, lr: 6.0e-05 loss: 1.624e-01] jess: 50%|████▉ | 3979/8000 [01:50<12:11, 5.50it/s, lr: 6.0e-05 loss: 1.624e-01] jess: 50%|████▉ | 3979/8000 [01:50<12:11, 5.50it/s, lr: 6.0e-05 loss: 4.413e-02] jess: 50%|████▉ | 3980/8000 [01:50<10:31, 6.37it/s, lr: 6.0e-05 loss: 4.413e-02] jess: 50%|████▉ | 3980/8000 [01:51<10:31, 6.37it/s, lr: 6.0e-05 loss: 1.667e-01] jess: 50%|████▉ | 3981/8000 [01:51<15:56, 4.20it/s, lr: 6.0e-05 loss: 1.667e-01] jess: 50%|████▉ | 3981/8000 [01:51<15:56, 4.20it/s, lr: 6.0e-05 loss: 5.170e-02] jess: 50%|████▉ | 3982/8000 [01:51<11:40, 5.74it/s, lr: 6.0e-05 loss: 5.170e-02] jess: 50%|████▉ | 3982/8000 [01:51<11:40, 5.74it/s, lr: 6.0e-05 loss: 8.454e-02] jess: 50%|████▉ | 3983/8000 [01:51<10:22, 6.46it/s, lr: 6.0e-05 loss: 8.454e-02] jess: 50%|████▉ | 3983/8000 [01:51<10:22, 6.46it/s, lr: 6.0e-05 loss: 3.126e-02] jess: 50%|████▉ | 3984/8000 [01:51<15:32, 4.31it/s, lr: 6.0e-05 loss: 3.126e-02] jess: 50%|████▉ | 3984/8000 [01:51<15:32, 4.31it/s, lr: 6.0e-05 loss: 8.405e-02] jess: 50%|████▉ | 3985/8000 [01:51<17:24, 3.85it/s, lr: 6.0e-05 loss: 8.405e-02] jess: 50%|████▉ | 3985/8000 [01:52<17:24, 3.85it/s, lr: 6.0e-05 loss: 1.278e-01] jess: 50%|████▉ | 3986/8000 [01:52<12:07, 5.52it/s, lr: 6.0e-05 loss: 1.278e-01] jess: 50%|████▉ | 3986/8000 [01:52<12:07, 5.52it/s, lr: 6.0e-05 loss: 9.426e-02] jess: 50%|████▉ | 3987/8000 [01:52<16:20, 4.09it/s, lr: 6.0e-05 loss: 9.426e-02] jess: 50%|████▉ | 3987/8000 [01:52<16:20, 4.09it/s, lr: 6.0e-05 loss: 1.453e-01] jess: 50%|████▉ | 3988/8000 [01:52<11:44, 5.69it/s, lr: 6.0e-05 loss: 1.453e-01] jess: 50%|████▉ | 3988/8000 [01:52<11:44, 5.69it/s, lr: 6.0e-05 loss: 7.989e-02] jess: 50%|████▉ | 3989/8000 [01:52<16:14, 4.12it/s, lr: 6.0e-05 loss: 7.989e-02] jess: 50%|████▉ | 3989/8000 [01:52<16:14, 4.12it/s, lr: 6.0e-05 loss: 6.089e-02] jess: 50%|████▉ | 3990/8000 [01:52<11:43, 5.70it/s, lr: 6.0e-05 loss: 6.089e-02] jess: 50%|████▉ | 3990/8000 [01:53<11:43, 5.70it/s, lr: 6.0e-05 loss: 6.520e-02] jess: 50%|████▉ | 3991/8000 [01:53<10:25, 6.41it/s, lr: 6.0e-05 loss: 6.520e-02] jess: 50%|████▉ | 3991/8000 [01:53<10:25, 6.41it/s, lr: 6.0e-05 loss: 1.413e-01] jess: 50%|████▉ | 3992/8000 [01:53<09:55, 6.73it/s, lr: 6.0e-05 loss: 1.413e-01] jess: 50%|████▉ | 3992/8000 [01:53<09:55, 6.73it/s, lr: 6.0e-05 loss: 1.357e-02] jess: 50%|████▉ | 3993/8000 [01:53<15:31, 4.30it/s, lr: 6.0e-05 loss: 1.357e-02] jess: 50%|████▉ | 3993/8000 [01:53<15:31, 4.30it/s, lr: 6.0e-05 loss: 5.548e-02] jess: 50%|████▉ | 3994/8000 [01:53<17:12, 3.88it/s, lr: 6.0e-05 loss: 5.548e-02] jess: 50%|████▉ | 3994/8000 [01:53<17:12, 3.88it/s, lr: 6.0e-05 loss: 6.060e-02] jess: 50%|████▉ | 3995/8000 [01:53<12:03, 5.53it/s, lr: 6.0e-05 loss: 6.060e-02] jess: 50%|████▉ | 3995/8000 [01:54<12:03, 5.53it/s, lr: 6.0e-05 loss: 1.897e-01] jess: 50%|████▉ | 3996/8000 [01:54<16:17, 4.10it/s, lr: 6.0e-05 loss: 1.897e-01] jess: 50%|████▉ | 3996/8000 [01:54<16:17, 4.10it/s, lr: 6.0e-05 loss: 1.486e-02] jess: 50%|████▉ | 3997/8000 [01:54<11:44, 5.68it/s, lr: 6.0e-05 loss: 1.486e-02] jess: 50%|████▉ | 3997/8000 [01:54<11:44, 5.68it/s, lr: 6.0e-05 loss: 7.335e-02] jess: 50%|████▉ | 3998/8000 [01:54<16:06, 4.14it/s, lr: 6.0e-05 loss: 7.335e-02] jess: 50%|████▉ | 3998/8000 [01:54<16:06, 4.14it/s, lr: 6.0e-05 loss: 8.118e-02] jess: 50%|████▉ | 3999/8000 [01:54<17:29, 3.81it/s, lr: 6.0e-05 loss: 8.118e-02] jess: 50%|████▉ | 3999/8000 [01:55<17:29, 3.81it/s, lr: 6.0e-05 loss: 8.827e-02] jess: 50%|█████ | 4000/8000 [01:55<25:20, 2.63it/s, lr: 6.0e-05 loss: 8.827e-02] jess: 50%|█████ | 4000/8000 [01:55<25:20, 2.63it/s, lr: 6.0e-05 loss: 8.985e-02]
Saving at step 4000
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004000.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.34s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.34s/it]
Generating Images: 100%|██████████| 3/3 [00:04<00:00, 1.35s/it]

jess: 50%|█████ | 4001/8000 [01:56<25:20, 2.63it/s, lr: 6.0e-05 loss: 1.422e-01] jess: 50%|█████ | 4002/8000 [01:56<19:43, 3.38it/s, lr: 6.0e-05 loss: 1.422e-01] jess: 50%|█████ | 4002/8000 [01:56<19:43, 3.38it/s, lr: 6.0e-05 loss: 1.223e-01] jess: 50%|█████ | 4003/8000 [01:56<14:03, 4.74it/s, lr: 6.0e-05 loss: 1.223e-01] jess: 50%|█████ | 4003/8000 [01:56<14:03, 4.74it/s, lr: 6.0e-05 loss: 1.743e-01] jess: 50%|█████ | 4004/8000 [01:56<16:43, 3.98it/s, lr: 6.0e-05 loss: 1.743e-01] jess: 50%|█████ | 4004/8000 [01:56<16:43, 3.98it/s, lr: 6.0e-05 loss: 1.996e-01] jess: 50%|█████ | 4005/8000 [01:56<12:31, 5.32it/s, lr: 6.0e-05 loss: 1.996e-01] jess: 50%|█████ | 4005/8000 [01:56<12:31, 5.32it/s, lr: 6.0e-05 loss: 5.504e-02] jess: 50%|█████ | 4006/8000 [01:56<16:25, 4.05it/s, lr: 6.0e-05 loss: 5.504e-02] jess: 50%|█████ | 4006/8000 [01:57<16:25, 4.05it/s, lr: 6.0e-05 loss: 1.451e-02] jess: 50%|█████ | 4007/8000 [01:57<16:19, 4.08it/s, lr: 6.0e-05 loss: 1.451e-02] jess: 50%|█████ | 4007/8000 [01:57<16:19, 4.08it/s, lr: 6.0e-05 loss: 3.959e-02] jess: 50%|█████ | 4008/8000 [01:57<11:47, 5.64it/s, lr: 6.0e-05 loss: 3.959e-02] jess: 50%|█████ | 4008/8000 [01:57<11:47, 5.64it/s, lr: 6.0e-05 loss: 9.091e-02] jess: 50%|█████ | 4009/8000 [01:57<10:27, 6.36it/s, lr: 6.0e-05 loss: 9.091e-02] jess: 50%|█████ | 4009/8000 [01:57<10:27, 6.36it/s, lr: 6.0e-05 loss: 3.108e-01] jess: 50%|█████ | 4010/8000 [01:57<15:44, 4.22it/s, lr: 6.0e-05 loss: 3.108e-01] jess: 50%|█████ | 4010/8000 [01:57<15:44, 4.22it/s, lr: 6.0e-05 loss: 9.165e-02] jess: 50%|█████ | 4011/8000 [01:57<17:33, 3.79it/s, lr: 6.0e-05 loss: 9.165e-02] jess: 50%|█████ | 4011/8000 [01:58<17:33, 3.79it/s, lr: 6.0e-05 loss: 2.821e-02] jess: 50%|█████ | 4012/8000 [01:58<12:12, 5.44it/s, lr: 6.0e-05 loss: 2.821e-02] jess: 50%|█████ | 4012/8000 [01:58<12:12, 5.44it/s, lr: 6.0e-05 loss: 5.337e-02] jess: 50%|█████ | 4013/8000 [01:58<15:59, 4.15it/s, lr: 6.0e-05 loss: 5.337e-02] jess: 50%|█████ | 4013/8000 [01:58<15:59, 4.15it/s, lr: 6.0e-05 loss: 5.532e-02] jess: 50%|█████ | 4014/8000 [01:58<17:26, 3.81it/s, lr: 6.0e-05 loss: 5.532e-02] jess: 50%|█████ | 4014/8000 [01:58<17:26, 3.81it/s, lr: 6.0e-05 loss: 1.031e-02] jess: 50%|█████ | 4015/8000 [01:58<17:44, 3.74it/s, lr: 6.0e-05 loss: 1.031e-02] jess: 50%|█████ | 4015/8000 [01:59<17:44, 3.74it/s, lr: 6.0e-05 loss: 9.364e-02] jess: 50%|█████ | 4016/8000 [01:59<12:10, 5.45it/s, lr: 6.0e-05 loss: 9.364e-02] jess: 50%|█████ | 4016/8000 [01:59<12:10, 5.45it/s, lr: 6.0e-05 loss: 7.422e-02] jess: 50%|█████ | 4017/8000 [01:59<10:27, 6.35it/s, lr: 6.0e-05 loss: 7.422e-02] jess: 50%|█████ | 4017/8000 [01:59<10:27, 6.35it/s, lr: 6.0e-05 loss: 1.255e-01] jess: 50%|█████ | 4018/8000 [01:59<15:45, 4.21it/s, lr: 6.0e-05 loss: 1.255e-01] jess: 50%|█████ | 4018/8000 [01:59<15:45, 4.21it/s, lr: 6.0e-05 loss: 8.721e-02] jess: 50%|█████ | 4019/8000 [01:59<17:15, 3.84it/s, lr: 6.0e-05 loss: 8.721e-02] jess: 50%|█████ | 4019/8000 [01:59<17:15, 3.84it/s, lr: 6.0e-05 loss: 7.642e-03] jess: 50%|█████ | 4020/8000 [01:59<11:59, 5.53it/s, lr: 6.0e-05 loss: 7.642e-03] jess: 50%|█████ | 4020/8000 [02:00<11:59, 5.53it/s, lr: 6.0e-05 loss: 1.076e-01] jess: 50%|█████ | 4021/8000 [02:00<16:14, 4.08it/s, lr: 6.0e-05 loss: 1.076e-01] jess: 50%|█████ | 4021/8000 [02:00<16:14, 4.08it/s, lr: 6.0e-05 loss: 1.235e-01] jess: 50%|█████ | 4022/8000 [02:00<17:30, 3.79it/s, lr: 6.0e-05 loss: 1.235e-01] jess: 50%|█████ | 4022/8000 [02:00<17:30, 3.79it/s, lr: 6.0e-05 loss: 1.687e-01] jess: 50%|█████ | 4023/8000 [02:00<17:47, 3.73it/s, lr: 6.0e-05 loss: 1.687e-01] jess: 50%|█████ | 4023/8000 [02:00<17:47, 3.73it/s, lr: 6.0e-05 loss: 1.315e-01] jess: 50%|█████ | 4024/8000 [02:00<12:09, 5.45it/s, lr: 6.0e-05 loss: 1.315e-01] jess: 50%|█████ | 4024/8000 [02:01<12:09, 5.45it/s, lr: 6.0e-05 loss: 5.379e-02] jess: 50%|█████ | 4025/8000 [02:01<10:28, 6.32it/s, lr: 6.0e-05 loss: 5.379e-02] jess: 50%|█████ | 4025/8000 [02:01<10:28, 6.32it/s, lr: 6.0e-05 loss: 5.738e-02] jess: 50%|█████ | 4026/8000 [02:01<23:38, 2.80it/s, lr: 6.0e-05 loss: 5.738e-02] jess: 50%|█████ | 4026/8000 [02:01<23:38, 2.80it/s, lr: 6.0e-05 loss: 1.560e-01] jess: 50%|█████ | 4027/8000 [02:01<19:40, 3.37it/s, lr: 6.0e-05 loss: 1.560e-01] jess: 50%|█████ | 4027/8000 [02:01<19:40, 3.37it/s, lr: 6.0e-05 loss: 1.240e-01] jess: 50%|█████ | 4028/8000 [02:01<12:45, 5.19it/s, lr: 6.0e-05 loss: 1.240e-01] jess: 50%|█████ | 4028/8000 [02:02<12:45, 5.19it/s, lr: 6.0e-05 loss: 8.186e-03] jess: 50%|█████ | 4029/8000 [02:02<10:41, 6.19it/s, lr: 6.0e-05 loss: 8.186e-03] jess: 50%|█████ | 4029/8000 [02:02<10:41, 6.19it/s, lr: 6.0e-05 loss: 9.549e-02] jess: 50%|█████ | 4030/8000 [02:02<10:01, 6.61it/s, lr: 6.0e-05 loss: 9.549e-02] jess: 50%|█████ | 4030/8000 [02:02<10:01, 6.61it/s, lr: 6.0e-05 loss: 5.828e-02] jess: 50%|█████ | 4031/8000 [02:02<15:39, 4.23it/s, lr: 6.0e-05 loss: 5.828e-02] jess: 50%|█████ | 4031/8000 [02:02<15:39, 4.23it/s, lr: 6.0e-05 loss: 1.361e-01] jess: 50%|█████ | 4032/8000 [02:02<11:28, 5.77it/s, lr: 6.0e-05 loss: 1.361e-01] jess: 50%|█████ | 4032/8000 [02:02<11:28, 5.77it/s, lr: 6.0e-05 loss: 1.235e-01] jess: 50%|█████ | 4033/8000 [02:02<16:00, 4.13it/s, lr: 6.0e-05 loss: 1.235e-01] jess: 50%|█████ | 4033/8000 [02:03<16:00, 4.13it/s, lr: 6.0e-05 loss: 2.658e-02] jess: 50%|█████ | 4034/8000 [02:03<17:16, 3.83it/s, lr: 6.0e-05 loss: 2.658e-02] jess: 50%|█████ | 4034/8000 [02:03<17:16, 3.83it/s, lr: 6.0e-05 loss: 1.136e-01] jess: 50%|█████ | 4035/8000 [02:03<11:59, 5.51it/s, lr: 6.0e-05 loss: 1.136e-01] jess: 50%|█████ | 4035/8000 [02:03<11:59, 5.51it/s, lr: 6.0e-05 loss: 3.328e-02] jess: 50%|█████ | 4036/8000 [02:03<16:08, 4.09it/s, lr: 6.0e-05 loss: 3.328e-02] jess: 50%|█████ | 4036/8000 [02:03<16:08, 4.09it/s, lr: 6.0e-05 loss: 3.951e-02] jess: 50%|█████ | 4037/8000 [02:03<11:35, 5.69it/s, lr: 6.0e-05 loss: 3.951e-02] jess: 50%|█████ | 4037/8000 [02:03<11:35, 5.69it/s, lr: 6.0e-05 loss: 1.591e-02] jess: 50%|█████ | 4038/8000 [02:03<15:53, 4.16it/s, lr: 6.0e-05 loss: 1.591e-02] jess: 50%|█████ | 4038/8000 [02:04<15:53, 4.16it/s, lr: 6.0e-05 loss: 1.513e-01] jess: 50%|█████ | 4039/8000 [02:04<17:18, 3.82it/s, lr: 6.0e-05 loss: 1.513e-01] jess: 50%|█████ | 4039/8000 [02:04<17:18, 3.82it/s, lr: 6.0e-05 loss: 5.258e-02] jess: 50%|█████ | 4040/8000 [02:04<11:58, 5.51it/s, lr: 6.0e-05 loss: 5.258e-02] jess: 50%|█████ | 4040/8000 [02:04<11:58, 5.51it/s, lr: 6.0e-05 loss: 2.103e-01] jess: 51%|█████ | 4041/8000 [02:04<16:33, 3.99it/s, lr: 6.0e-05 loss: 2.103e-01] jess: 51%|█████ | 4041/8000 [02:04<16:33, 3.99it/s, lr: 6.0e-05 loss: 1.656e-01] jess: 51%|█████ | 4042/8000 [02:04<17:28, 3.77it/s, lr: 6.0e-05 loss: 1.656e-01] jess: 51%|█████ | 4042/8000 [02:05<17:28, 3.77it/s, lr: 6.0e-05 loss: 1.773e-02] jess: 51%|█████ | 4043/8000 [02:05<17:29, 3.77it/s, lr: 6.0e-05 loss: 1.773e-02] jess: 51%|█████ | 4043/8000 [02:05<17:29, 3.77it/s, lr: 6.0e-05 loss: 8.237e-02] jess: 51%|█████ | 4044/8000 [02:05<12:05, 5.45it/s, lr: 6.0e-05 loss: 8.237e-02] jess: 51%|█████ | 4044/8000 [02:05<12:05, 5.45it/s, lr: 6.0e-05 loss: 1.011e-01] jess: 51%|█████ | 4045/8000 [02:05<16:13, 4.06it/s, lr: 6.0e-05 loss: 1.011e-01] jess: 51%|█████ | 4045/8000 [02:05<16:13, 4.06it/s, lr: 6.0e-05 loss: 5.034e-02] jess: 51%|█████ | 4046/8000 [02:05<11:50, 5.57it/s, lr: 6.0e-05 loss: 5.034e-02] jess: 51%|█████ | 4046/8000 [02:05<11:50, 5.57it/s, lr: 6.0e-05 loss: 5.761e-02] jess: 51%|█████ | 4047/8000 [02:05<10:20, 6.37it/s, lr: 6.0e-05 loss: 5.761e-02] jess: 51%|█████ | 4047/8000 [02:06<10:20, 6.37it/s, lr: 6.0e-05 loss: 9.665e-03] jess: 51%|█████ | 4048/8000 [02:06<14:19, 4.60it/s, lr: 6.0e-05 loss: 9.665e-03] jess: 51%|█████ | 4048/8000 [02:06<14:19, 4.60it/s, lr: 6.0e-05 loss: 9.704e-02] jess: 51%|█████ | 4049/8000 [02:06<11:04, 5.94it/s, lr: 6.0e-05 loss: 9.704e-02] jess: 51%|█████ | 4049/8000 [02:06<11:04, 5.94it/s, lr: 6.0e-05 loss: 2.338e-01] jess: 51%|█████ | 4050/8000 [02:06<28:35, 2.30it/s, lr: 6.0e-05 loss: 2.338e-01] jess: 51%|█████ | 4050/8000 [02:07<28:35, 2.30it/s, lr: 6.0e-05 loss: 6.380e-03]
Saving at step 4050
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004050.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 51%|█████ | 4051/8000 [02:07<28:35, 2.30it/s, lr: 6.0e-05 loss: 7.601e-02] jess: 51%|█████ | 4052/8000 [02:07<16:57, 3.88it/s, lr: 6.0e-05 loss: 7.601e-02] jess: 51%|█████ | 4052/8000 [02:07<16:57, 3.88it/s, lr: 6.0e-05 loss: 9.011e-02] jess: 51%|█████ | 4053/8000 [02:07<17:25, 3.78it/s, lr: 6.0e-05 loss: 9.011e-02] jess: 51%|█████ | 4053/8000 [02:07<17:25, 3.78it/s, lr: 6.0e-05 loss: 1.389e-01] jess: 51%|█████ | 4054/8000 [02:07<17:41, 3.72it/s, lr: 6.0e-05 loss: 1.389e-01] jess: 51%|█████ | 4054/8000 [02:08<17:41, 3.72it/s, lr: 6.0e-05 loss: 2.328e-01] jess: 51%|█████ | 4055/8000 [02:08<12:11, 5.39it/s, lr: 6.0e-05 loss: 2.328e-01] jess: 51%|█████ | 4055/8000 [02:08<12:11, 5.39it/s, lr: 6.0e-05 loss: 1.289e-01] jess: 51%|█████ | 4056/8000 [02:08<10:30, 6.26it/s, lr: 6.0e-05 loss: 1.289e-01] jess: 51%|█████ | 4056/8000 [02:08<10:30, 6.26it/s, lr: 6.0e-05 loss: 1.545e-01] jess: 51%|█████ | 4057/8000 [02:08<15:40, 4.19it/s, lr: 6.0e-05 loss: 1.545e-01] jess: 51%|█████ | 4057/8000 [02:08<15:40, 4.19it/s, lr: 6.0e-05 loss: 1.213e-01] jess: 51%|█████ | 4058/8000 [02:08<11:33, 5.68it/s, lr: 6.0e-05 loss: 1.213e-01] jess: 51%|█████ | 4058/8000 [02:08<11:33, 5.68it/s, lr: 6.0e-05 loss: 1.825e-01] jess: 51%|█████ | 4059/8000 [02:08<15:54, 4.13it/s, lr: 6.0e-05 loss: 1.825e-01] jess: 51%|█████ | 4059/8000 [02:09<15:54, 4.13it/s, lr: 6.0e-05 loss: 1.119e-01] jess: 51%|█████ | 4060/8000 [02:09<11:34, 5.67it/s, lr: 6.0e-05 loss: 1.119e-01] jess: 51%|█████ | 4060/8000 [02:09<11:34, 5.67it/s, lr: 6.0e-05 loss: 8.673e-02] jess: 51%|█████ | 4061/8000 [02:09<10:17, 6.37it/s, lr: 6.0e-05 loss: 8.673e-02] jess: 51%|█████ | 4061/8000 [02:09<10:17, 6.37it/s, lr: 6.0e-05 loss: 9.365e-02] jess: 51%|█████ | 4062/8000 [02:09<09:51, 6.66it/s, lr: 6.0e-05 loss: 9.365e-02] jess: 51%|█████ | 4062/8000 [02:09<09:51, 6.66it/s, lr: 6.0e-05 loss: 9.116e-02] jess: 51%|█████ | 4063/8000 [02:09<15:28, 4.24it/s, lr: 6.0e-05 loss: 9.116e-02] jess: 51%|█████ | 4063/8000 [02:09<15:28, 4.24it/s, lr: 6.0e-05 loss: 1.355e-01] jess: 51%|█████ | 4064/8000 [02:09<11:25, 5.74it/s, lr: 6.0e-05 loss: 1.355e-01] jess: 51%|█████ | 4064/8000 [02:10<11:25, 5.74it/s, lr: 6.0e-05 loss: 2.054e-01] jess: 51%|█████ | 4065/8000 [02:10<15:56, 4.12it/s, lr: 6.0e-05 loss: 2.054e-01] jess: 51%|█████ | 4065/8000 [02:10<15:56, 4.12it/s, lr: 6.0e-05 loss: 3.211e-03] jess: 51%|█████ | 4066/8000 [02:10<17:09, 3.82it/s, lr: 6.0e-05 loss: 3.211e-03] jess: 51%|█████ | 4066/8000 [02:10<17:09, 3.82it/s, lr: 6.0e-05 loss: 4.967e-02] jess: 51%|█████ | 4067/8000 [02:10<11:54, 5.50it/s, lr: 6.0e-05 loss: 4.967e-02] jess: 51%|█████ | 4067/8000 [02:10<11:54, 5.50it/s, lr: 6.0e-05 loss: 1.490e-01] jess: 51%|█████ | 4068/8000 [02:10<10:21, 6.33it/s, lr: 6.0e-05 loss: 1.490e-01] jess: 51%|█████ | 4068/8000 [02:10<10:21, 6.33it/s, lr: 6.0e-05 loss: 6.917e-02] jess: 51%|█████ | 4069/8000 [02:10<15:34, 4.20it/s, lr: 6.0e-05 loss: 6.917e-02] jess: 51%|█████ | 4069/8000 [02:11<15:34, 4.20it/s, lr: 6.0e-05 loss: 1.078e-01] jess: 51%|█████ | 4070/8000 [02:11<17:07, 3.82it/s, lr: 6.0e-05 loss: 1.078e-01] jess: 51%|█████ | 4070/8000 [02:11<17:07, 3.82it/s, lr: 6.0e-05 loss: 7.176e-03] jess: 51%|█████ | 4071/8000 [02:11<11:54, 5.50it/s, lr: 6.0e-05 loss: 7.176e-03] jess: 51%|█████ | 4071/8000 [02:11<11:54, 5.50it/s, lr: 6.0e-05 loss: 8.459e-02] jess: 51%|█████ | 4072/8000 [02:11<10:18, 6.35it/s, lr: 6.0e-05 loss: 8.459e-02] jess: 51%|█████ | 4072/8000 [02:11<10:18, 6.35it/s, lr: 6.0e-05 loss: 1.290e-01] jess: 51%|█████ | 4073/8000 [02:11<15:33, 4.21it/s, lr: 6.0e-05 loss: 1.290e-01] jess: 51%|█████ | 4073/8000 [02:12<15:33, 4.21it/s, lr: 6.0e-05 loss: 5.884e-02] jess: 51%|█████ | 4074/8000 [02:12<16:59, 3.85it/s, lr: 6.0e-05 loss: 5.884e-02] jess: 51%|█████ | 4074/8000 [02:12<16:59, 3.85it/s, lr: 6.0e-05 loss: 1.515e-01] jess: 51%|█████ | 4075/8000 [02:12<17:31, 3.73it/s, lr: 6.0e-05 loss: 1.515e-01] jess: 51%|█████ | 4075/8000 [02:12<17:31, 3.73it/s, lr: 6.0e-05 loss: 6.129e-02] jess: 51%|█████ | 4076/8000 [02:12<30:38, 2.13it/s, lr: 6.0e-05 loss: 6.129e-02] jess: 51%|█████ | 4076/8000 [02:13<30:38, 2.13it/s, lr: 6.0e-05 loss: 1.105e-01] jess: 51%|█████ | 4077/8000 [02:13<22:14, 2.94it/s, lr: 6.0e-05 loss: 1.105e-01] jess: 51%|█████ | 4077/8000 [02:13<22:14, 2.94it/s, lr: 6.0e-05 loss: 5.940e-02] jess: 51%|█████ | 4078/8000 [02:13<13:22, 4.89it/s, lr: 6.0e-05 loss: 5.940e-02] jess: 51%|█████ | 4078/8000 [02:13<13:22, 4.89it/s, lr: 6.0e-05 loss: 2.100e-02] jess: 51%|█████ | 4079/8000 [02:13<10:47, 6.05it/s, lr: 6.0e-05 loss: 2.100e-02] jess: 51%|█████ | 4079/8000 [02:13<10:47, 6.05it/s, lr: 6.0e-05 loss: 9.385e-02] jess: 51%|█████ | 4080/8000 [02:13<15:29, 4.22it/s, lr: 6.0e-05 loss: 9.385e-02] jess: 51%|█████ | 4080/8000 [02:13<15:29, 4.22it/s, lr: 6.0e-05 loss: 1.089e-01] jess: 51%|█████ | 4081/8000 [02:13<17:03, 3.83it/s, lr: 6.0e-05 loss: 1.089e-01] jess: 51%|█████ | 4081/8000 [02:14<17:03, 3.83it/s, lr: 6.0e-05 loss: 5.601e-02] jess: 51%|█████ | 4082/8000 [02:14<17:35, 3.71it/s, lr: 6.0e-05 loss: 5.601e-02] jess: 51%|█████ | 4082/8000 [02:14<17:35, 3.71it/s, lr: 6.0e-05 loss: 9.910e-02] jess: 51%|█████ | 4083/8000 [02:14<12:02, 5.42it/s, lr: 6.0e-05 loss: 9.910e-02] jess: 51%|█████ | 4083/8000 [02:14<12:02, 5.42it/s, lr: 6.0e-05 loss: 2.634e-01] jess: 51%|█████ | 4084/8000 [02:14<10:19, 6.32it/s, lr: 6.0e-05 loss: 2.634e-01] jess: 51%|█████ | 4084/8000 [02:14<10:19, 6.32it/s, lr: 6.0e-05 loss: 1.255e-01] jess: 51%|█████ | 4085/8000 [02:14<09:48, 6.65it/s, lr: 6.0e-05 loss: 1.255e-01] jess: 51%|█████ | 4085/8000 [02:14<09:48, 6.65it/s, lr: 6.0e-05 loss: 1.247e-01] jess: 51%|█████ | 4086/8000 [02:14<09:40, 6.74it/s, lr: 6.0e-05 loss: 1.247e-01] jess: 51%|█████ | 4086/8000 [02:15<09:40, 6.74it/s, lr: 6.0e-05 loss: 1.877e-01] jess: 51%|█████ | 4087/8000 [02:15<09:36, 6.79it/s, lr: 6.0e-05 loss: 1.877e-01] jess: 51%|█████ | 4087/8000 [02:15<09:36, 6.79it/s, lr: 6.0e-05 loss: 8.412e-02] jess: 51%|█████ | 4088/8000 [02:15<09:35, 6.80it/s, lr: 6.0e-05 loss: 8.412e-02] jess: 51%|█████ | 4088/8000 [02:15<09:35, 6.80it/s, lr: 6.0e-05 loss: 8.843e-02] jess: 51%|█████ | 4089/8000 [02:15<09:35, 6.80it/s, lr: 6.0e-05 loss: 8.843e-02] jess: 51%|█████ | 4089/8000 [02:15<09:35, 6.80it/s, lr: 6.0e-05 loss: 1.093e-01] jess: 51%|█████ | 4090/8000 [02:15<15:16, 4.27it/s, lr: 6.0e-05 loss: 1.093e-01] jess: 51%|█████ | 4090/8000 [02:15<15:16, 4.27it/s, lr: 6.0e-05 loss: 1.183e-01] jess: 51%|█████ | 4091/8000 [02:15<11:18, 5.76it/s, lr: 6.0e-05 loss: 1.183e-01] jess: 51%|█████ | 4091/8000 [02:15<11:18, 5.76it/s, lr: 6.0e-05 loss: 8.764e-02] jess: 51%|█████ | 4092/8000 [02:15<10:02, 6.48it/s, lr: 6.0e-05 loss: 8.764e-02] jess: 51%|█████ | 4092/8000 [02:16<10:02, 6.48it/s, lr: 6.0e-05 loss: 2.344e-02] jess: 51%|█████ | 4093/8000 [02:16<15:07, 4.30it/s, lr: 6.0e-05 loss: 2.344e-02] jess: 51%|█████ | 4093/8000 [02:16<15:07, 4.30it/s, lr: 6.0e-05 loss: 3.655e-03] jess: 51%|█████ | 4094/8000 [02:16<11:10, 5.83it/s, lr: 6.0e-05 loss: 3.655e-03] jess: 51%|█████ | 4094/8000 [02:16<11:10, 5.83it/s, lr: 6.0e-05 loss: 1.453e-01] jess: 51%|█████ | 4095/8000 [02:16<15:39, 4.16it/s, lr: 6.0e-05 loss: 1.453e-01] jess: 51%|█████ | 4095/8000 [02:16<15:39, 4.16it/s, lr: 6.0e-05 loss: 2.063e-03] jess: 51%|█████ | 4096/8000 [02:16<11:27, 5.68it/s, lr: 6.0e-05 loss: 2.063e-03] jess: 51%|█████ | 4096/8000 [02:16<11:27, 5.68it/s, lr: 6.0e-05 loss: 6.400e-02] jess: 51%|█████ | 4097/8000 [02:16<10:08, 6.42it/s, lr: 6.0e-05 loss: 6.400e-02] jess: 51%|█████ | 4097/8000 [02:16<10:08, 6.42it/s, lr: 6.0e-05 loss: 1.106e-01] jess: 51%|█████ | 4098/8000 [02:16<09:40, 6.72it/s, lr: 6.0e-05 loss: 1.106e-01] jess: 51%|█████ | 4098/8000 [02:17<09:40, 6.72it/s, lr: 6.0e-05 loss: 4.011e-02] jess: 51%|█████ | 4099/8000 [02:17<15:17, 4.25it/s, lr: 6.0e-05 loss: 4.011e-02] jess: 51%|█████ | 4099/8000 [02:17<15:17, 4.25it/s, lr: 6.0e-05 loss: 1.297e-01] jess: 51%|█████▏ | 4100/8000 [02:17<29:41, 2.19it/s, lr: 6.0e-05 loss: 1.297e-01] jess: 51%|█████▏ | 4100/8000 [02:18<29:41, 2.19it/s, lr: 6.0e-05 loss: 7.813e-02]
Saving at step 4100
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004100.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.34s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.34s/it]
Generating Images: 100%|██████████| 3/3 [00:04<00:00, 1.35s/it]

jess: 51%|█████▏ | 4101/8000 [02:18<29:41, 2.19it/s, lr: 6.0e-05 loss: 1.663e-01] jess: 51%|█████▏ | 4102/8000 [02:18<19:58, 3.25it/s, lr: 6.0e-05 loss: 1.663e-01] jess: 51%|█████▏ | 4102/8000 [02:18<19:58, 3.25it/s, lr: 6.0e-05 loss: 2.143e-02] jess: 51%|█████▏ | 4103/8000 [02:18<18:34, 3.50it/s, lr: 6.0e-05 loss: 2.143e-02] jess: 51%|█████▏ | 4103/8000 [02:19<18:34, 3.50it/s, lr: 6.0e-05 loss: 3.975e-02] jess: 51%|█████▏ | 4104/8000 [02:19<12:40, 5.12it/s, lr: 6.0e-05 loss: 3.975e-02] jess: 51%|█████▏ | 4104/8000 [02:19<12:40, 5.12it/s, lr: 6.0e-05 loss: 7.565e-02] jess: 51%|█████▏ | 4105/8000 [02:19<16:04, 4.04it/s, lr: 6.0e-05 loss: 7.565e-02] jess: 51%|█████▏ | 4105/8000 [02:19<16:04, 4.04it/s, lr: 6.0e-05 loss: 1.648e-01] jess: 51%|█████▏ | 4106/8000 [02:19<17:03, 3.80it/s, lr: 6.0e-05 loss: 1.648e-01] jess: 51%|█████▏ | 4106/8000 [02:19<17:03, 3.80it/s, lr: 6.0e-05 loss: 7.844e-02] jess: 51%|█████▏ | 4107/8000 [02:19<17:31, 3.70it/s, lr: 6.0e-05 loss: 7.844e-02] jess: 51%|█████▏ | 4107/8000 [02:20<17:31, 3.70it/s, lr: 6.0e-05 loss: 1.104e-01] jess: 51%|█████▏ | 4108/8000 [02:20<12:00, 5.40it/s, lr: 6.0e-05 loss: 1.104e-01] jess: 51%|█████▏ | 4108/8000 [02:20<12:00, 5.40it/s, lr: 6.0e-05 loss: 1.911e-01] jess: 51%|█████▏ | 4109/8000 [02:20<15:58, 4.06it/s, lr: 6.0e-05 loss: 1.911e-01] jess: 51%|█████▏ | 4109/8000 [02:20<15:58, 4.06it/s, lr: 6.0e-05 loss: 8.718e-03] jess: 51%|█████▏ | 4110/8000 [02:20<15:52, 4.09it/s, lr: 6.0e-05 loss: 8.718e-03] jess: 51%|█████▏ | 4110/8000 [02:20<15:52, 4.09it/s, lr: 6.0e-05 loss: 1.531e-01] jess: 51%|█████▏ | 4111/8000 [02:20<17:09, 3.78it/s, lr: 6.0e-05 loss: 1.531e-01] jess: 51%|█████▏ | 4111/8000 [02:21<17:09, 3.78it/s, lr: 6.0e-05 loss: 1.617e-01] jess: 51%|█████▏ | 4112/8000 [02:21<17:18, 3.74it/s, lr: 6.0e-05 loss: 1.617e-01] jess: 51%|█████▏ | 4112/8000 [02:21<17:18, 3.74it/s, lr: 6.0e-05 loss: 4.652e-03] jess: 51%|█████▏ | 4113/8000 [02:21<12:18, 5.27it/s, lr: 6.0e-05 loss: 4.652e-03] jess: 51%|█████▏ | 4113/8000 [02:21<12:18, 5.27it/s, lr: 6.0e-05 loss: 9.646e-02] jess: 51%|█████▏ | 4114/8000 [02:21<10:17, 6.30it/s, lr: 6.0e-05 loss: 9.646e-02] jess: 51%|█████▏ | 4114/8000 [02:21<10:17, 6.30it/s, lr: 6.0e-05 loss: 8.980e-02] jess: 51%|█████▏ | 4115/8000 [02:21<09:44, 6.65it/s, lr: 6.0e-05 loss: 8.980e-02] jess: 51%|█████▏ | 4115/8000 [02:21<09:44, 6.65it/s, lr: 6.0e-05 loss: 3.628e-02] jess: 51%|█████▏ | 4116/8000 [02:21<09:37, 6.72it/s, lr: 6.0e-05 loss: 3.628e-02] jess: 51%|█████▏ | 4116/8000 [02:21<09:37, 6.72it/s, lr: 6.0e-05 loss: 3.410e-01] jess: 51%|█████▏ | 4117/8000 [02:21<15:07, 4.28it/s, lr: 6.0e-05 loss: 3.410e-01] jess: 51%|█████▏ | 4117/8000 [02:22<15:07, 4.28it/s, lr: 6.0e-05 loss: 8.061e-02] jess: 51%|█████▏ | 4118/8000 [02:22<11:18, 5.72it/s, lr: 6.0e-05 loss: 8.061e-02] jess: 51%|█████▏ | 4118/8000 [02:22<11:18, 5.72it/s, lr: 6.0e-05 loss: 1.754e-01] jess: 51%|█████▏ | 4119/8000 [02:22<10:10, 6.36it/s, lr: 6.0e-05 loss: 1.754e-01] jess: 51%|█████▏ | 4119/8000 [02:22<10:10, 6.36it/s, lr: 6.0e-05 loss: 1.755e-01] jess: 52%|█████▏ | 4120/8000 [02:22<15:21, 4.21it/s, lr: 6.0e-05 loss: 1.755e-01] jess: 52%|█████▏ | 4120/8000 [02:22<15:21, 4.21it/s, lr: 6.0e-05 loss: 1.997e-02] jess: 52%|█████▏ | 4121/8000 [02:22<15:41, 4.12it/s, lr: 6.0e-05 loss: 1.997e-02] jess: 52%|█████▏ | 4121/8000 [02:23<15:41, 4.12it/s, lr: 6.0e-05 loss: 1.400e-01] jess: 52%|█████▏ | 4122/8000 [02:23<16:57, 3.81it/s, lr: 6.0e-05 loss: 1.400e-01] jess: 52%|█████▏ | 4122/8000 [02:23<16:57, 3.81it/s, lr: 6.0e-05 loss: 1.283e-01] jess: 52%|█████▏ | 4123/8000 [02:23<11:43, 5.51it/s, lr: 6.0e-05 loss: 1.283e-01] jess: 52%|█████▏ | 4123/8000 [02:23<11:43, 5.51it/s, lr: 6.0e-05 loss: 6.689e-02] jess: 52%|█████▏ | 4124/8000 [02:23<15:38, 4.13it/s, lr: 6.0e-05 loss: 6.689e-02] jess: 52%|█████▏ | 4124/8000 [02:23<15:38, 4.13it/s, lr: 6.0e-05 loss: 2.273e-01] jess: 52%|█████▏ | 4125/8000 [02:23<16:59, 3.80it/s, lr: 6.0e-05 loss: 2.273e-01] jess: 52%|█████▏ | 4125/8000 [02:24<16:59, 3.80it/s, lr: 6.0e-05 loss: 2.542e-02] jess: 52%|█████▏ | 4126/8000 [02:24<25:01, 2.58it/s, lr: 6.0e-05 loss: 2.542e-02] jess: 52%|█████▏ | 4126/8000 [02:24<25:01, 2.58it/s, lr: 6.0e-05 loss: 1.735e-01] jess: 52%|█████▏ | 4127/8000 [02:24<14:13, 4.54it/s, lr: 6.0e-05 loss: 1.735e-01] jess: 52%|█████▏ | 4127/8000 [02:24<14:13, 4.54it/s, lr: 6.0e-05 loss: 2.228e-01] jess: 52%|█████▏ | 4128/8000 [02:24<16:34, 3.90it/s, lr: 6.0e-05 loss: 2.228e-01] jess: 52%|█████▏ | 4128/8000 [02:24<16:34, 3.90it/s, lr: 6.0e-05 loss: 1.177e-01] jess: 52%|█████▏ | 4129/8000 [02:24<17:09, 3.76it/s, lr: 6.0e-05 loss: 1.177e-01] jess: 52%|█████▏ | 4129/8000 [02:25<17:09, 3.76it/s, lr: 6.0e-05 loss: 8.001e-02] jess: 52%|█████▏ | 4130/8000 [02:25<17:18, 3.72it/s, lr: 6.0e-05 loss: 8.001e-02] jess: 52%|█████▏ | 4130/8000 [02:25<17:18, 3.72it/s, lr: 6.0e-05 loss: 1.814e-01] jess: 52%|█████▏ | 4131/8000 [02:25<17:30, 3.68it/s, lr: 6.0e-05 loss: 1.814e-01] jess: 52%|█████▏ | 4131/8000 [02:25<17:30, 3.68it/s, lr: 6.0e-05 loss: 1.350e-01] jess: 52%|█████▏ | 4132/8000 [02:25<11:55, 5.41it/s, lr: 6.0e-05 loss: 1.350e-01] jess: 52%|█████▏ | 4132/8000 [02:25<11:55, 5.41it/s, lr: 6.0e-05 loss: 2.324e-01] jess: 52%|█████▏ | 4133/8000 [02:25<15:50, 4.07it/s, lr: 6.0e-05 loss: 2.324e-01] jess: 52%|█████▏ | 4133/8000 [02:25<15:50, 4.07it/s, lr: 6.0e-05 loss: 2.088e-03] jess: 52%|█████▏ | 4134/8000 [02:25<11:34, 5.56it/s, lr: 6.0e-05 loss: 2.088e-03] jess: 52%|█████▏ | 4134/8000 [02:26<11:34, 5.56it/s, lr: 6.0e-05 loss: 1.082e-01] jess: 52%|█████▏ | 4135/8000 [02:26<15:42, 4.10it/s, lr: 6.0e-05 loss: 1.082e-01] jess: 52%|█████▏ | 4135/8000 [02:26<15:42, 4.10it/s, lr: 6.0e-05 loss: 1.271e-01] jess: 52%|█████▏ | 4136/8000 [02:26<16:47, 3.83it/s, lr: 6.0e-05 loss: 1.271e-01] jess: 52%|█████▏ | 4136/8000 [02:26<16:47, 3.83it/s, lr: 6.0e-05 loss: 3.565e-02] jess: 52%|█████▏ | 4137/8000 [02:26<17:18, 3.72it/s, lr: 6.0e-05 loss: 3.565e-02] jess: 52%|█████▏ | 4137/8000 [02:27<17:18, 3.72it/s, lr: 6.0e-05 loss: 6.161e-02] jess: 52%|█████▏ | 4138/8000 [02:27<17:26, 3.69it/s, lr: 6.0e-05 loss: 6.161e-02] jess: 52%|█████▏ | 4138/8000 [02:27<17:26, 3.69it/s, lr: 6.0e-05 loss: 6.570e-02] jess: 52%|█████▏ | 4139/8000 [02:27<11:52, 5.42it/s, lr: 6.0e-05 loss: 6.570e-02] jess: 52%|█████▏ | 4139/8000 [02:27<11:52, 5.42it/s, lr: 6.0e-05 loss: 1.395e-01] jess: 52%|█████▏ | 4140/8000 [02:27<10:10, 6.32it/s, lr: 6.0e-05 loss: 1.395e-01] jess: 52%|█████▏ | 4140/8000 [02:27<10:10, 6.32it/s, lr: 6.0e-05 loss: 8.327e-02] jess: 52%|█████▏ | 4141/8000 [02:27<15:47, 4.07it/s, lr: 6.0e-05 loss: 8.327e-02] jess: 52%|█████▏ | 4141/8000 [02:27<15:47, 4.07it/s, lr: 6.0e-05 loss: 5.242e-02] jess: 52%|█████▏ | 4142/8000 [02:27<11:18, 5.69it/s, lr: 6.0e-05 loss: 5.242e-02] jess: 52%|█████▏ | 4142/8000 [02:28<11:18, 5.69it/s, lr: 6.0e-05 loss: 8.315e-03] jess: 52%|█████▏ | 4143/8000 [02:28<15:18, 4.20it/s, lr: 6.0e-05 loss: 8.315e-03] jess: 52%|█████▏ | 4143/8000 [02:28<15:18, 4.20it/s, lr: 6.0e-05 loss: 1.138e-01] jess: 52%|█████▏ | 4144/8000 [02:28<16:39, 3.86it/s, lr: 6.0e-05 loss: 1.138e-01] jess: 52%|█████▏ | 4144/8000 [02:28<16:39, 3.86it/s, lr: 6.0e-05 loss: 1.887e-01] jess: 52%|█████▏ | 4145/8000 [02:28<17:12, 3.73it/s, lr: 6.0e-05 loss: 1.887e-01] jess: 52%|█████▏ | 4145/8000 [02:28<17:12, 3.73it/s, lr: 6.0e-05 loss: 1.045e-01] jess: 52%|█████▏ | 4146/8000 [02:28<11:45, 5.46it/s, lr: 6.0e-05 loss: 1.045e-01] jess: 52%|█████▏ | 4146/8000 [02:28<11:45, 5.46it/s, lr: 6.0e-05 loss: 1.836e-01] jess: 52%|█████▏ | 4147/8000 [02:28<10:09, 6.32it/s, lr: 6.0e-05 loss: 1.836e-01] jess: 52%|█████▏ | 4147/8000 [02:29<10:09, 6.32it/s, lr: 6.0e-05 loss: 1.863e-01] jess: 52%|█████▏ | 4148/8000 [02:29<09:37, 6.67it/s, lr: 6.0e-05 loss: 1.863e-01] jess: 52%|█████▏ | 4148/8000 [02:29<09:37, 6.67it/s, lr: 6.0e-05 loss: 2.063e-01] jess: 52%|█████▏ | 4149/8000 [02:29<15:31, 4.13it/s, lr: 6.0e-05 loss: 2.063e-01] jess: 52%|█████▏ | 4149/8000 [02:29<15:31, 4.13it/s, lr: 6.0e-05 loss: 1.019e-01] jess: 52%|█████▏ | 4150/8000 [02:29<29:10, 2.20it/s, lr: 6.0e-05 loss: 1.019e-01] jess: 52%|█████▏ | 4150/8000 [02:29<29:10, 2.20it/s, lr: 6.0e-05 loss: 4.983e-02]
Saving at step 4150
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004150.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 52%|█████▏ | 4151/8000 [02:30<29:10, 2.20it/s, lr: 6.0e-05 loss: 9.879e-02] jess: 52%|█████▏ | 4152/8000 [02:30<19:49, 3.24it/s, lr: 6.0e-05 loss: 9.879e-02] jess: 52%|█████▏ | 4152/8000 [02:30<19:49, 3.24it/s, lr: 6.0e-05 loss: 1.813e-01] jess: 52%|█████▏ | 4153/8000 [02:30<18:25, 3.48it/s, lr: 6.0e-05 loss: 1.813e-01] jess: 52%|█████▏ | 4153/8000 [02:31<18:25, 3.48it/s, lr: 6.0e-05 loss: 1.527e-01] jess: 52%|█████▏ | 4154/8000 [02:31<17:39, 3.63it/s, lr: 6.0e-05 loss: 1.527e-01] jess: 52%|█████▏ | 4154/8000 [02:31<17:39, 3.63it/s, lr: 6.0e-05 loss: 1.031e-02] jess: 52%|█████▏ | 4155/8000 [02:31<16:18, 3.93it/s, lr: 6.0e-05 loss: 1.031e-02] jess: 52%|█████▏ | 4155/8000 [02:31<16:18, 3.93it/s, lr: 6.0e-05 loss: 8.138e-02] jess: 52%|█████▏ | 4156/8000 [02:31<11:32, 5.55it/s, lr: 6.0e-05 loss: 8.138e-02] jess: 52%|█████▏ | 4156/8000 [02:31<11:32, 5.55it/s, lr: 6.0e-05 loss: 8.536e-02] jess: 52%|█████▏ | 4157/8000 [02:31<15:37, 4.10it/s, lr: 6.0e-05 loss: 8.536e-02] jess: 52%|█████▏ | 4157/8000 [02:31<15:37, 4.10it/s, lr: 6.0e-05 loss: 1.865e-01] jess: 52%|█████▏ | 4158/8000 [02:31<11:16, 5.68it/s, lr: 6.0e-05 loss: 1.865e-01] jess: 52%|█████▏ | 4158/8000 [02:32<11:16, 5.68it/s, lr: 6.0e-05 loss: 4.638e-02] jess: 52%|█████▏ | 4159/8000 [02:32<15:27, 4.14it/s, lr: 6.0e-05 loss: 4.638e-02] jess: 52%|█████▏ | 4159/8000 [02:32<15:27, 4.14it/s, lr: 6.0e-05 loss: 8.817e-02] jess: 52%|█████▏ | 4160/8000 [02:32<16:55, 3.78it/s, lr: 6.0e-05 loss: 8.817e-02] jess: 52%|█████▏ | 4160/8000 [02:32<16:55, 3.78it/s, lr: 6.0e-05 loss: 7.008e-02] jess: 52%|█████▏ | 4161/8000 [02:32<11:42, 5.47it/s, lr: 6.0e-05 loss: 7.008e-02] jess: 52%|█████▏ | 4161/8000 [02:32<11:42, 5.47it/s, lr: 6.0e-05 loss: 2.825e-01] jess: 52%|█████▏ | 4162/8000 [02:32<10:01, 6.39it/s, lr: 6.0e-05 loss: 2.825e-01] jess: 52%|█████▏ | 4162/8000 [02:33<10:01, 6.39it/s, lr: 6.0e-05 loss: 1.426e-01] jess: 52%|█████▏ | 4163/8000 [02:33<15:10, 4.22it/s, lr: 6.0e-05 loss: 1.426e-01] jess: 52%|█████▏ | 4163/8000 [02:33<15:10, 4.22it/s, lr: 6.0e-05 loss: 8.949e-02] jess: 52%|█████▏ | 4164/8000 [02:33<16:40, 3.83it/s, lr: 6.0e-05 loss: 8.949e-02] jess: 52%|█████▏ | 4164/8000 [02:33<16:40, 3.83it/s, lr: 6.0e-05 loss: 2.532e-02] jess: 52%|█████▏ | 4165/8000 [02:33<11:34, 5.52it/s, lr: 6.0e-05 loss: 2.532e-02] jess: 52%|█████▏ | 4165/8000 [02:33<11:34, 5.52it/s, lr: 6.0e-05 loss: 2.114e-01] jess: 52%|█████▏ | 4166/8000 [02:33<10:03, 6.36it/s, lr: 6.0e-05 loss: 2.114e-01] jess: 52%|█████▏ | 4166/8000 [02:33<10:03, 6.36it/s, lr: 6.0e-05 loss: 1.072e-01] jess: 52%|█████▏ | 4167/8000 [02:33<09:33, 6.68it/s, lr: 6.0e-05 loss: 1.072e-01] jess: 52%|█████▏ | 4167/8000 [02:33<09:33, 6.68it/s, lr: 6.0e-05 loss: 1.586e-01] jess: 52%|█████▏ | 4168/8000 [02:33<14:58, 4.27it/s, lr: 6.0e-05 loss: 1.586e-01] jess: 52%|█████▏ | 4168/8000 [02:34<14:58, 4.27it/s, lr: 6.0e-05 loss: 5.031e-02] jess: 52%|█████▏ | 4169/8000 [02:34<11:04, 5.76it/s, lr: 6.0e-05 loss: 5.031e-02] jess: 52%|█████▏ | 4169/8000 [02:34<11:04, 5.76it/s, lr: 6.0e-05 loss: 1.339e-03] jess: 52%|█████▏ | 4170/8000 [02:34<09:53, 6.45it/s, lr: 6.0e-05 loss: 1.339e-03] jess: 52%|█████▏ | 4170/8000 [02:34<09:53, 6.45it/s, lr: 6.0e-05 loss: 1.991e-01] jess: 52%|█████▏ | 4171/8000 [02:34<15:06, 4.22it/s, lr: 6.0e-05 loss: 1.991e-01] jess: 52%|█████▏ | 4171/8000 [02:34<15:06, 4.22it/s, lr: 6.0e-05 loss: 7.201e-02] jess: 52%|█████▏ | 4172/8000 [02:34<11:04, 5.76it/s, lr: 6.0e-05 loss: 7.201e-02] jess: 52%|█████▏ | 4172/8000 [02:34<11:04, 5.76it/s, lr: 6.0e-05 loss: 1.790e-02] jess: 52%|█████▏ | 4173/8000 [02:34<15:18, 4.16it/s, lr: 6.0e-05 loss: 1.790e-02] jess: 52%|█████▏ | 4173/8000 [02:35<15:18, 4.16it/s, lr: 6.0e-05 loss: 6.835e-02] jess: 52%|█████▏ | 4174/8000 [02:35<16:35, 3.84it/s, lr: 6.0e-05 loss: 6.835e-02] jess: 52%|█████▏ | 4174/8000 [02:35<16:35, 3.84it/s, lr: 6.0e-05 loss: 6.036e-02] jess: 52%|█████▏ | 4175/8000 [02:35<16:50, 3.79it/s, lr: 6.0e-05 loss: 6.036e-02] jess: 52%|█████▏ | 4175/8000 [02:35<16:50, 3.79it/s, lr: 6.0e-05 loss: 1.109e-01] jess: 52%|█████▏ | 4176/8000 [02:35<24:46, 2.57it/s, lr: 6.0e-05 loss: 1.109e-01] jess: 52%|█████▏ | 4176/8000 [02:36<24:46, 2.57it/s, lr: 6.0e-05 loss: 3.829e-02] jess: 52%|█████▏ | 4177/8000 [02:36<14:06, 4.52it/s, lr: 6.0e-05 loss: 3.829e-02] jess: 52%|█████▏ | 4177/8000 [02:36<14:06, 4.52it/s, lr: 6.0e-05 loss: 5.539e-02] jess: 52%|█████▏ | 4178/8000 [02:36<10:49, 5.89it/s, lr: 6.0e-05 loss: 5.539e-02] jess: 52%|█████▏ | 4178/8000 [02:36<10:49, 5.89it/s, lr: 6.0e-05 loss: 1.295e-01] jess: 52%|█████▏ | 4179/8000 [02:36<09:45, 6.52it/s, lr: 6.0e-05 loss: 1.295e-01] jess: 52%|█████▏ | 4179/8000 [02:36<09:45, 6.52it/s, lr: 6.0e-05 loss: 2.015e-01] jess: 52%|█████▏ | 4180/8000 [02:36<09:29, 6.70it/s, lr: 6.0e-05 loss: 2.015e-01] jess: 52%|█████▏ | 4180/8000 [02:36<09:29, 6.70it/s, lr: 6.0e-05 loss: 9.241e-02] jess: 52%|█████▏ | 4181/8000 [02:36<14:59, 4.24it/s, lr: 6.0e-05 loss: 9.241e-02] jess: 52%|█████▏ | 4181/8000 [02:37<14:59, 4.24it/s, lr: 6.0e-05 loss: 5.112e-02] jess: 52%|█████▏ | 4182/8000 [02:37<16:38, 3.82it/s, lr: 6.0e-05 loss: 5.112e-02] jess: 52%|█████▏ | 4182/8000 [02:37<16:38, 3.82it/s, lr: 6.0e-05 loss: 3.668e-03] jess: 52%|█████▏ | 4183/8000 [02:37<11:30, 5.53it/s, lr: 6.0e-05 loss: 3.668e-03] jess: 52%|█████▏ | 4183/8000 [02:37<11:30, 5.53it/s, lr: 6.0e-05 loss: 1.929e-01] jess: 52%|█████▏ | 4184/8000 [02:37<15:26, 4.12it/s, lr: 6.0e-05 loss: 1.929e-01] jess: 52%|█████▏ | 4184/8000 [02:37<15:26, 4.12it/s, lr: 6.0e-05 loss: 7.225e-02] jess: 52%|█████▏ | 4185/8000 [02:37<17:10, 3.70it/s, lr: 6.0e-05 loss: 7.225e-02] jess: 52%|█████▏ | 4185/8000 [02:38<17:10, 3.70it/s, lr: 6.0e-05 loss: 7.087e-02] jess: 52%|█████▏ | 4186/8000 [02:38<17:11, 3.70it/s, lr: 6.0e-05 loss: 7.087e-02] jess: 52%|█████▏ | 4186/8000 [02:38<17:11, 3.70it/s, lr: 6.0e-05 loss: 1.529e-01] jess: 52%|█████▏ | 4187/8000 [02:38<11:44, 5.41it/s, lr: 6.0e-05 loss: 1.529e-01] jess: 52%|█████▏ | 4187/8000 [02:38<11:44, 5.41it/s, lr: 6.0e-05 loss: 1.481e-01] jess: 52%|█████▏ | 4188/8000 [02:38<15:34, 4.08it/s, lr: 6.0e-05 loss: 1.481e-01] jess: 52%|█████▏ | 4188/8000 [02:38<15:34, 4.08it/s, lr: 6.0e-05 loss: 1.284e-01] jess: 52%|█████▏ | 4189/8000 [02:38<11:19, 5.61it/s, lr: 6.0e-05 loss: 1.284e-01] jess: 52%|█████▏ | 4189/8000 [02:38<11:19, 5.61it/s, lr: 6.0e-05 loss: 5.063e-02] jess: 52%|█████▏ | 4190/8000 [02:38<15:32, 4.09it/s, lr: 6.0e-05 loss: 5.063e-02] jess: 52%|█████▏ | 4190/8000 [02:39<15:32, 4.09it/s, lr: 6.0e-05 loss: 1.760e-01] jess: 52%|█████▏ | 4191/8000 [02:39<11:13, 5.66it/s, lr: 6.0e-05 loss: 1.760e-01] jess: 52%|█████▏ | 4191/8000 [02:39<11:13, 5.66it/s, lr: 6.0e-05 loss: 1.160e-01] jess: 52%|█████▏ | 4192/8000 [02:39<09:51, 6.44it/s, lr: 6.0e-05 loss: 1.160e-01] jess: 52%|█████▏ | 4192/8000 [02:39<09:51, 6.44it/s, lr: 6.0e-05 loss: 1.091e-01] jess: 52%|█████▏ | 4193/8000 [02:39<14:54, 4.26it/s, lr: 6.0e-05 loss: 1.091e-01] jess: 52%|█████▏ | 4193/8000 [02:39<14:54, 4.26it/s, lr: 6.0e-05 loss: 9.184e-02] jess: 52%|█████▏ | 4194/8000 [02:39<10:59, 5.77it/s, lr: 6.0e-05 loss: 9.184e-02] jess: 52%|█████▏ | 4194/8000 [02:39<10:59, 5.77it/s, lr: 6.0e-05 loss: 6.491e-03] jess: 52%|█████▏ | 4195/8000 [02:39<09:46, 6.49it/s, lr: 6.0e-05 loss: 6.491e-03] jess: 52%|█████▏ | 4195/8000 [02:39<09:46, 6.49it/s, lr: 6.0e-05 loss: 4.083e-02] jess: 52%|█████▏ | 4196/8000 [02:39<09:27, 6.70it/s, lr: 6.0e-05 loss: 4.083e-02] jess: 52%|█████▏ | 4196/8000 [02:40<09:27, 6.70it/s, lr: 6.0e-05 loss: 1.864e-01] jess: 52%|█████▏ | 4197/8000 [02:40<14:45, 4.29it/s, lr: 6.0e-05 loss: 1.864e-01] jess: 52%|█████▏ | 4197/8000 [02:40<14:45, 4.29it/s, lr: 6.0e-05 loss: 8.884e-02] jess: 52%|█████▏ | 4198/8000 [02:40<16:28, 3.85it/s, lr: 6.0e-05 loss: 8.884e-02] jess: 52%|█████▏ | 4198/8000 [02:40<16:28, 3.85it/s, lr: 6.0e-05 loss: 6.220e-02] jess: 52%|█████▏ | 4199/8000 [02:40<11:27, 5.53it/s, lr: 6.0e-05 loss: 6.220e-02] jess: 52%|█████▏ | 4199/8000 [02:41<11:27, 5.53it/s, lr: 6.0e-05 loss: 2.623e-02] jess: 52%|█████▎ | 4200/8000 [02:41<23:04, 2.75it/s, lr: 6.0e-05 loss: 2.623e-02] jess: 52%|█████▎ | 4200/8000 [02:41<23:04, 2.75it/s, lr: 6.0e-05 loss: 1.275e-01]
Saving at step 4200
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004200.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.34s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.33s/it]
Generating Images: 100%|██████████| 3/3 [00:04<00:00, 1.34s/it]

jess: 53%|█████▎ | 4201/8000 [02:41<23:03, 2.75it/s, lr: 6.0e-05 loss: 5.130e-02] jess: 53%|█████▎ | 4202/8000 [02:41<15:43, 4.03it/s, lr: 6.0e-05 loss: 5.130e-02] jess: 53%|█████▎ | 4202/8000 [02:42<15:43, 4.03it/s, lr: 6.0e-05 loss: 4.228e-02] jess: 53%|█████▎ | 4203/8000 [02:42<16:38, 3.80it/s, lr: 6.0e-05 loss: 4.228e-02] jess: 53%|█████▎ | 4203/8000 [02:42<16:38, 3.80it/s, lr: 6.0e-05 loss: 1.332e-01] jess: 53%|█████▎ | 4204/8000 [02:42<17:05, 3.70it/s, lr: 6.0e-05 loss: 1.332e-01] jess: 53%|█████▎ | 4204/8000 [02:42<17:05, 3.70it/s, lr: 6.0e-05 loss: 3.660e-02] jess: 53%|█████▎ | 4205/8000 [02:42<16:56, 3.73it/s, lr: 6.0e-05 loss: 3.660e-02] jess: 53%|█████▎ | 4205/8000 [02:42<16:56, 3.73it/s, lr: 6.0e-05 loss: 6.899e-02] jess: 53%|█████▎ | 4206/8000 [02:42<11:42, 5.40it/s, lr: 6.0e-05 loss: 6.899e-02] jess: 53%|█████▎ | 4206/8000 [02:42<11:42, 5.40it/s, lr: 6.0e-05 loss: 1.460e-01] jess: 53%|█████▎ | 4207/8000 [02:42<15:34, 4.06it/s, lr: 6.0e-05 loss: 1.460e-01] jess: 53%|█████▎ | 4207/8000 [02:43<15:34, 4.06it/s, lr: 6.0e-05 loss: 1.165e-01] jess: 53%|█████▎ | 4208/8000 [02:43<16:42, 3.78it/s, lr: 6.0e-05 loss: 1.165e-01] jess: 53%|█████▎ | 4208/8000 [02:43<16:42, 3.78it/s, lr: 6.0e-05 loss: 1.223e-01] jess: 53%|█████▎ | 4209/8000 [02:43<17:03, 3.70it/s, lr: 6.0e-05 loss: 1.223e-01] jess: 53%|█████▎ | 4209/8000 [02:43<17:03, 3.70it/s, lr: 6.0e-05 loss: 1.521e-01] jess: 53%|█████▎ | 4210/8000 [02:43<15:56, 3.96it/s, lr: 6.0e-05 loss: 1.521e-01] jess: 53%|█████▎ | 4210/8000 [02:43<15:56, 3.96it/s, lr: 6.0e-05 loss: 1.097e-01] jess: 53%|█████▎ | 4211/8000 [02:43<11:23, 5.55it/s, lr: 6.0e-05 loss: 1.097e-01] jess: 53%|█████▎ | 4211/8000 [02:44<11:23, 5.55it/s, lr: 6.0e-05 loss: 1.125e-01] jess: 53%|█████▎ | 4212/8000 [02:44<15:26, 4.09it/s, lr: 6.0e-05 loss: 1.125e-01] jess: 53%|█████▎ | 4212/8000 [02:44<15:26, 4.09it/s, lr: 6.0e-05 loss: 1.632e-01] jess: 53%|█████▎ | 4213/8000 [02:44<16:37, 3.80it/s, lr: 6.0e-05 loss: 1.632e-01] jess: 53%|█████▎ | 4213/8000 [02:44<16:37, 3.80it/s, lr: 6.0e-05 loss: 2.131e-03] jess: 53%|█████▎ | 4214/8000 [02:44<11:34, 5.45it/s, lr: 6.0e-05 loss: 2.131e-03] jess: 53%|█████▎ | 4214/8000 [02:44<11:34, 5.45it/s, lr: 6.0e-05 loss: 1.097e-02] jess: 53%|█████▎ | 4215/8000 [02:44<15:25, 4.09it/s, lr: 6.0e-05 loss: 1.097e-02] jess: 53%|█████▎ | 4215/8000 [02:45<15:25, 4.09it/s, lr: 6.0e-05 loss: 1.032e-01] jess: 53%|█████▎ | 4216/8000 [02:45<11:18, 5.58it/s, lr: 6.0e-05 loss: 1.032e-01] jess: 53%|█████▎ | 4216/8000 [02:45<11:18, 5.58it/s, lr: 6.0e-05 loss: 4.876e-02] jess: 53%|█████▎ | 4217/8000 [02:45<09:57, 6.33it/s, lr: 6.0e-05 loss: 4.876e-02] jess: 53%|█████▎ | 4217/8000 [02:45<09:57, 6.33it/s, lr: 6.0e-05 loss: 7.536e-02] jess: 53%|█████▎ | 4218/8000 [02:45<09:28, 6.65it/s, lr: 6.0e-05 loss: 7.536e-02] jess: 53%|█████▎ | 4218/8000 [02:45<09:28, 6.65it/s, lr: 6.0e-05 loss: 7.708e-02] jess: 53%|█████▎ | 4219/8000 [02:45<14:45, 4.27it/s, lr: 6.0e-05 loss: 7.708e-02] jess: 53%|█████▎ | 4219/8000 [02:45<14:45, 4.27it/s, lr: 6.0e-05 loss: 8.649e-02] jess: 53%|█████▎ | 4220/8000 [02:45<10:54, 5.78it/s, lr: 6.0e-05 loss: 8.649e-02] jess: 53%|█████▎ | 4220/8000 [02:45<10:54, 5.78it/s, lr: 6.0e-05 loss: 1.721e-02] jess: 53%|█████▎ | 4221/8000 [02:45<10:08, 6.21it/s, lr: 6.0e-05 loss: 1.721e-02] jess: 53%|█████▎ | 4221/8000 [02:46<10:08, 6.21it/s, lr: 6.0e-05 loss: 1.336e-01] jess: 53%|█████▎ | 4222/8000 [02:46<09:30, 6.62it/s, lr: 6.0e-05 loss: 1.336e-01] jess: 53%|█████▎ | 4222/8000 [02:46<09:30, 6.62it/s, lr: 6.0e-05 loss: 1.997e-02] jess: 53%|█████▎ | 4223/8000 [02:46<09:27, 6.65it/s, lr: 6.0e-05 loss: 1.997e-02] jess: 53%|█████▎ | 4223/8000 [02:46<09:27, 6.65it/s, lr: 6.0e-05 loss: 1.643e-01] jess: 53%|█████▎ | 4224/8000 [02:46<14:49, 4.25it/s, lr: 6.0e-05 loss: 1.643e-01] jess: 53%|█████▎ | 4224/8000 [02:46<14:49, 4.25it/s, lr: 6.0e-05 loss: 1.220e-01] jess: 53%|█████▎ | 4225/8000 [02:46<10:58, 5.74it/s, lr: 6.0e-05 loss: 1.220e-01] jess: 53%|█████▎ | 4225/8000 [02:47<10:58, 5.74it/s, lr: 6.0e-05 loss: 2.484e-01] jess: 53%|█████▎ | 4226/8000 [02:47<27:46, 2.26it/s, lr: 6.0e-05 loss: 2.484e-01] jess: 53%|█████▎ | 4226/8000 [02:47<27:46, 2.26it/s, lr: 6.0e-05 loss: 6.499e-02] jess: 53%|█████▎ | 4227/8000 [02:47<20:23, 3.08it/s, lr: 6.0e-05 loss: 6.499e-02] jess: 53%|█████▎ | 4227/8000 [02:47<20:23, 3.08it/s, lr: 6.0e-05 loss: 9.301e-02] jess: 53%|█████▎ | 4228/8000 [02:47<18:09, 3.46it/s, lr: 6.0e-05 loss: 9.301e-02] jess: 53%|█████▎ | 4228/8000 [02:48<18:09, 3.46it/s, lr: 6.0e-05 loss: 1.907e-01] jess: 53%|█████▎ | 4229/8000 [02:48<17:27, 3.60it/s, lr: 6.0e-05 loss: 1.907e-01] jess: 53%|█████▎ | 4229/8000 [02:48<17:27, 3.60it/s, lr: 6.0e-05 loss: 8.990e-03] jess: 53%|█████▎ | 4230/8000 [02:48<11:44, 5.35it/s, lr: 6.0e-05 loss: 8.990e-03] jess: 53%|█████▎ | 4230/8000 [02:48<11:44, 5.35it/s, lr: 6.0e-05 loss: 8.716e-02] jess: 53%|█████▎ | 4231/8000 [02:48<15:35, 4.03it/s, lr: 6.0e-05 loss: 8.716e-02] jess: 53%|█████▎ | 4231/8000 [02:48<15:35, 4.03it/s, lr: 6.0e-05 loss: 1.590e-01] jess: 53%|█████▎ | 4232/8000 [02:48<16:42, 3.76it/s, lr: 6.0e-05 loss: 1.590e-01] jess: 53%|█████▎ | 4232/8000 [02:48<16:42, 3.76it/s, lr: 6.0e-05 loss: 8.821e-02] jess: 53%|█████▎ | 4233/8000 [02:48<11:46, 5.33it/s, lr: 6.0e-05 loss: 8.821e-02] jess: 53%|█████▎ | 4233/8000 [02:48<11:46, 5.33it/s, lr: 6.0e-05 loss: 3.641e-02] jess: 53%|█████▎ | 4234/8000 [02:49<10:02, 6.25it/s, lr: 6.0e-05 loss: 3.641e-02] jess: 53%|█████▎ | 4234/8000 [02:49<10:02, 6.25it/s, lr: 6.0e-05 loss: 9.130e-02] jess: 53%|█████▎ | 4235/8000 [02:49<09:33, 6.57it/s, lr: 6.0e-05 loss: 9.130e-02] jess: 53%|█████▎ | 4235/8000 [02:49<09:33, 6.57it/s, lr: 6.0e-05 loss: 3.458e-02] jess: 53%|█████▎ | 4236/8000 [02:49<09:24, 6.67it/s, lr: 6.0e-05 loss: 3.458e-02] jess: 53%|█████▎ | 4236/8000 [02:49<09:24, 6.67it/s, lr: 6.0e-05 loss: 2.147e-01] jess: 53%|█████▎ | 4237/8000 [02:49<14:45, 4.25it/s, lr: 6.0e-05 loss: 2.147e-01] jess: 53%|█████▎ | 4237/8000 [02:49<14:45, 4.25it/s, lr: 6.0e-05 loss: 1.044e-01] jess: 53%|█████▎ | 4238/8000 [02:49<16:24, 3.82it/s, lr: 6.0e-05 loss: 1.044e-01] jess: 53%|█████▎ | 4238/8000 [02:49<16:24, 3.82it/s, lr: 6.0e-05 loss: 6.571e-02] jess: 53%|█████▎ | 4239/8000 [02:49<11:29, 5.46it/s, lr: 6.0e-05 loss: 6.571e-02] jess: 53%|█████▎ | 4239/8000 [02:50<11:29, 5.46it/s, lr: 6.0e-05 loss: 1.272e-02] jess: 53%|█████▎ | 4240/8000 [02:50<14:09, 4.42it/s, lr: 6.0e-05 loss: 1.272e-02] jess: 53%|█████▎ | 4240/8000 [02:50<14:09, 4.42it/s, lr: 6.0e-05 loss: 8.139e-02] jess: 53%|█████▎ | 4241/8000 [02:50<10:46, 5.82it/s, lr: 6.0e-05 loss: 8.139e-02] jess: 53%|█████▎ | 4241/8000 [02:50<10:46, 5.82it/s, lr: 6.0e-05 loss: 8.626e-02] jess: 53%|█████▎ | 4242/8000 [02:50<15:08, 4.14it/s, lr: 6.0e-05 loss: 8.626e-02] jess: 53%|█████▎ | 4242/8000 [02:50<15:08, 4.14it/s, lr: 6.0e-05 loss: 3.871e-03] jess: 53%|█████▎ | 4243/8000 [02:50<11:04, 5.66it/s, lr: 6.0e-05 loss: 3.871e-03] jess: 53%|█████▎ | 4243/8000 [02:51<11:04, 5.66it/s, lr: 6.0e-05 loss: 9.183e-03] jess: 53%|█████▎ | 4244/8000 [02:51<15:07, 4.14it/s, lr: 6.0e-05 loss: 9.183e-03] jess: 53%|█████▎ | 4244/8000 [02:51<15:07, 4.14it/s, lr: 6.0e-05 loss: 1.778e-01] jess: 53%|█████▎ | 4245/8000 [02:51<11:01, 5.68it/s, lr: 6.0e-05 loss: 1.778e-01] jess: 53%|█████▎ | 4245/8000 [02:51<11:01, 5.68it/s, lr: 6.0e-05 loss: 8.978e-02] jess: 53%|█████▎ | 4246/8000 [02:51<09:46, 6.40it/s, lr: 6.0e-05 loss: 8.978e-02] jess: 53%|█████▎ | 4246/8000 [02:51<09:46, 6.40it/s, lr: 6.0e-05 loss: 1.583e-01] jess: 53%|█████▎ | 4247/8000 [02:51<14:43, 4.25it/s, lr: 6.0e-05 loss: 1.583e-01] jess: 53%|█████▎ | 4247/8000 [02:51<14:43, 4.25it/s, lr: 6.0e-05 loss: 1.045e-01] jess: 53%|█████▎ | 4248/8000 [02:51<16:18, 3.83it/s, lr: 6.0e-05 loss: 1.045e-01] jess: 53%|█████▎ | 4248/8000 [02:52<16:18, 3.83it/s, lr: 6.0e-05 loss: 9.155e-02] jess: 53%|█████▎ | 4249/8000 [02:52<16:58, 3.68it/s, lr: 6.0e-05 loss: 9.155e-02] jess: 53%|█████▎ | 4249/8000 [02:52<16:58, 3.68it/s, lr: 6.0e-05 loss: 3.194e-02] jess: 53%|█████▎ | 4250/8000 [02:52<29:33, 2.11it/s, lr: 6.0e-05 loss: 3.194e-02] jess: 53%|█████▎ | 4250/8000 [02:52<29:33, 2.11it/s, lr: 6.0e-05 loss: 1.101e-01]
Saving at step 4250
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004250.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 53%|█████▎ | 4251/8000 [02:53<29:33, 2.11it/s, lr: 6.0e-05 loss: 9.367e-02] jess: 53%|█████▎ | 4252/8000 [02:53<16:32, 3.78it/s, lr: 6.0e-05 loss: 9.367e-02] jess: 53%|█████▎ | 4252/8000 [02:53<16:32, 3.78it/s, lr: 6.0e-05 loss: 1.080e-01] jess: 53%|█████▎ | 4253/8000 [02:53<16:50, 3.71it/s, lr: 6.0e-05 loss: 1.080e-01] jess: 53%|█████▎ | 4253/8000 [02:53<16:50, 3.71it/s, lr: 6.0e-05 loss: 8.532e-02] jess: 53%|█████▎ | 4254/8000 [02:53<12:01, 5.19it/s, lr: 6.0e-05 loss: 8.532e-02] jess: 53%|█████▎ | 4254/8000 [02:54<12:01, 5.19it/s, lr: 6.0e-05 loss: 1.628e-02] jess: 53%|█████▎ | 4255/8000 [02:54<15:08, 4.12it/s, lr: 6.0e-05 loss: 1.628e-02] jess: 53%|█████▎ | 4255/8000 [02:54<15:08, 4.12it/s, lr: 6.0e-05 loss: 1.653e-01] jess: 53%|█████▎ | 4256/8000 [02:54<16:19, 3.82it/s, lr: 6.0e-05 loss: 1.653e-01] jess: 53%|█████▎ | 4256/8000 [02:54<16:19, 3.82it/s, lr: 6.0e-05 loss: 1.204e-01] jess: 53%|█████▎ | 4257/8000 [02:54<17:09, 3.64it/s, lr: 6.0e-05 loss: 1.204e-01] jess: 53%|█████▎ | 4257/8000 [02:54<17:09, 3.64it/s, lr: 6.0e-05 loss: 6.023e-02] jess: 53%|█████▎ | 4258/8000 [02:54<16:57, 3.68it/s, lr: 6.0e-05 loss: 6.023e-02] jess: 53%|█████▎ | 4258/8000 [02:55<16:57, 3.68it/s, lr: 6.0e-05 loss: 1.116e-01] jess: 53%|█████▎ | 4259/8000 [02:55<16:57, 3.68it/s, lr: 6.0e-05 loss: 1.116e-01] jess: 53%|█████▎ | 4259/8000 [02:55<16:57, 3.68it/s, lr: 6.0e-05 loss: 4.655e-02] jess: 53%|█████▎ | 4260/8000 [02:55<15:40, 3.97it/s, lr: 6.0e-05 loss: 4.655e-02] jess: 53%|█████▎ | 4260/8000 [02:55<15:40, 3.97it/s, lr: 6.0e-05 loss: 1.422e-01] jess: 53%|█████▎ | 4261/8000 [02:55<11:13, 5.55it/s, lr: 6.0e-05 loss: 1.422e-01] jess: 53%|█████▎ | 4261/8000 [02:55<11:13, 5.55it/s, lr: 6.0e-05 loss: 2.418e-01] jess: 53%|█████▎ | 4262/8000 [02:55<09:47, 6.37it/s, lr: 6.0e-05 loss: 2.418e-01] jess: 53%|█████▎ | 4262/8000 [02:55<09:47, 6.37it/s, lr: 6.0e-05 loss: 9.969e-02] jess: 53%|█████▎ | 4263/8000 [02:55<09:22, 6.64it/s, lr: 6.0e-05 loss: 9.969e-02] jess: 53%|█████▎ | 4263/8000 [02:56<09:22, 6.64it/s, lr: 6.0e-05 loss: 1.664e-02] jess: 53%|█████▎ | 4264/8000 [02:56<14:24, 4.32it/s, lr: 6.0e-05 loss: 1.664e-02] jess: 53%|█████▎ | 4264/8000 [02:56<14:24, 4.32it/s, lr: 6.0e-05 loss: 6.959e-02] jess: 53%|█████▎ | 4265/8000 [02:56<10:47, 5.77it/s, lr: 6.0e-05 loss: 6.959e-02] jess: 53%|█████▎ | 4265/8000 [02:56<10:47, 5.77it/s, lr: 6.0e-05 loss: 2.455e-01] jess: 53%|█████▎ | 4266/8000 [02:56<09:40, 6.43it/s, lr: 6.0e-05 loss: 2.455e-01] jess: 53%|█████▎ | 4266/8000 [02:56<09:40, 6.43it/s, lr: 6.0e-05 loss: 9.452e-02] jess: 53%|█████▎ | 4267/8000 [02:56<14:48, 4.20it/s, lr: 6.0e-05 loss: 9.452e-02] jess: 53%|█████▎ | 4267/8000 [02:56<14:48, 4.20it/s, lr: 6.0e-05 loss: 7.190e-02] jess: 53%|█████▎ | 4268/8000 [02:56<16:20, 3.81it/s, lr: 6.0e-05 loss: 7.190e-02] jess: 53%|█████▎ | 4268/8000 [02:57<16:20, 3.81it/s, lr: 6.0e-05 loss: 1.538e-01] jess: 53%|█████▎ | 4269/8000 [02:57<16:45, 3.71it/s, lr: 6.0e-05 loss: 1.538e-01] jess: 53%|█████▎ | 4269/8000 [02:57<16:45, 3.71it/s, lr: 6.0e-05 loss: 8.374e-02] jess: 53%|█████▎ | 4270/8000 [02:57<11:30, 5.40it/s, lr: 6.0e-05 loss: 8.374e-02] jess: 53%|█████▎ | 4270/8000 [02:57<11:30, 5.40it/s, lr: 6.0e-05 loss: 1.393e-01] jess: 53%|█████▎ | 4271/8000 [02:57<09:56, 6.25it/s, lr: 6.0e-05 loss: 1.393e-01] jess: 53%|█████▎ | 4271/8000 [02:57<09:56, 6.25it/s, lr: 6.0e-05 loss: 4.895e-02] jess: 53%|█████▎ | 4272/8000 [02:57<14:51, 4.18it/s, lr: 6.0e-05 loss: 4.895e-02] jess: 53%|█████▎ | 4272/8000 [02:58<14:51, 4.18it/s, lr: 6.0e-05 loss: 1.631e-01] jess: 53%|█████▎ | 4273/8000 [02:58<16:14, 3.83it/s, lr: 6.0e-05 loss: 1.631e-01] jess: 53%|█████▎ | 4273/8000 [02:58<16:14, 3.83it/s, lr: 6.0e-05 loss: 1.617e-01] jess: 53%|█████▎ | 4274/8000 [02:58<11:19, 5.48it/s, lr: 6.0e-05 loss: 1.617e-01] jess: 53%|█████▎ | 4274/8000 [02:58<11:19, 5.48it/s, lr: 6.0e-05 loss: 6.136e-02] jess: 53%|█████▎ | 4275/8000 [02:58<09:47, 6.34it/s, lr: 6.0e-05 loss: 6.136e-02] jess: 53%|█████▎ | 4275/8000 [02:58<09:47, 6.34it/s, lr: 6.0e-05 loss: 1.565e-01] jess: 53%|█████▎ | 4276/8000 [02:58<22:26, 2.77it/s, lr: 6.0e-05 loss: 1.565e-01] jess: 53%|█████▎ | 4276/8000 [02:59<22:26, 2.77it/s, lr: 6.0e-05 loss: 4.909e-02] jess: 53%|█████▎ | 4277/8000 [02:59<18:37, 3.33it/s, lr: 6.0e-05 loss: 4.909e-02] jess: 53%|█████▎ | 4277/8000 [02:59<18:37, 3.33it/s, lr: 6.0e-05 loss: 1.880e-01] jess: 53%|█████▎ | 4278/8000 [02:59<17:17, 3.59it/s, lr: 6.0e-05 loss: 1.880e-01] jess: 53%|█████▎ | 4278/8000 [02:59<17:17, 3.59it/s, lr: 6.0e-05 loss: 1.355e-01] jess: 53%|█████▎ | 4279/8000 [02:59<11:35, 5.35it/s, lr: 6.0e-05 loss: 1.355e-01] jess: 53%|█████▎ | 4279/8000 [02:59<11:35, 5.35it/s, lr: 6.0e-05 loss: 1.322e-01] jess: 54%|█████▎ | 4280/8000 [02:59<15:15, 4.06it/s, lr: 6.0e-05 loss: 1.322e-01] jess: 54%|█████▎ | 4280/8000 [02:59<15:15, 4.06it/s, lr: 6.0e-05 loss: 4.223e-02] jess: 54%|█████▎ | 4281/8000 [02:59<11:02, 5.62it/s, lr: 6.0e-05 loss: 4.223e-02] jess: 54%|█████▎ | 4281/8000 [03:00<11:02, 5.62it/s, lr: 6.0e-05 loss: 1.080e-01] jess: 54%|█████▎ | 4282/8000 [03:00<14:59, 4.13it/s, lr: 6.0e-05 loss: 1.080e-01] jess: 54%|█████▎ | 4282/8000 [03:00<14:59, 4.13it/s, lr: 6.0e-05 loss: 4.072e-03] jess: 54%|█████▎ | 4283/8000 [03:00<10:53, 5.69it/s, lr: 6.0e-05 loss: 4.072e-03] jess: 54%|█████▎ | 4283/8000 [03:00<10:53, 5.69it/s, lr: 6.0e-05 loss: 6.222e-02] jess: 54%|█████▎ | 4284/8000 [03:00<09:39, 6.41it/s, lr: 6.0e-05 loss: 6.222e-02] jess: 54%|█████▎ | 4284/8000 [03:00<09:39, 6.41it/s, lr: 6.0e-05 loss: 9.459e-03] jess: 54%|█████▎ | 4285/8000 [03:00<14:32, 4.26it/s, lr: 6.0e-05 loss: 9.459e-03] jess: 54%|█████▎ | 4285/8000 [03:00<14:32, 4.26it/s, lr: 6.0e-05 loss: 6.577e-03] jess: 54%|█████▎ | 4286/8000 [03:00<10:45, 5.75it/s, lr: 6.0e-05 loss: 6.577e-03] jess: 54%|█████▎ | 4286/8000 [03:01<10:45, 5.75it/s, lr: 6.0e-05 loss: 9.967e-02] jess: 54%|█████▎ | 4287/8000 [03:01<14:59, 4.13it/s, lr: 6.0e-05 loss: 9.967e-02] jess: 54%|█████▎ | 4287/8000 [03:01<14:59, 4.13it/s, lr: 6.0e-05 loss: 1.606e-01] jess: 54%|█████▎ | 4288/8000 [03:01<10:53, 5.68it/s, lr: 6.0e-05 loss: 1.606e-01] jess: 54%|█████▎ | 4288/8000 [03:01<10:53, 5.68it/s, lr: 6.0e-05 loss: 6.562e-02] jess: 54%|█████▎ | 4289/8000 [03:01<15:01, 4.11it/s, lr: 6.0e-05 loss: 6.562e-02] jess: 54%|█████▎ | 4289/8000 [03:01<15:01, 4.11it/s, lr: 6.0e-05 loss: 1.313e-01] jess: 54%|█████▎ | 4290/8000 [03:01<10:53, 5.68it/s, lr: 6.0e-05 loss: 1.313e-01] jess: 54%|█████▎ | 4290/8000 [03:01<10:53, 5.68it/s, lr: 6.0e-05 loss: 5.203e-02] jess: 54%|█████▎ | 4291/8000 [03:01<15:03, 4.10it/s, lr: 6.0e-05 loss: 5.203e-02] jess: 54%|█████▎ | 4291/8000 [03:02<15:03, 4.10it/s, lr: 6.0e-05 loss: 1.940e-01] jess: 54%|█████▎ | 4292/8000 [03:02<16:08, 3.83it/s, lr: 6.0e-05 loss: 1.940e-01] jess: 54%|█████▎ | 4292/8000 [03:02<16:08, 3.83it/s, lr: 6.0e-05 loss: 1.505e-01] jess: 54%|█████▎ | 4293/8000 [03:02<17:03, 3.62it/s, lr: 6.0e-05 loss: 1.505e-01] jess: 54%|█████▎ | 4293/8000 [03:02<17:03, 3.62it/s, lr: 6.0e-05 loss: 1.139e-01] jess: 54%|█████▎ | 4294/8000 [03:02<11:26, 5.40it/s, lr: 6.0e-05 loss: 1.139e-01] jess: 54%|█████▎ | 4294/8000 [03:02<11:26, 5.40it/s, lr: 6.0e-05 loss: 1.160e-02] jess: 54%|█████▎ | 4295/8000 [03:02<15:02, 4.11it/s, lr: 6.0e-05 loss: 1.160e-02] jess: 54%|█████▎ | 4295/8000 [03:03<15:02, 4.11it/s, lr: 6.0e-05 loss: 1.219e-01] jess: 54%|█████▎ | 4296/8000 [03:03<16:15, 3.80it/s, lr: 6.0e-05 loss: 1.219e-01] jess: 54%|█████▎ | 4296/8000 [03:03<16:15, 3.80it/s, lr: 6.0e-05 loss: 7.789e-02] jess: 54%|█████▎ | 4297/8000 [03:03<16:36, 3.72it/s, lr: 6.0e-05 loss: 7.789e-02] jess: 54%|█████▎ | 4297/8000 [03:03<16:36, 3.72it/s, lr: 6.0e-05 loss: 1.119e-01] jess: 54%|█████▎ | 4298/8000 [03:03<16:42, 3.69it/s, lr: 6.0e-05 loss: 1.119e-01] jess: 54%|█████▎ | 4298/8000 [03:04<16:42, 3.69it/s, lr: 6.0e-05 loss: 1.819e-01] jess: 54%|█████▎ | 4299/8000 [03:04<16:38, 3.71it/s, lr: 6.0e-05 loss: 1.819e-01] jess: 54%|█████▎ | 4299/8000 [03:04<16:38, 3.71it/s, lr: 6.0e-05 loss: 7.899e-02] jess: 54%|█████▍ | 4300/8000 [03:04<24:31, 2.51it/s, lr: 6.0e-05 loss: 7.899e-02] jess: 54%|█████▍ | 4300/8000 [03:04<24:31, 2.51it/s, lr: 6.0e-05 loss: 2.832e-02]
Saving at step 4300
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004300.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.32s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.32s/it]
Generating Images: 100%|██████████| 3/3 [00:03<00:00, 1.32s/it]

jess: 54%|█████▍ | 4301/8000 [03:05<24:31, 2.51it/s, lr: 6.0e-05 loss: 1.471e-01] jess: 54%|█████▍ | 4302/8000 [03:05<14:57, 4.12it/s, lr: 6.0e-05 loss: 1.471e-01] jess: 54%|█████▍ | 4302/8000 [03:05<14:57, 4.12it/s, lr: 6.0e-05 loss: 2.345e-01] jess: 54%|█████▍ | 4303/8000 [03:05<15:55, 3.87it/s, lr: 6.0e-05 loss: 2.345e-01] jess: 54%|█████▍ | 4303/8000 [03:05<15:55, 3.87it/s, lr: 6.0e-05 loss: 7.095e-02] jess: 54%|█████▍ | 4304/8000 [03:05<11:27, 5.38it/s, lr: 6.0e-05 loss: 7.095e-02] jess: 54%|█████▍ | 4304/8000 [03:05<11:27, 5.38it/s, lr: 6.0e-05 loss: 1.882e-01] jess: 54%|█████▍ | 4305/8000 [03:05<09:45, 6.31it/s, lr: 6.0e-05 loss: 1.882e-01] jess: 54%|█████▍ | 4305/8000 [03:06<09:45, 6.31it/s, lr: 6.0e-05 loss: 2.428e-02] jess: 54%|█████▍ | 4306/8000 [03:06<14:19, 4.30it/s, lr: 6.0e-05 loss: 2.428e-02] jess: 54%|█████▍ | 4306/8000 [03:06<14:19, 4.30it/s, lr: 6.0e-05 loss: 7.301e-02] jess: 54%|█████▍ | 4307/8000 [03:06<10:36, 5.80it/s, lr: 6.0e-05 loss: 7.301e-02] jess: 54%|█████▍ | 4307/8000 [03:06<10:36, 5.80it/s, lr: 6.0e-05 loss: 1.921e-01] jess: 54%|█████▍ | 4308/8000 [03:06<14:52, 4.14it/s, lr: 6.0e-05 loss: 1.921e-01] jess: 54%|█████▍ | 4308/8000 [03:06<14:52, 4.14it/s, lr: 6.0e-05 loss: 8.265e-02] jess: 54%|█████▍ | 4309/8000 [03:06<16:07, 3.81it/s, lr: 6.0e-05 loss: 8.265e-02] jess: 54%|█████▍ | 4309/8000 [03:06<16:07, 3.81it/s, lr: 6.0e-05 loss: 6.902e-02] jess: 54%|█████▍ | 4310/8000 [03:06<16:26, 3.74it/s, lr: 6.0e-05 loss: 6.902e-02] jess: 54%|█████▍ | 4310/8000 [03:07<16:26, 3.74it/s, lr: 6.0e-05 loss: 1.710e-01] jess: 54%|█████▍ | 4311/8000 [03:07<16:39, 3.69it/s, lr: 6.0e-05 loss: 1.710e-01] jess: 54%|█████▍ | 4311/8000 [03:07<16:39, 3.69it/s, lr: 6.0e-05 loss: 1.024e-01] jess: 54%|█████▍ | 4312/8000 [03:07<11:21, 5.41it/s, lr: 6.0e-05 loss: 1.024e-01] jess: 54%|█████▍ | 4312/8000 [03:07<11:21, 5.41it/s, lr: 6.0e-05 loss: 1.252e-01] jess: 54%|█████▍ | 4313/8000 [03:07<09:43, 6.32it/s, lr: 6.0e-05 loss: 1.252e-01] jess: 54%|█████▍ | 4313/8000 [03:07<09:43, 6.32it/s, lr: 6.0e-05 loss: 1.522e-01] jess: 54%|█████▍ | 4314/8000 [03:07<14:36, 4.20it/s, lr: 6.0e-05 loss: 1.522e-01] jess: 54%|█████▍ | 4314/8000 [03:07<14:36, 4.20it/s, lr: 6.0e-05 loss: 1.079e-01] jess: 54%|█████▍ | 4315/8000 [03:07<10:39, 5.76it/s, lr: 6.0e-05 loss: 1.079e-01] jess: 54%|█████▍ | 4315/8000 [03:08<10:39, 5.76it/s, lr: 6.0e-05 loss: 1.106e-01] jess: 54%|█████▍ | 4316/8000 [03:08<14:48, 4.15it/s, lr: 6.0e-05 loss: 1.106e-01] jess: 54%|█████▍ | 4316/8000 [03:08<14:48, 4.15it/s, lr: 6.0e-05 loss: 1.863e-02] jess: 54%|█████▍ | 4317/8000 [03:08<10:45, 5.70it/s, lr: 6.0e-05 loss: 1.863e-02] jess: 54%|█████▍ | 4317/8000 [03:08<10:45, 5.70it/s, lr: 6.0e-05 loss: 1.222e-01] jess: 54%|█████▍ | 4318/8000 [03:08<09:33, 6.42it/s, lr: 6.0e-05 loss: 1.222e-01] jess: 54%|█████▍ | 4318/8000 [03:08<09:33, 6.42it/s, lr: 6.0e-05 loss: 1.184e-02] jess: 54%|█████▍ | 4319/8000 [03:08<13:18, 4.61it/s, lr: 6.0e-05 loss: 1.184e-02] jess: 54%|█████▍ | 4319/8000 [03:09<13:18, 4.61it/s, lr: 6.0e-05 loss: 5.537e-02] jess: 54%|█████▍ | 4320/8000 [03:09<15:42, 3.91it/s, lr: 6.0e-05 loss: 5.537e-02] jess: 54%|█████▍ | 4320/8000 [03:09<15:42, 3.91it/s, lr: 6.0e-05 loss: 1.104e-01] jess: 54%|█████▍ | 4321/8000 [03:09<11:06, 5.52it/s, lr: 6.0e-05 loss: 1.104e-01] jess: 54%|█████▍ | 4321/8000 [03:09<11:06, 5.52it/s, lr: 6.0e-05 loss: 1.377e-01] jess: 54%|█████▍ | 4322/8000 [03:09<14:59, 4.09it/s, lr: 6.0e-05 loss: 1.377e-01] jess: 54%|█████▍ | 4322/8000 [03:09<14:59, 4.09it/s, lr: 6.0e-05 loss: 1.245e-01] jess: 54%|█████▍ | 4323/8000 [03:09<16:00, 3.83it/s, lr: 6.0e-05 loss: 1.245e-01] jess: 54%|█████▍ | 4323/8000 [03:09<16:00, 3.83it/s, lr: 6.0e-05 loss: 2.309e-01] jess: 54%|█████▍ | 4324/8000 [03:09<11:06, 5.51it/s, lr: 6.0e-05 loss: 2.309e-01] jess: 54%|█████▍ | 4324/8000 [03:10<11:06, 5.51it/s, lr: 6.0e-05 loss: 8.800e-02] jess: 54%|█████▍ | 4325/8000 [03:10<09:35, 6.39it/s, lr: 6.0e-05 loss: 8.800e-02] jess: 54%|█████▍ | 4325/8000 [03:10<09:35, 6.39it/s, lr: 6.0e-05 loss: 8.826e-03] jess: 54%|█████▍ | 4326/8000 [03:10<21:02, 2.91it/s, lr: 6.0e-05 loss: 8.826e-03] jess: 54%|█████▍ | 4326/8000 [03:10<21:02, 2.91it/s, lr: 6.0e-05 loss: 8.163e-02] jess: 54%|█████▍ | 4327/8000 [03:10<12:40, 4.83it/s, lr: 6.0e-05 loss: 8.163e-02] jess: 54%|█████▍ | 4327/8000 [03:10<12:40, 4.83it/s, lr: 6.0e-05 loss: 1.532e-02] jess: 54%|█████▍ | 4328/8000 [03:10<10:04, 6.08it/s, lr: 6.0e-05 loss: 1.532e-02] jess: 54%|█████▍ | 4328/8000 [03:11<10:04, 6.08it/s, lr: 6.0e-05 loss: 6.952e-02] jess: 54%|█████▍ | 4329/8000 [03:11<15:09, 4.04it/s, lr: 6.0e-05 loss: 6.952e-02] jess: 54%|█████▍ | 4329/8000 [03:11<15:09, 4.04it/s, lr: 6.0e-05 loss: 1.201e-01] jess: 54%|█████▍ | 4330/8000 [03:11<10:48, 5.66it/s, lr: 6.0e-05 loss: 1.201e-01] jess: 54%|█████▍ | 4330/8000 [03:11<10:48, 5.66it/s, lr: 6.0e-05 loss: 1.250e-01] jess: 54%|█████▍ | 4331/8000 [03:11<14:46, 4.14it/s, lr: 6.0e-05 loss: 1.250e-01] jess: 54%|█████▍ | 4331/8000 [03:11<14:46, 4.14it/s, lr: 6.0e-05 loss: 1.309e-01] jess: 54%|█████▍ | 4332/8000 [03:11<15:58, 3.83it/s, lr: 6.0e-05 loss: 1.309e-01] jess: 54%|█████▍ | 4332/8000 [03:11<15:58, 3.83it/s, lr: 6.0e-05 loss: 1.162e-01] jess: 54%|█████▍ | 4333/8000 [03:11<11:02, 5.53it/s, lr: 6.0e-05 loss: 1.162e-01] jess: 54%|█████▍ | 4333/8000 [03:11<11:02, 5.53it/s, lr: 6.0e-05 loss: 1.040e-01] jess: 54%|█████▍ | 4334/8000 [03:11<09:36, 6.36it/s, lr: 6.0e-05 loss: 1.040e-01] jess: 54%|█████▍ | 4334/8000 [03:12<09:36, 6.36it/s, lr: 6.0e-05 loss: 1.628e-01] jess: 54%|█████▍ | 4335/8000 [03:12<09:08, 6.68it/s, lr: 6.0e-05 loss: 1.628e-01] jess: 54%|█████▍ | 4335/8000 [03:12<09:08, 6.68it/s, lr: 6.0e-05 loss: 1.317e-01] jess: 54%|█████▍ | 4336/8000 [03:12<09:01, 6.76it/s, lr: 6.0e-05 loss: 1.317e-01] jess: 54%|█████▍ | 4336/8000 [03:12<09:01, 6.76it/s, lr: 6.0e-05 loss: 5.425e-02] jess: 54%|█████▍ | 4337/8000 [03:12<14:24, 4.24it/s, lr: 6.0e-05 loss: 5.425e-02] jess: 54%|█████▍ | 4337/8000 [03:12<14:24, 4.24it/s, lr: 6.0e-05 loss: 4.769e-02] jess: 54%|█████▍ | 4338/8000 [03:12<10:36, 5.75it/s, lr: 6.0e-05 loss: 4.769e-02] jess: 54%|█████▍ | 4338/8000 [03:12<10:36, 5.75it/s, lr: 6.0e-05 loss: 4.558e-02] jess: 54%|█████▍ | 4339/8000 [03:12<13:37, 4.48it/s, lr: 6.0e-05 loss: 4.558e-02] jess: 54%|█████▍ | 4339/8000 [03:13<13:37, 4.48it/s, lr: 6.0e-05 loss: 3.284e-02] jess: 54%|█████▍ | 4340/8000 [03:13<15:24, 3.96it/s, lr: 6.0e-05 loss: 3.284e-02] jess: 54%|█████▍ | 4340/8000 [03:13<15:24, 3.96it/s, lr: 6.0e-05 loss: 5.303e-02] jess: 54%|█████▍ | 4341/8000 [03:13<16:15, 3.75it/s, lr: 6.0e-05 loss: 5.303e-02] jess: 54%|█████▍ | 4341/8000 [03:13<16:15, 3.75it/s, lr: 6.0e-05 loss: 5.596e-02] jess: 54%|█████▍ | 4342/8000 [03:13<11:07, 5.48it/s, lr: 6.0e-05 loss: 5.596e-02] jess: 54%|█████▍ | 4342/8000 [03:13<11:07, 5.48it/s, lr: 6.0e-05 loss: 1.446e-01] jess: 54%|█████▍ | 4343/8000 [03:13<09:39, 6.31it/s, lr: 6.0e-05 loss: 1.446e-01] jess: 54%|█████▍ | 4343/8000 [03:13<09:39, 6.31it/s, lr: 6.0e-05 loss: 1.833e-01] jess: 54%|█████▍ | 4344/8000 [03:13<09:09, 6.65it/s, lr: 6.0e-05 loss: 1.833e-01] jess: 54%|█████▍ | 4344/8000 [03:14<09:09, 6.65it/s, lr: 6.0e-05 loss: 1.073e-01] jess: 54%|█████▍ | 4345/8000 [03:14<09:01, 6.76it/s, lr: 6.0e-05 loss: 1.073e-01] jess: 54%|█████▍ | 4345/8000 [03:14<09:01, 6.76it/s, lr: 6.0e-05 loss: 8.714e-02] jess: 54%|█████▍ | 4346/8000 [03:14<14:19, 4.25it/s, lr: 6.0e-05 loss: 8.714e-02] jess: 54%|█████▍ | 4346/8000 [03:14<14:19, 4.25it/s, lr: 6.0e-05 loss: 1.510e-01] jess: 54%|█████▍ | 4347/8000 [03:14<16:20, 3.73it/s, lr: 6.0e-05 loss: 1.510e-01] jess: 54%|█████▍ | 4347/8000 [03:14<16:20, 3.73it/s, lr: 6.0e-05 loss: 1.693e-01] jess: 54%|█████▍ | 4348/8000 [03:14<11:16, 5.40it/s, lr: 6.0e-05 loss: 1.693e-01] jess: 54%|█████▍ | 4348/8000 [03:15<11:16, 5.40it/s, lr: 6.0e-05 loss: 8.289e-02] jess: 54%|█████▍ | 4349/8000 [03:15<15:01, 4.05it/s, lr: 6.0e-05 loss: 8.289e-02] jess: 54%|█████▍ | 4349/8000 [03:15<15:01, 4.05it/s, lr: 6.0e-05 loss: 9.588e-02] jess: 54%|█████▍ | 4350/8000 [03:15<22:44, 2.67it/s, lr: 6.0e-05 loss: 9.588e-02] jess: 54%|█████▍ | 4350/8000 [03:15<22:44, 2.67it/s, lr: 6.0e-05 loss: 9.546e-02]
Saving at step 4350
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004350.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 54%|█████▍ | 4351/8000 [03:16<22:44, 2.67it/s, lr: 6.0e-05 loss: 1.667e-01] jess: 54%|█████▍ | 4352/8000 [03:16<17:21, 3.50it/s, lr: 6.0e-05 loss: 1.667e-01] jess: 54%|█████▍ | 4352/8000 [03:16<17:21, 3.50it/s, lr: 6.0e-05 loss: 2.479e-03] jess: 54%|█████▍ | 4353/8000 [03:16<12:31, 4.85it/s, lr: 6.0e-05 loss: 2.479e-03] jess: 54%|█████▍ | 4353/8000 [03:16<12:31, 4.85it/s, lr: 6.0e-05 loss: 1.167e-01] jess: 54%|█████▍ | 4354/8000 [03:16<15:07, 4.02it/s, lr: 6.0e-05 loss: 1.167e-01] jess: 54%|█████▍ | 4354/8000 [03:16<15:07, 4.02it/s, lr: 6.0e-05 loss: 6.710e-03] jess: 54%|█████▍ | 4355/8000 [03:16<10:52, 5.58it/s, lr: 6.0e-05 loss: 6.710e-03] jess: 54%|█████▍ | 4355/8000 [03:17<10:52, 5.58it/s, lr: 6.0e-05 loss: 9.921e-02] jess: 54%|█████▍ | 4356/8000 [03:17<14:46, 4.11it/s, lr: 6.0e-05 loss: 9.921e-02] jess: 54%|█████▍ | 4356/8000 [03:17<14:46, 4.11it/s, lr: 6.0e-05 loss: 7.200e-03] jess: 54%|█████▍ | 4357/8000 [03:17<15:50, 3.83it/s, lr: 6.0e-05 loss: 7.200e-03] jess: 54%|█████▍ | 4357/8000 [03:17<15:50, 3.83it/s, lr: 6.0e-05 loss: 1.070e-01] jess: 54%|█████▍ | 4358/8000 [03:17<10:59, 5.52it/s, lr: 6.0e-05 loss: 1.070e-01] jess: 54%|█████▍ | 4358/8000 [03:17<10:59, 5.52it/s, lr: 6.0e-05 loss: 1.412e-01] jess: 54%|█████▍ | 4359/8000 [03:17<14:45, 4.11it/s, lr: 6.0e-05 loss: 1.412e-01] jess: 54%|█████▍ | 4359/8000 [03:17<14:45, 4.11it/s, lr: 6.0e-05 loss: 1.016e-01] jess: 55%|█████▍ | 4360/8000 [03:17<15:52, 3.82it/s, lr: 6.0e-05 loss: 1.016e-01] jess: 55%|█████▍ | 4360/8000 [03:18<15:52, 3.82it/s, lr: 6.0e-05 loss: 1.206e-01] jess: 55%|█████▍ | 4361/8000 [03:18<16:49, 3.60it/s, lr: 6.0e-05 loss: 1.206e-01] jess: 55%|█████▍ | 4361/8000 [03:18<16:49, 3.60it/s, lr: 6.0e-05 loss: 1.789e-01] jess: 55%|█████▍ | 4362/8000 [03:18<16:33, 3.66it/s, lr: 6.0e-05 loss: 1.789e-01] jess: 55%|█████▍ | 4362/8000 [03:18<16:33, 3.66it/s, lr: 6.0e-05 loss: 8.597e-02] jess: 55%|█████▍ | 4363/8000 [03:18<11:11, 5.42it/s, lr: 6.0e-05 loss: 8.597e-02] jess: 55%|█████▍ | 4363/8000 [03:18<11:11, 5.42it/s, lr: 6.0e-05 loss: 5.735e-02] jess: 55%|█████▍ | 4364/8000 [03:18<14:45, 4.11it/s, lr: 6.0e-05 loss: 5.735e-02] jess: 55%|█████▍ | 4364/8000 [03:19<14:45, 4.11it/s, lr: 6.0e-05 loss: 1.852e-03] jess: 55%|█████▍ | 4365/8000 [03:19<10:59, 5.51it/s, lr: 6.0e-05 loss: 1.852e-03] jess: 55%|█████▍ | 4365/8000 [03:19<10:59, 5.51it/s, lr: 6.0e-05 loss: 5.550e-02] jess: 55%|█████▍ | 4366/8000 [03:19<09:32, 6.34it/s, lr: 6.0e-05 loss: 5.550e-02] jess: 55%|█████▍ | 4366/8000 [03:19<09:32, 6.34it/s, lr: 6.0e-05 loss: 1.125e-01] jess: 55%|█████▍ | 4367/8000 [03:19<09:08, 6.63it/s, lr: 6.0e-05 loss: 1.125e-01] jess: 55%|█████▍ | 4367/8000 [03:19<09:08, 6.63it/s, lr: 6.0e-05 loss: 8.810e-02] jess: 55%|█████▍ | 4368/8000 [03:19<14:13, 4.26it/s, lr: 6.0e-05 loss: 8.810e-02] jess: 55%|█████▍ | 4368/8000 [03:19<14:13, 4.26it/s, lr: 6.0e-05 loss: 9.385e-02] jess: 55%|█████▍ | 4369/8000 [03:19<10:30, 5.75it/s, lr: 6.0e-05 loss: 9.385e-02] jess: 55%|█████▍ | 4369/8000 [03:19<10:30, 5.75it/s, lr: 6.0e-05 loss: 6.484e-02] jess: 55%|█████▍ | 4370/8000 [03:19<09:23, 6.45it/s, lr: 6.0e-05 loss: 6.484e-02] jess: 55%|█████▍ | 4370/8000 [03:20<09:23, 6.45it/s, lr: 6.0e-05 loss: 4.465e-03] jess: 55%|█████▍ | 4371/8000 [03:20<09:06, 6.64it/s, lr: 6.0e-05 loss: 4.465e-03] jess: 55%|█████▍ | 4371/8000 [03:20<09:06, 6.64it/s, lr: 6.0e-05 loss: 5.727e-02] jess: 55%|█████▍ | 4372/8000 [03:20<14:11, 4.26it/s, lr: 6.0e-05 loss: 5.727e-02] jess: 55%|█████▍ | 4372/8000 [03:20<14:11, 4.26it/s, lr: 6.0e-05 loss: 7.074e-02] jess: 55%|█████▍ | 4373/8000 [03:20<15:41, 3.85it/s, lr: 6.0e-05 loss: 7.074e-02] jess: 55%|█████▍ | 4373/8000 [03:20<15:41, 3.85it/s, lr: 6.0e-05 loss: 1.621e-01] jess: 55%|█████▍ | 4374/8000 [03:20<10:59, 5.50it/s, lr: 6.0e-05 loss: 1.621e-01] jess: 55%|█████▍ | 4374/8000 [03:21<10:59, 5.50it/s, lr: 6.0e-05 loss: 2.630e-01] jess: 55%|█████▍ | 4375/8000 [03:21<14:39, 4.12it/s, lr: 6.0e-05 loss: 2.630e-01] jess: 55%|█████▍ | 4375/8000 [03:21<14:39, 4.12it/s, lr: 6.0e-05 loss: 1.887e-01] jess: 55%|█████▍ | 4376/8000 [03:21<26:54, 2.24it/s, lr: 6.0e-05 loss: 1.887e-01] jess: 55%|█████▍ | 4376/8000 [03:21<26:54, 2.24it/s, lr: 6.0e-05 loss: 2.113e-01] jess: 55%|█████▍ | 4377/8000 [03:21<19:36, 3.08it/s, lr: 6.0e-05 loss: 2.113e-01] jess: 55%|█████▍ | 4377/8000 [03:22<19:36, 3.08it/s, lr: 6.0e-05 loss: 6.595e-02] jess: 55%|█████▍ | 4378/8000 [03:22<12:07, 4.98it/s, lr: 6.0e-05 loss: 6.595e-02] jess: 55%|█████▍ | 4378/8000 [03:22<12:07, 4.98it/s, lr: 6.0e-05 loss: 1.226e-01] jess: 55%|█████▍ | 4379/8000 [03:22<15:06, 3.99it/s, lr: 6.0e-05 loss: 1.226e-01] jess: 55%|█████▍ | 4379/8000 [03:22<15:06, 3.99it/s, lr: 6.0e-05 loss: 1.986e-01] jess: 55%|█████▍ | 4380/8000 [03:22<15:58, 3.78it/s, lr: 6.0e-05 loss: 1.986e-01] jess: 55%|█████▍ | 4380/8000 [03:22<15:58, 3.78it/s, lr: 6.0e-05 loss: 7.953e-03] jess: 55%|█████▍ | 4381/8000 [03:22<16:10, 3.73it/s, lr: 6.0e-05 loss: 7.953e-03] jess: 55%|█████▍ | 4381/8000 [03:23<16:10, 3.73it/s, lr: 6.0e-05 loss: 1.918e-02] jess: 55%|█████▍ | 4382/8000 [03:23<15:07, 3.99it/s, lr: 6.0e-05 loss: 1.918e-02] jess: 55%|█████▍ | 4382/8000 [03:23<15:07, 3.99it/s, lr: 6.0e-05 loss: 2.224e-02] jess: 55%|█████▍ | 4383/8000 [03:23<10:57, 5.50it/s, lr: 6.0e-05 loss: 2.224e-02] jess: 55%|█████▍ | 4383/8000 [03:23<10:57, 5.50it/s, lr: 6.0e-05 loss: 1.090e-01] jess: 55%|█████▍ | 4384/8000 [03:23<09:40, 6.23it/s, lr: 6.0e-05 loss: 1.090e-01] jess: 55%|█████▍ | 4384/8000 [03:23<09:40, 6.23it/s, lr: 6.0e-05 loss: 2.155e-01] jess: 55%|█████▍ | 4385/8000 [03:23<14:24, 4.18it/s, lr: 6.0e-05 loss: 2.155e-01] jess: 55%|█████▍ | 4385/8000 [03:23<14:24, 4.18it/s, lr: 6.0e-05 loss: 8.392e-02] jess: 55%|█████▍ | 4386/8000 [03:23<15:42, 3.84it/s, lr: 6.0e-05 loss: 8.392e-02] jess: 55%|█████▍ | 4386/8000 [03:24<15:42, 3.84it/s, lr: 6.0e-05 loss: 5.228e-02] jess: 55%|█████▍ | 4387/8000 [03:24<10:59, 5.47it/s, lr: 6.0e-05 loss: 5.228e-02] jess: 55%|█████▍ | 4387/8000 [03:24<10:59, 5.47it/s, lr: 6.0e-05 loss: 1.220e-01] jess: 55%|█████▍ | 4388/8000 [03:24<14:44, 4.08it/s, lr: 6.0e-05 loss: 1.220e-01] jess: 55%|█████▍ | 4388/8000 [03:24<14:44, 4.08it/s, lr: 6.0e-05 loss: 1.416e-01] jess: 55%|█████▍ | 4389/8000 [03:24<10:39, 5.64it/s, lr: 6.0e-05 loss: 1.416e-01] jess: 55%|█████▍ | 4389/8000 [03:24<10:39, 5.64it/s, lr: 6.0e-05 loss: 1.293e-01] jess: 55%|█████▍ | 4390/8000 [03:24<14:38, 4.11it/s, lr: 6.0e-05 loss: 1.293e-01] jess: 55%|█████▍ | 4390/8000 [03:25<14:38, 4.11it/s, lr: 6.0e-05 loss: 1.299e-02] jess: 55%|█████▍ | 4391/8000 [03:25<15:36, 3.85it/s, lr: 6.0e-05 loss: 1.299e-02] jess: 55%|█████▍ | 4391/8000 [03:25<15:36, 3.85it/s, lr: 6.0e-05 loss: 4.670e-02] jess: 55%|█████▍ | 4392/8000 [03:25<10:52, 5.53it/s, lr: 6.0e-05 loss: 4.670e-02] jess: 55%|█████▍ | 4392/8000 [03:25<10:52, 5.53it/s, lr: 6.0e-05 loss: 1.639e-01] jess: 55%|█████▍ | 4393/8000 [03:25<09:28, 6.35it/s, lr: 6.0e-05 loss: 1.639e-01] jess: 55%|█████▍ | 4393/8000 [03:25<09:28, 6.35it/s, lr: 6.0e-05 loss: 2.346e-02] jess: 55%|█████▍ | 4394/8000 [03:25<09:02, 6.65it/s, lr: 6.0e-05 loss: 2.346e-02] jess: 55%|█████▍ | 4394/8000 [03:25<09:02, 6.65it/s, lr: 6.0e-05 loss: 7.288e-02] jess: 55%|█████▍ | 4395/8000 [03:25<14:01, 4.28it/s, lr: 6.0e-05 loss: 7.288e-02] jess: 55%|█████▍ | 4395/8000 [03:26<14:01, 4.28it/s, lr: 6.0e-05 loss: 4.807e-02] jess: 55%|█████▍ | 4396/8000 [03:26<15:39, 3.84it/s, lr: 6.0e-05 loss: 4.807e-02] jess: 55%|█████▍ | 4396/8000 [03:26<15:39, 3.84it/s, lr: 6.0e-05 loss: 1.409e-01] jess: 55%|█████▍ | 4397/8000 [03:26<16:05, 3.73it/s, lr: 6.0e-05 loss: 1.409e-01] jess: 55%|█████▍ | 4397/8000 [03:26<16:05, 3.73it/s, lr: 6.0e-05 loss: 3.630e-02] jess: 55%|█████▍ | 4398/8000 [03:26<11:04, 5.42it/s, lr: 6.0e-05 loss: 3.630e-02] jess: 55%|█████▍ | 4398/8000 [03:26<11:04, 5.42it/s, lr: 6.0e-05 loss: 9.581e-02] jess: 55%|█████▍ | 4399/8000 [03:26<14:42, 4.08it/s, lr: 6.0e-05 loss: 9.581e-02] jess: 55%|█████▍ | 4399/8000 [03:27<14:42, 4.08it/s, lr: 6.0e-05 loss: 1.389e-01] jess: 55%|█████▌ | 4400/8000 [03:27<22:24, 2.68it/s, lr: 6.0e-05 loss: 1.389e-01] jess: 55%|█████▌ | 4400/8000 [03:27<22:24, 2.68it/s, lr: 6.0e-05 loss: 1.118e-01]
Saving at step 4400
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004400.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.31s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.32s/it]
Generating Images: 100%|██████████| 3/3 [00:03<00:00, 1.32s/it]

jess: 55%|█████▌ | 4401/8000 [03:27<22:24, 2.68it/s, lr: 6.0e-05 loss: 2.803e-02] jess: 55%|█████▌ | 4402/8000 [03:27<17:15, 3.48it/s, lr: 6.0e-05 loss: 2.803e-02] jess: 55%|█████▌ | 4402/8000 [03:27<17:15, 3.48it/s, lr: 6.0e-05 loss: 2.051e-01] jess: 55%|█████▌ | 4403/8000 [03:27<12:27, 4.81it/s, lr: 6.0e-05 loss: 2.051e-01] jess: 55%|█████▌ | 4403/8000 [03:28<12:27, 4.81it/s, lr: 6.0e-05 loss: 6.480e-02] jess: 55%|█████▌ | 4404/8000 [03:28<10:00, 5.99it/s, lr: 6.0e-05 loss: 6.480e-02] jess: 55%|█████▌ | 4404/8000 [03:28<10:00, 5.99it/s, lr: 6.0e-05 loss: 1.310e-01] jess: 55%|█████▌ | 4405/8000 [03:28<14:18, 4.19it/s, lr: 6.0e-05 loss: 1.310e-01] jess: 55%|█████▌ | 4405/8000 [03:28<14:18, 4.19it/s, lr: 6.0e-05 loss: 3.931e-02] jess: 55%|█████▌ | 4406/8000 [03:28<15:42, 3.81it/s, lr: 6.0e-05 loss: 3.931e-02] jess: 55%|█████▌ | 4406/8000 [03:28<15:42, 3.81it/s, lr: 6.0e-05 loss: 3.766e-03] jess: 55%|█████▌ | 4407/8000 [03:28<10:54, 5.49it/s, lr: 6.0e-05 loss: 3.766e-03] jess: 55%|█████▌ | 4407/8000 [03:28<10:54, 5.49it/s, lr: 6.0e-05 loss: 2.274e-01] jess: 55%|█████▌ | 4408/8000 [03:28<09:24, 6.36it/s, lr: 6.0e-05 loss: 2.274e-01] jess: 55%|█████▌ | 4408/8000 [03:29<09:24, 6.36it/s, lr: 6.0e-05 loss: 1.449e-01] jess: 55%|█████▌ | 4409/8000 [03:29<14:16, 4.19it/s, lr: 6.0e-05 loss: 1.449e-01] jess: 55%|█████▌ | 4409/8000 [03:29<14:16, 4.19it/s, lr: 6.0e-05 loss: 1.344e-01] jess: 55%|█████▌ | 4410/8000 [03:29<15:38, 3.83it/s, lr: 6.0e-05 loss: 1.344e-01] jess: 55%|█████▌ | 4410/8000 [03:29<15:38, 3.83it/s, lr: 6.0e-05 loss: 1.277e-01] jess: 55%|█████▌ | 4411/8000 [03:29<16:05, 3.72it/s, lr: 6.0e-05 loss: 1.277e-01] jess: 55%|█████▌ | 4411/8000 [03:29<16:05, 3.72it/s, lr: 6.0e-05 loss: 5.371e-03] jess: 55%|█████▌ | 4412/8000 [03:29<10:58, 5.45it/s, lr: 6.0e-05 loss: 5.371e-03] jess: 55%|█████▌ | 4412/8000 [03:30<10:58, 5.45it/s, lr: 6.0e-05 loss: 1.472e-01] jess: 55%|█████▌ | 4413/8000 [03:30<14:38, 4.08it/s, lr: 6.0e-05 loss: 1.472e-01] jess: 55%|█████▌ | 4413/8000 [03:30<14:38, 4.08it/s, lr: 6.0e-05 loss: 1.069e-01] jess: 55%|█████▌ | 4414/8000 [03:30<10:36, 5.63it/s, lr: 6.0e-05 loss: 1.069e-01] jess: 55%|█████▌ | 4414/8000 [03:30<10:36, 5.63it/s, lr: 6.0e-05 loss: 1.064e-01] jess: 55%|█████▌ | 4415/8000 [03:30<09:19, 6.40it/s, lr: 6.0e-05 loss: 1.064e-01] jess: 55%|█████▌ | 4415/8000 [03:30<09:19, 6.40it/s, lr: 6.0e-05 loss: 2.146e-01] jess: 55%|█████▌ | 4416/8000 [03:30<14:03, 4.25it/s, lr: 6.0e-05 loss: 2.146e-01] jess: 55%|█████▌ | 4416/8000 [03:30<14:03, 4.25it/s, lr: 6.0e-05 loss: 1.258e-01] jess: 55%|█████▌ | 4417/8000 [03:30<10:25, 5.73it/s, lr: 6.0e-05 loss: 1.258e-01] jess: 55%|█████▌ | 4417/8000 [03:31<10:25, 5.73it/s, lr: 6.0e-05 loss: 1.610e-01] jess: 55%|█████▌ | 4418/8000 [03:31<14:27, 4.13it/s, lr: 6.0e-05 loss: 1.610e-01] jess: 55%|█████▌ | 4418/8000 [03:31<14:27, 4.13it/s, lr: 6.0e-05 loss: 1.294e-01] jess: 55%|█████▌ | 4419/8000 [03:31<10:28, 5.70it/s, lr: 6.0e-05 loss: 1.294e-01] jess: 55%|█████▌ | 4419/8000 [03:31<10:28, 5.70it/s, lr: 6.0e-05 loss: 1.248e-01] jess: 55%|█████▌ | 4420/8000 [03:31<14:27, 4.13it/s, lr: 6.0e-05 loss: 1.248e-01] jess: 55%|█████▌ | 4420/8000 [03:31<14:27, 4.13it/s, lr: 6.0e-05 loss: 1.342e-02] jess: 55%|█████▌ | 4421/8000 [03:31<10:33, 5.65it/s, lr: 6.0e-05 loss: 1.342e-02] jess: 55%|█████▌ | 4421/8000 [03:32<10:33, 5.65it/s, lr: 6.0e-05 loss: 9.440e-02] jess: 55%|█████▌ | 4422/8000 [03:32<14:21, 4.15it/s, lr: 6.0e-05 loss: 9.440e-02] jess: 55%|█████▌ | 4422/8000 [03:32<14:21, 4.15it/s, lr: 6.0e-05 loss: 1.095e-01] jess: 55%|█████▌ | 4423/8000 [03:32<15:32, 3.84it/s, lr: 6.0e-05 loss: 1.095e-01] jess: 55%|█████▌ | 4423/8000 [03:32<15:32, 3.84it/s, lr: 6.0e-05 loss: 9.851e-02] jess: 55%|█████▌ | 4424/8000 [03:32<10:48, 5.52it/s, lr: 6.0e-05 loss: 9.851e-02] jess: 55%|█████▌ | 4424/8000 [03:32<10:48, 5.52it/s, lr: 6.0e-05 loss: 8.036e-02] jess: 55%|█████▌ | 4425/8000 [03:32<14:28, 4.12it/s, lr: 6.0e-05 loss: 8.036e-02] jess: 55%|█████▌ | 4425/8000 [03:33<14:28, 4.12it/s, lr: 6.0e-05 loss: 1.581e-01] jess: 55%|█████▌ | 4426/8000 [03:33<26:28, 2.25it/s, lr: 6.0e-05 loss: 1.581e-01] jess: 55%|█████▌ | 4426/8000 [03:33<26:28, 2.25it/s, lr: 6.0e-05 loss: 1.280e-01] jess: 55%|█████▌ | 4427/8000 [03:33<19:17, 3.09it/s, lr: 6.0e-05 loss: 1.280e-01] jess: 55%|█████▌ | 4427/8000 [03:33<19:17, 3.09it/s, lr: 6.0e-05 loss: 5.504e-03] jess: 55%|█████▌ | 4428/8000 [03:33<16:52, 3.53it/s, lr: 6.0e-05 loss: 5.504e-03] jess: 55%|█████▌ | 4428/8000 [03:34<16:52, 3.53it/s, lr: 6.0e-05 loss: 1.902e-02] jess: 55%|█████▌ | 4429/8000 [03:34<15:12, 3.91it/s, lr: 6.0e-05 loss: 1.902e-02] jess: 55%|█████▌ | 4429/8000 [03:34<15:12, 3.91it/s, lr: 6.0e-05 loss: 2.206e-02] jess: 55%|█████▌ | 4430/8000 [03:34<10:42, 5.56it/s, lr: 6.0e-05 loss: 2.206e-02] jess: 55%|█████▌ | 4430/8000 [03:34<10:42, 5.56it/s, lr: 6.0e-05 loss: 1.869e-02] jess: 55%|█████▌ | 4431/8000 [03:34<14:23, 4.13it/s, lr: 6.0e-05 loss: 1.869e-02] jess: 55%|█████▌ | 4431/8000 [03:34<14:23, 4.13it/s, lr: 6.0e-05 loss: 1.126e-01] jess: 55%|█████▌ | 4432/8000 [03:34<10:26, 5.69it/s, lr: 6.0e-05 loss: 1.126e-01] jess: 55%|█████▌ | 4432/8000 [03:34<10:26, 5.69it/s, lr: 6.0e-05 loss: 7.290e-02] jess: 55%|█████▌ | 4433/8000 [03:34<09:12, 6.46it/s, lr: 6.0e-05 loss: 7.290e-02] jess: 55%|█████▌ | 4433/8000 [03:34<09:12, 6.46it/s, lr: 6.0e-05 loss: 8.453e-02] jess: 55%|█████▌ | 4434/8000 [03:34<08:52, 6.70it/s, lr: 6.0e-05 loss: 8.453e-02] jess: 55%|█████▌ | 4434/8000 [03:35<08:52, 6.70it/s, lr: 6.0e-05 loss: 4.653e-02] jess: 55%|█████▌ | 4435/8000 [03:35<08:47, 6.75it/s, lr: 6.0e-05 loss: 4.653e-02] jess: 55%|█████▌ | 4435/8000 [03:35<08:47, 6.75it/s, lr: 6.0e-05 loss: 1.239e-01] jess: 55%|█████▌ | 4436/8000 [03:35<13:53, 4.28it/s, lr: 6.0e-05 loss: 1.239e-01] jess: 55%|█████▌ | 4436/8000 [03:35<13:53, 4.28it/s, lr: 6.0e-05 loss: 6.330e-02] jess: 55%|█████▌ | 4437/8000 [03:35<10:43, 5.54it/s, lr: 6.0e-05 loss: 6.330e-02] jess: 55%|█████▌ | 4437/8000 [03:35<10:43, 5.54it/s, lr: 6.0e-05 loss: 7.048e-02] jess: 55%|█████▌ | 4438/8000 [03:35<09:15, 6.41it/s, lr: 6.0e-05 loss: 7.048e-02] jess: 55%|█████▌ | 4438/8000 [03:35<09:15, 6.41it/s, lr: 6.0e-05 loss: 4.036e-02] jess: 55%|█████▌ | 4439/8000 [03:35<14:00, 4.24it/s, lr: 6.0e-05 loss: 4.036e-02] jess: 55%|█████▌ | 4439/8000 [03:36<14:00, 4.24it/s, lr: 6.0e-05 loss: 1.500e-01] jess: 56%|█████▌ | 4440/8000 [03:36<15:30, 3.83it/s, lr: 6.0e-05 loss: 1.500e-01] jess: 56%|█████▌ | 4440/8000 [03:36<15:30, 3.83it/s, lr: 6.0e-05 loss: 2.261e-01] jess: 56%|█████▌ | 4441/8000 [03:36<15:55, 3.73it/s, lr: 6.0e-05 loss: 2.261e-01] jess: 56%|█████▌ | 4441/8000 [03:36<15:55, 3.73it/s, lr: 6.0e-05 loss: 1.243e-01] jess: 56%|█████▌ | 4442/8000 [03:36<10:54, 5.44it/s, lr: 6.0e-05 loss: 1.243e-01] jess: 56%|█████▌ | 4442/8000 [03:36<10:54, 5.44it/s, lr: 6.0e-05 loss: 7.816e-02] jess: 56%|█████▌ | 4443/8000 [03:36<09:23, 6.31it/s, lr: 6.0e-05 loss: 7.816e-02] jess: 56%|█████▌ | 4443/8000 [03:36<09:23, 6.31it/s, lr: 6.0e-05 loss: 4.992e-02] jess: 56%|█████▌ | 4444/8000 [03:36<08:49, 6.71it/s, lr: 6.0e-05 loss: 4.992e-02] jess: 56%|█████▌ | 4444/8000 [03:37<08:49, 6.71it/s, lr: 6.0e-05 loss: 1.370e-01] jess: 56%|█████▌ | 4445/8000 [03:37<13:55, 4.25it/s, lr: 6.0e-05 loss: 1.370e-01] jess: 56%|█████▌ | 4445/8000 [03:37<13:55, 4.25it/s, lr: 6.0e-05 loss: 4.903e-02] jess: 56%|█████▌ | 4446/8000 [03:37<10:18, 5.75it/s, lr: 6.0e-05 loss: 4.903e-02] jess: 56%|█████▌ | 4446/8000 [03:37<10:18, 5.75it/s, lr: 6.0e-05 loss: 7.093e-02] jess: 56%|█████▌ | 4447/8000 [03:37<14:05, 4.20it/s, lr: 6.0e-05 loss: 7.093e-02] jess: 56%|█████▌ | 4447/8000 [03:37<14:05, 4.20it/s, lr: 6.0e-05 loss: 5.739e-02] jess: 56%|█████▌ | 4448/8000 [03:37<10:21, 5.72it/s, lr: 6.0e-05 loss: 5.739e-02] jess: 56%|█████▌ | 4448/8000 [03:37<10:21, 5.72it/s, lr: 6.0e-05 loss: 5.116e-03] jess: 56%|█████▌ | 4449/8000 [03:37<14:13, 4.16it/s, lr: 6.0e-05 loss: 5.116e-03] jess: 56%|█████▌ | 4449/8000 [03:38<14:13, 4.16it/s, lr: 6.0e-05 loss: 1.484e-02] jess: 56%|█████▌ | 4450/8000 [03:38<25:18, 2.34it/s, lr: 6.0e-05 loss: 1.484e-02] jess: 56%|█████▌ | 4450/8000 [03:38<25:18, 2.34it/s, lr: 6.0e-05 loss: 1.303e-01]
Saving at step 4450
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004450.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 56%|█████▌ | 4451/8000 [03:39<25:18, 2.34it/s, lr: 6.0e-05 loss: 1.545e-01] jess: 56%|█████▌ | 4452/8000 [03:39<17:34, 3.36it/s, lr: 6.0e-05 loss: 1.545e-01] jess: 56%|█████▌ | 4452/8000 [03:39<17:34, 3.36it/s, lr: 6.0e-05 loss: 1.304e-01] jess: 56%|█████▌ | 4453/8000 [03:39<16:42, 3.54it/s, lr: 6.0e-05 loss: 1.304e-01] jess: 56%|█████▌ | 4453/8000 [03:39<16:42, 3.54it/s, lr: 6.0e-05 loss: 1.742e-01] jess: 56%|█████▌ | 4454/8000 [03:39<11:27, 5.16it/s, lr: 6.0e-05 loss: 1.742e-01] jess: 56%|█████▌ | 4454/8000 [03:39<11:27, 5.16it/s, lr: 6.0e-05 loss: 1.388e-01] jess: 56%|█████▌ | 4455/8000 [03:39<14:35, 4.05it/s, lr: 6.0e-05 loss: 1.388e-01] jess: 56%|█████▌ | 4455/8000 [03:40<14:35, 4.05it/s, lr: 6.0e-05 loss: 1.338e-01] jess: 56%|█████▌ | 4456/8000 [03:40<15:34, 3.79it/s, lr: 6.0e-05 loss: 1.338e-01] jess: 56%|█████▌ | 4456/8000 [03:40<15:34, 3.79it/s, lr: 6.0e-05 loss: 6.781e-02] jess: 56%|█████▌ | 4457/8000 [03:40<10:45, 5.49it/s, lr: 6.0e-05 loss: 6.781e-02] jess: 56%|█████▌ | 4457/8000 [03:40<10:45, 5.49it/s, lr: 6.0e-05 loss: 3.118e-03] jess: 56%|█████▌ | 4458/8000 [03:40<09:17, 6.35it/s, lr: 6.0e-05 loss: 3.118e-03] jess: 56%|█████▌ | 4458/8000 [03:40<09:17, 6.35it/s, lr: 6.0e-05 loss: 1.007e-01] jess: 56%|█████▌ | 4459/8000 [03:40<13:59, 4.22it/s, lr: 6.0e-05 loss: 1.007e-01] jess: 56%|█████▌ | 4459/8000 [03:41<13:59, 4.22it/s, lr: 6.0e-05 loss: 1.531e-01] jess: 56%|█████▌ | 4460/8000 [03:41<15:23, 3.83it/s, lr: 6.0e-05 loss: 1.531e-01] jess: 56%|█████▌ | 4460/8000 [03:41<15:23, 3.83it/s, lr: 6.0e-05 loss: 9.210e-02] jess: 56%|█████▌ | 4461/8000 [03:41<10:45, 5.49it/s, lr: 6.0e-05 loss: 9.210e-02] jess: 56%|█████▌ | 4461/8000 [03:41<10:45, 5.49it/s, lr: 6.0e-05 loss: 1.768e-01] jess: 56%|█████▌ | 4462/8000 [03:41<14:18, 4.12it/s, lr: 6.0e-05 loss: 1.768e-01] jess: 56%|█████▌ | 4462/8000 [03:41<14:18, 4.12it/s, lr: 6.0e-05 loss: 8.810e-02] jess: 56%|█████▌ | 4463/8000 [03:41<15:30, 3.80it/s, lr: 6.0e-05 loss: 8.810e-02] jess: 56%|█████▌ | 4463/8000 [03:41<15:30, 3.80it/s, lr: 6.0e-05 loss: 6.543e-02] jess: 56%|█████▌ | 4464/8000 [03:41<10:45, 5.48it/s, lr: 6.0e-05 loss: 6.543e-02] jess: 56%|█████▌ | 4464/8000 [03:42<10:45, 5.48it/s, lr: 6.0e-05 loss: 4.436e-03] jess: 56%|█████▌ | 4465/8000 [03:42<09:20, 6.30it/s, lr: 6.0e-05 loss: 4.436e-03] jess: 56%|█████▌ | 4465/8000 [03:42<09:20, 6.30it/s, lr: 6.0e-05 loss: 1.222e-01] jess: 56%|█████▌ | 4466/8000 [03:42<13:58, 4.21it/s, lr: 6.0e-05 loss: 1.222e-01] jess: 56%|█████▌ | 4466/8000 [03:42<13:58, 4.21it/s, lr: 6.0e-05 loss: 1.716e-01] jess: 56%|█████▌ | 4467/8000 [03:42<10:16, 5.73it/s, lr: 6.0e-05 loss: 1.716e-01] jess: 56%|█████▌ | 4467/8000 [03:42<10:16, 5.73it/s, lr: 6.0e-05 loss: 7.403e-03] jess: 56%|█████▌ | 4468/8000 [03:42<09:10, 6.41it/s, lr: 6.0e-05 loss: 7.403e-03] jess: 56%|█████▌ | 4468/8000 [03:42<09:10, 6.41it/s, lr: 6.0e-05 loss: 1.007e-01] jess: 56%|█████▌ | 4469/8000 [03:42<13:54, 4.23it/s, lr: 6.0e-05 loss: 1.007e-01] jess: 56%|█████▌ | 4469/8000 [03:43<13:54, 4.23it/s, lr: 6.0e-05 loss: 8.195e-02] jess: 56%|█████▌ | 4470/8000 [03:43<10:15, 5.73it/s, lr: 6.0e-05 loss: 8.195e-02] jess: 56%|█████▌ | 4470/8000 [03:43<10:15, 5.73it/s, lr: 6.0e-05 loss: 1.282e-01] jess: 56%|█████▌ | 4471/8000 [03:43<14:11, 4.14it/s, lr: 6.0e-05 loss: 1.282e-01] jess: 56%|█████▌ | 4471/8000 [03:43<14:11, 4.14it/s, lr: 6.0e-05 loss: 6.255e-02] jess: 56%|█████▌ | 4472/8000 [03:43<10:17, 5.71it/s, lr: 6.0e-05 loss: 6.255e-02] jess: 56%|█████▌ | 4472/8000 [03:43<10:17, 5.71it/s, lr: 6.0e-05 loss: 5.562e-02] jess: 56%|█████▌ | 4473/8000 [03:43<09:32, 6.16it/s, lr: 6.0e-05 loss: 5.562e-02] jess: 56%|█████▌ | 4473/8000 [03:43<09:32, 6.16it/s, lr: 6.0e-05 loss: 1.922e-01] jess: 56%|█████▌ | 4474/8000 [03:43<14:01, 4.19it/s, lr: 6.0e-05 loss: 1.922e-01] jess: 56%|█████▌ | 4474/8000 [03:44<14:01, 4.19it/s, lr: 6.0e-05 loss: 8.001e-02] jess: 56%|█████▌ | 4475/8000 [03:44<15:19, 3.83it/s, lr: 6.0e-05 loss: 8.001e-02] jess: 56%|█████▌ | 4475/8000 [03:44<15:19, 3.83it/s, lr: 6.0e-05 loss: 3.622e-02] jess: 56%|█████▌ | 4476/8000 [03:44<22:13, 2.64it/s, lr: 6.0e-05 loss: 3.622e-02] jess: 56%|█████▌ | 4476/8000 [03:44<22:13, 2.64it/s, lr: 6.0e-05 loss: 1.009e-01] jess: 56%|█████▌ | 4477/8000 [03:44<12:51, 4.57it/s, lr: 6.0e-05 loss: 1.009e-01] jess: 56%|█████▌ | 4477/8000 [03:44<12:51, 4.57it/s, lr: 6.0e-05 loss: 1.669e-01] jess: 56%|█████▌ | 4478/8000 [03:44<15:02, 3.90it/s, lr: 6.0e-05 loss: 1.669e-01] jess: 56%|█████▌ | 4478/8000 [03:45<15:02, 3.90it/s, lr: 6.0e-05 loss: 8.513e-02] jess: 56%|█████▌ | 4479/8000 [03:45<15:41, 3.74it/s, lr: 6.0e-05 loss: 8.513e-02] jess: 56%|█████▌ | 4479/8000 [03:45<15:41, 3.74it/s, lr: 6.0e-05 loss: 6.345e-03] jess: 56%|█████▌ | 4480/8000 [03:45<10:44, 5.46it/s, lr: 6.0e-05 loss: 6.345e-03] jess: 56%|█████▌ | 4480/8000 [03:45<10:44, 5.46it/s, lr: 6.0e-05 loss: 1.125e-01] jess: 56%|█████▌ | 4481/8000 [03:45<09:39, 6.07it/s, lr: 6.0e-05 loss: 1.125e-01] jess: 56%|█████▌ | 4481/8000 [03:45<09:39, 6.07it/s, lr: 6.0e-05 loss: 1.384e-01] jess: 56%|█████▌ | 4482/8000 [03:45<08:55, 6.57it/s, lr: 6.0e-05 loss: 1.384e-01] jess: 56%|█████▌ | 4482/8000 [03:45<08:55, 6.57it/s, lr: 6.0e-05 loss: 1.189e-01] jess: 56%|█████▌ | 4483/8000 [03:45<13:48, 4.24it/s, lr: 6.0e-05 loss: 1.189e-01] jess: 56%|█████▌ | 4483/8000 [03:46<13:48, 4.24it/s, lr: 6.0e-05 loss: 3.575e-02] jess: 56%|█████▌ | 4484/8000 [03:46<10:09, 5.76it/s, lr: 6.0e-05 loss: 3.575e-02] jess: 56%|█████▌ | 4484/8000 [03:46<10:09, 5.76it/s, lr: 6.0e-05 loss: 1.819e-01] jess: 56%|█████▌ | 4485/8000 [03:46<14:09, 4.14it/s, lr: 6.0e-05 loss: 1.819e-01] jess: 56%|█████▌ | 4485/8000 [03:46<14:09, 4.14it/s, lr: 6.0e-05 loss: 3.820e-02] jess: 56%|█████▌ | 4486/8000 [03:46<15:27, 3.79it/s, lr: 6.0e-05 loss: 3.820e-02] jess: 56%|█████▌ | 4486/8000 [03:46<15:27, 3.79it/s, lr: 6.0e-05 loss: 1.399e-01] jess: 56%|█████▌ | 4487/8000 [03:46<10:41, 5.47it/s, lr: 6.0e-05 loss: 1.399e-01] jess: 56%|█████▌ | 4487/8000 [03:46<10:41, 5.47it/s, lr: 6.0e-05 loss: 1.431e-01] jess: 56%|█████▌ | 4488/8000 [03:46<09:15, 6.33it/s, lr: 6.0e-05 loss: 1.431e-01] jess: 56%|█████▌ | 4488/8000 [03:47<09:15, 6.33it/s, lr: 6.0e-05 loss: 4.324e-02] jess: 56%|█████▌ | 4489/8000 [03:47<13:37, 4.29it/s, lr: 6.0e-05 loss: 4.324e-02] jess: 56%|█████▌ | 4489/8000 [03:47<13:37, 4.29it/s, lr: 6.0e-05 loss: 6.024e-03] jess: 56%|█████▌ | 4490/8000 [03:47<15:04, 3.88it/s, lr: 6.0e-05 loss: 6.024e-03] jess: 56%|█████▌ | 4490/8000 [03:47<15:04, 3.88it/s, lr: 6.0e-05 loss: 7.425e-02] jess: 56%|█████▌ | 4491/8000 [03:47<15:39, 3.73it/s, lr: 6.0e-05 loss: 7.425e-02] jess: 56%|█████▌ | 4491/8000 [03:47<15:39, 3.73it/s, lr: 6.0e-05 loss: 7.260e-02] jess: 56%|█████▌ | 4492/8000 [03:47<10:41, 5.47it/s, lr: 6.0e-05 loss: 7.260e-02] jess: 56%|█████▌ | 4492/8000 [03:48<10:41, 5.47it/s, lr: 6.0e-05 loss: 2.985e-03] jess: 56%|█████▌ | 4493/8000 [03:48<09:13, 6.34it/s, lr: 6.0e-05 loss: 2.985e-03] jess: 56%|█████▌ | 4493/8000 [03:48<09:13, 6.34it/s, lr: 6.0e-05 loss: 6.602e-02] jess: 56%|█████▌ | 4494/8000 [03:48<08:48, 6.63it/s, lr: 6.0e-05 loss: 6.602e-02] jess: 56%|█████▌ | 4494/8000 [03:48<08:48, 6.63it/s, lr: 6.0e-05 loss: 6.109e-03] jess: 56%|█████▌ | 4495/8000 [03:48<12:34, 4.65it/s, lr: 6.0e-05 loss: 6.109e-03] jess: 56%|█████▌ | 4495/8000 [03:48<12:34, 4.65it/s, lr: 6.0e-05 loss: 1.361e-01] jess: 56%|█████▌ | 4496/8000 [03:48<09:47, 5.96it/s, lr: 6.0e-05 loss: 1.361e-01] jess: 56%|█████▌ | 4496/8000 [03:48<09:47, 5.96it/s, lr: 6.0e-05 loss: 1.182e-01] jess: 56%|█████▌ | 4497/8000 [03:48<13:55, 4.19it/s, lr: 6.0e-05 loss: 1.182e-01] jess: 56%|█████▌ | 4497/8000 [03:49<13:55, 4.19it/s, lr: 6.0e-05 loss: 2.153e-01] jess: 56%|█████▌ | 4498/8000 [03:49<15:14, 3.83it/s, lr: 6.0e-05 loss: 2.153e-01] jess: 56%|█████▌ | 4498/8000 [03:49<15:14, 3.83it/s, lr: 6.0e-05 loss: 2.087e-01] jess: 56%|█████▌ | 4499/8000 [03:49<10:34, 5.51it/s, lr: 6.0e-05 loss: 2.087e-01] jess: 56%|█████▌ | 4499/8000 [03:49<10:34, 5.51it/s, lr: 6.0e-05 loss: 3.225e-02] jess: 56%|█████▋ | 4500/8000 [03:49<20:41, 2.82it/s, lr: 6.0e-05 loss: 3.225e-02] jess: 56%|█████▋ | 4500/8000 [03:49<20:41, 2.82it/s, lr: 6.0e-05 loss: 1.268e-01]
Saving at step 4500
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004500.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.33s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.34s/it]
Generating Images: 100%|██████████| 3/3 [00:04<00:00, 1.34s/it]

jess: 56%|█████▋ | 4501/8000 [03:50<20:40, 2.82it/s, lr: 6.0e-05 loss: 1.938e-01] jess: 56%|█████▋ | 4502/8000 [03:50<16:39, 3.50it/s, lr: 6.0e-05 loss: 1.938e-01] jess: 56%|█████▋ | 4502/8000 [03:50<16:39, 3.50it/s, lr: 6.0e-05 loss: 1.395e-01] jess: 56%|█████▋ | 4503/8000 [03:50<12:01, 4.85it/s, lr: 6.0e-05 loss: 1.395e-01] jess: 56%|█████▋ | 4503/8000 [03:50<12:01, 4.85it/s, lr: 6.0e-05 loss: 2.431e-01] jess: 56%|█████▋ | 4504/8000 [03:50<14:32, 4.01it/s, lr: 6.0e-05 loss: 2.431e-01] jess: 56%|█████▋ | 4504/8000 [03:51<14:32, 4.01it/s, lr: 6.0e-05 loss: 1.939e-01] jess: 56%|█████▋ | 4505/8000 [03:51<15:17, 3.81it/s, lr: 6.0e-05 loss: 1.939e-01] jess: 56%|█████▋ | 4505/8000 [03:51<15:17, 3.81it/s, lr: 6.0e-05 loss: 1.474e-01] jess: 56%|█████▋ | 4506/8000 [03:51<15:39, 3.72it/s, lr: 6.0e-05 loss: 1.474e-01] jess: 56%|█████▋ | 4506/8000 [03:51<15:39, 3.72it/s, lr: 6.0e-05 loss: 8.142e-02] jess: 56%|█████▋ | 4507/8000 [03:51<15:44, 3.70it/s, lr: 6.0e-05 loss: 8.142e-02] jess: 56%|█████▋ | 4507/8000 [03:51<15:44, 3.70it/s, lr: 6.0e-05 loss: 7.741e-02] jess: 56%|█████▋ | 4508/8000 [03:51<15:39, 3.72it/s, lr: 6.0e-05 loss: 7.741e-02] jess: 56%|█████▋ | 4508/8000 [03:52<15:39, 3.72it/s, lr: 6.0e-05 loss: 1.276e-01] jess: 56%|█████▋ | 4509/8000 [03:52<11:07, 5.23it/s, lr: 6.0e-05 loss: 1.276e-01] jess: 56%|█████▋ | 4509/8000 [03:52<11:07, 5.23it/s, lr: 6.0e-05 loss: 6.139e-02] jess: 56%|█████▋ | 4510/8000 [03:52<09:14, 6.29it/s, lr: 6.0e-05 loss: 6.139e-02] jess: 56%|█████▋ | 4510/8000 [03:52<09:14, 6.29it/s, lr: 6.0e-05 loss: 1.346e-01] jess: 56%|█████▋ | 4511/8000 [03:52<08:45, 6.64it/s, lr: 6.0e-05 loss: 1.346e-01] jess: 56%|█████▋ | 4511/8000 [03:52<08:45, 6.64it/s, lr: 6.0e-05 loss: 1.558e-02] jess: 56%|█████▋ | 4512/8000 [03:52<08:38, 6.73it/s, lr: 6.0e-05 loss: 1.558e-02] jess: 56%|█████▋ | 4512/8000 [03:52<08:38, 6.73it/s, lr: 6.0e-05 loss: 9.203e-03] jess: 56%|█████▋ | 4513/8000 [03:52<12:29, 4.66it/s, lr: 6.0e-05 loss: 9.203e-03] jess: 56%|█████▋ | 4513/8000 [03:52<12:29, 4.66it/s, lr: 6.0e-05 loss: 5.973e-02] jess: 56%|█████▋ | 4514/8000 [03:52<09:43, 5.97it/s, lr: 6.0e-05 loss: 5.973e-02] jess: 56%|█████▋ | 4514/8000 [03:53<09:43, 5.97it/s, lr: 6.0e-05 loss: 2.451e-02] jess: 56%|█████▋ | 4515/8000 [03:53<13:40, 4.25it/s, lr: 6.0e-05 loss: 2.451e-02] jess: 56%|█████▋ | 4515/8000 [03:53<13:40, 4.25it/s, lr: 6.0e-05 loss: 6.552e-03] jess: 56%|█████▋ | 4516/8000 [03:53<14:59, 3.87it/s, lr: 6.0e-05 loss: 6.552e-03] jess: 56%|█████▋ | 4516/8000 [03:53<14:59, 3.87it/s, lr: 6.0e-05 loss: 2.169e-01] jess: 56%|█████▋ | 4517/8000 [03:53<15:33, 3.73it/s, lr: 6.0e-05 loss: 2.169e-01] jess: 56%|█████▋ | 4517/8000 [03:53<15:33, 3.73it/s, lr: 6.0e-05 loss: 8.017e-02] jess: 56%|█████▋ | 4518/8000 [03:53<10:38, 5.46it/s, lr: 6.0e-05 loss: 8.017e-02] jess: 56%|█████▋ | 4518/8000 [03:54<10:38, 5.46it/s, lr: 6.0e-05 loss: 1.160e-01] jess: 56%|█████▋ | 4519/8000 [03:54<14:07, 4.11it/s, lr: 6.0e-05 loss: 1.160e-01] jess: 56%|█████▋ | 4519/8000 [03:54<14:07, 4.11it/s, lr: 6.0e-05 loss: 9.389e-02] jess: 56%|█████▋ | 4520/8000 [03:54<10:13, 5.67it/s, lr: 6.0e-05 loss: 9.389e-02] jess: 56%|█████▋ | 4520/8000 [03:54<10:13, 5.67it/s, lr: 6.0e-05 loss: 6.481e-02] jess: 57%|█████▋ | 4521/8000 [03:54<14:05, 4.11it/s, lr: 6.0e-05 loss: 6.481e-02] jess: 57%|█████▋ | 4521/8000 [03:54<14:05, 4.11it/s, lr: 6.0e-05 loss: 2.093e-01] jess: 57%|█████▋ | 4522/8000 [03:54<15:07, 3.83it/s, lr: 6.0e-05 loss: 2.093e-01] jess: 57%|█████▋ | 4522/8000 [03:55<15:07, 3.83it/s, lr: 6.0e-05 loss: 1.038e-01] jess: 57%|█████▋ | 4523/8000 [03:55<15:32, 3.73it/s, lr: 6.0e-05 loss: 1.038e-01] jess: 57%|█████▋ | 4523/8000 [03:55<15:32, 3.73it/s, lr: 6.0e-05 loss: 1.015e-01] jess: 57%|█████▋ | 4524/8000 [03:55<10:38, 5.45it/s, lr: 6.0e-05 loss: 1.015e-01] jess: 57%|█████▋ | 4524/8000 [03:55<10:38, 5.45it/s, lr: 6.0e-05 loss: 1.868e-01] jess: 57%|█████▋ | 4525/8000 [03:55<14:07, 4.10it/s, lr: 6.0e-05 loss: 1.868e-01] jess: 57%|█████▋ | 4525/8000 [03:56<14:07, 4.10it/s, lr: 6.0e-05 loss: 1.513e-01] jess: 57%|█████▋ | 4526/8000 [03:56<26:04, 2.22it/s, lr: 6.0e-05 loss: 1.513e-01] jess: 57%|█████▋ | 4526/8000 [03:56<26:04, 2.22it/s, lr: 6.0e-05 loss: 5.588e-02] jess: 57%|█████▋ | 4527/8000 [03:56<13:54, 4.16it/s, lr: 6.0e-05 loss: 5.588e-02] jess: 57%|█████▋ | 4527/8000 [03:56<13:54, 4.16it/s, lr: 6.0e-05 loss: 1.030e-01] jess: 57%|█████▋ | 4528/8000 [03:56<15:12, 3.80it/s, lr: 6.0e-05 loss: 1.030e-01] jess: 57%|█████▋ | 4528/8000 [03:56<15:12, 3.80it/s, lr: 6.0e-05 loss: 7.134e-03] jess: 57%|█████▋ | 4529/8000 [03:56<10:30, 5.51it/s, lr: 6.0e-05 loss: 7.134e-03] jess: 57%|█████▋ | 4529/8000 [03:56<10:30, 5.51it/s, lr: 6.0e-05 loss: 5.266e-02] jess: 57%|█████▋ | 4530/8000 [03:56<14:07, 4.09it/s, lr: 6.0e-05 loss: 5.266e-02] jess: 57%|█████▋ | 4530/8000 [03:57<14:07, 4.09it/s, lr: 6.0e-05 loss: 4.657e-02] jess: 57%|█████▋ | 4531/8000 [03:57<10:15, 5.64it/s, lr: 6.0e-05 loss: 4.657e-02] jess: 57%|█████▋ | 4531/8000 [03:57<10:15, 5.64it/s, lr: 6.0e-05 loss: 1.037e-01] jess: 57%|█████▋ | 4532/8000 [03:57<09:00, 6.41it/s, lr: 6.0e-05 loss: 1.037e-01] jess: 57%|█████▋ | 4532/8000 [03:57<09:00, 6.41it/s, lr: 6.0e-05 loss: 6.492e-02] jess: 57%|█████▋ | 4533/8000 [03:57<13:44, 4.21it/s, lr: 6.0e-05 loss: 6.492e-02] jess: 57%|█████▋ | 4533/8000 [03:57<13:44, 4.21it/s, lr: 6.0e-05 loss: 1.484e-01] jess: 57%|█████▋ | 4534/8000 [03:57<14:59, 3.85it/s, lr: 6.0e-05 loss: 1.484e-01] jess: 57%|█████▋ | 4534/8000 [03:57<14:59, 3.85it/s, lr: 6.0e-05 loss: 1.438e-02] jess: 57%|█████▋ | 4535/8000 [03:57<10:27, 5.53it/s, lr: 6.0e-05 loss: 1.438e-02] jess: 57%|█████▋ | 4535/8000 [03:58<10:27, 5.53it/s, lr: 6.0e-05 loss: 7.091e-02] jess: 57%|█████▋ | 4536/8000 [03:58<14:03, 4.11it/s, lr: 6.0e-05 loss: 7.091e-02] jess: 57%|█████▋ | 4536/8000 [03:58<14:03, 4.11it/s, lr: 6.0e-05 loss: 5.082e-02] jess: 57%|█████▋ | 4537/8000 [03:58<10:10, 5.67it/s, lr: 6.0e-05 loss: 5.082e-02] jess: 57%|█████▋ | 4537/8000 [03:58<10:10, 5.67it/s, lr: 6.0e-05 loss: 1.615e-01] jess: 57%|█████▋ | 4538/8000 [03:58<14:00, 4.12it/s, lr: 6.0e-05 loss: 1.615e-01] jess: 57%|█████▋ | 4538/8000 [03:58<14:00, 4.12it/s, lr: 6.0e-05 loss: 4.274e-02] jess: 57%|█████▋ | 4539/8000 [03:58<10:09, 5.68it/s, lr: 6.0e-05 loss: 4.274e-02] jess: 57%|█████▋ | 4539/8000 [03:58<10:09, 5.68it/s, lr: 6.0e-05 loss: 5.402e-02] jess: 57%|█████▋ | 4540/8000 [03:58<14:00, 4.12it/s, lr: 6.0e-05 loss: 5.402e-02] jess: 57%|█████▋ | 4540/8000 [03:59<14:00, 4.12it/s, lr: 6.0e-05 loss: 5.770e-02] jess: 57%|█████▋ | 4541/8000 [03:59<15:12, 3.79it/s, lr: 6.0e-05 loss: 5.770e-02] jess: 57%|█████▋ | 4541/8000 [03:59<15:12, 3.79it/s, lr: 6.0e-05 loss: 4.884e-02] jess: 57%|█████▋ | 4542/8000 [03:59<10:28, 5.50it/s, lr: 6.0e-05 loss: 4.884e-02] jess: 57%|█████▋ | 4542/8000 [03:59<10:28, 5.50it/s, lr: 6.0e-05 loss: 1.015e-01] jess: 57%|█████▋ | 4543/8000 [03:59<09:03, 6.36it/s, lr: 6.0e-05 loss: 1.015e-01] jess: 57%|█████▋ | 4543/8000 [03:59<09:03, 6.36it/s, lr: 6.0e-05 loss: 6.900e-02] jess: 57%|█████▋ | 4544/8000 [03:59<13:39, 4.22it/s, lr: 6.0e-05 loss: 6.900e-02] jess: 57%|█████▋ | 4544/8000 [04:00<13:39, 4.22it/s, lr: 6.0e-05 loss: 1.877e-01] jess: 57%|█████▋ | 4545/8000 [04:00<15:26, 3.73it/s, lr: 6.0e-05 loss: 1.877e-01] jess: 57%|█████▋ | 4545/8000 [04:00<15:26, 3.73it/s, lr: 6.0e-05 loss: 4.450e-03] jess: 57%|█████▋ | 4546/8000 [04:00<15:25, 3.73it/s, lr: 6.0e-05 loss: 4.450e-03] jess: 57%|█████▋ | 4546/8000 [04:00<15:25, 3.73it/s, lr: 6.0e-05 loss: 3.413e-02] jess: 57%|█████▋ | 4547/8000 [04:00<15:34, 3.70it/s, lr: 6.0e-05 loss: 3.413e-02] jess: 57%|█████▋ | 4547/8000 [04:00<15:34, 3.70it/s, lr: 6.0e-05 loss: 4.315e-02] jess: 57%|█████▋ | 4548/8000 [04:00<15:31, 3.71it/s, lr: 6.0e-05 loss: 4.315e-02] jess: 57%|█████▋ | 4548/8000 [04:01<15:31, 3.71it/s, lr: 6.0e-05 loss: 1.782e-01] jess: 57%|█████▋ | 4549/8000 [04:01<15:37, 3.68it/s, lr: 6.0e-05 loss: 1.782e-01] jess: 57%|█████▋ | 4549/8000 [04:01<15:37, 3.68it/s, lr: 6.0e-05 loss: 9.094e-02] jess: 57%|█████▋ | 4550/8000 [04:01<21:53, 2.63it/s, lr: 6.0e-05 loss: 9.094e-02] jess: 57%|█████▋ | 4550/8000 [04:01<21:53, 2.63it/s, lr: 6.0e-05 loss: 2.404e-01]
Saving at step 4550
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004550.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 57%|█████▋ | 4551/8000 [04:02<21:53, 2.63it/s, lr: 6.0e-05 loss: 1.017e-01] jess: 57%|█████▋ | 4552/8000 [04:02<13:52, 4.14it/s, lr: 6.0e-05 loss: 1.017e-01] jess: 57%|█████▋ | 4552/8000 [04:02<13:52, 4.14it/s, lr: 6.0e-05 loss: 7.397e-02] jess: 57%|█████▋ | 4553/8000 [04:02<10:45, 5.34it/s, lr: 6.0e-05 loss: 7.397e-02] jess: 57%|█████▋ | 4553/8000 [04:02<10:45, 5.34it/s, lr: 6.0e-05 loss: 3.177e-02] jess: 57%|█████▋ | 4554/8000 [04:02<13:49, 4.15it/s, lr: 6.0e-05 loss: 3.177e-02] jess: 57%|█████▋ | 4554/8000 [04:02<13:49, 4.15it/s, lr: 6.0e-05 loss: 1.241e-01] jess: 57%|█████▋ | 4555/8000 [04:02<10:06, 5.68it/s, lr: 6.0e-05 loss: 1.241e-01] jess: 57%|█████▋ | 4555/8000 [04:02<10:06, 5.68it/s, lr: 6.0e-05 loss: 1.060e-01] jess: 57%|█████▋ | 4556/8000 [04:02<08:56, 6.42it/s, lr: 6.0e-05 loss: 1.060e-01] jess: 57%|█████▋ | 4556/8000 [04:03<08:56, 6.42it/s, lr: 6.0e-05 loss: 7.984e-02] jess: 57%|█████▋ | 4557/8000 [04:03<13:31, 4.24it/s, lr: 6.0e-05 loss: 7.984e-02] jess: 57%|█████▋ | 4557/8000 [04:03<13:31, 4.24it/s, lr: 6.0e-05 loss: 1.168e-02] jess: 57%|█████▋ | 4558/8000 [04:03<14:43, 3.89it/s, lr: 6.0e-05 loss: 1.168e-02] jess: 57%|█████▋ | 4558/8000 [04:03<14:43, 3.89it/s, lr: 6.0e-05 loss: 1.948e-01] jess: 57%|█████▋ | 4559/8000 [04:03<15:14, 3.76it/s, lr: 6.0e-05 loss: 1.948e-01] jess: 57%|█████▋ | 4559/8000 [04:03<15:14, 3.76it/s, lr: 6.0e-05 loss: 4.963e-02] jess: 57%|█████▋ | 4560/8000 [04:03<15:28, 3.71it/s, lr: 6.0e-05 loss: 4.963e-02] jess: 57%|█████▋ | 4560/8000 [04:04<15:28, 3.71it/s, lr: 6.0e-05 loss: 2.147e-01] jess: 57%|█████▋ | 4561/8000 [04:04<15:33, 3.69it/s, lr: 6.0e-05 loss: 2.147e-01] jess: 57%|█████▋ | 4561/8000 [04:04<15:33, 3.69it/s, lr: 6.0e-05 loss: 5.706e-02] jess: 57%|█████▋ | 4562/8000 [04:04<10:32, 5.43it/s, lr: 6.0e-05 loss: 5.706e-02] jess: 57%|█████▋ | 4562/8000 [04:04<10:32, 5.43it/s, lr: 6.0e-05 loss: 1.214e-01] jess: 57%|█████▋ | 4563/8000 [04:04<09:01, 6.35it/s, lr: 6.0e-05 loss: 1.214e-01] jess: 57%|█████▋ | 4563/8000 [04:04<09:01, 6.35it/s, lr: 6.0e-05 loss: 1.849e-01] jess: 57%|█████▋ | 4564/8000 [04:04<13:30, 4.24it/s, lr: 6.0e-05 loss: 1.849e-01] jess: 57%|█████▋ | 4564/8000 [04:05<13:30, 4.24it/s, lr: 6.0e-05 loss: 9.428e-02] jess: 57%|█████▋ | 4565/8000 [04:05<14:54, 3.84it/s, lr: 6.0e-05 loss: 9.428e-02] jess: 57%|█████▋ | 4565/8000 [04:05<14:54, 3.84it/s, lr: 6.0e-05 loss: 1.173e-01] jess: 57%|█████▋ | 4566/8000 [04:05<15:23, 3.72it/s, lr: 6.0e-05 loss: 1.173e-01] jess: 57%|█████▋ | 4566/8000 [04:05<15:23, 3.72it/s, lr: 6.0e-05 loss: 1.309e-01] jess: 57%|█████▋ | 4567/8000 [04:05<10:28, 5.46it/s, lr: 6.0e-05 loss: 1.309e-01] jess: 57%|█████▋ | 4567/8000 [04:05<10:28, 5.46it/s, lr: 6.0e-05 loss: 8.557e-03] jess: 57%|█████▋ | 4568/8000 [04:05<12:55, 4.43it/s, lr: 6.0e-05 loss: 8.557e-03] jess: 57%|█████▋ | 4568/8000 [04:05<12:55, 4.43it/s, lr: 6.0e-05 loss: 1.683e-02] jess: 57%|█████▋ | 4569/8000 [04:05<09:55, 5.76it/s, lr: 6.0e-05 loss: 1.683e-02] jess: 57%|█████▋ | 4569/8000 [04:06<09:55, 5.76it/s, lr: 6.0e-05 loss: 1.488e-01] jess: 57%|█████▋ | 4570/8000 [04:06<08:53, 6.43it/s, lr: 6.0e-05 loss: 1.488e-01] jess: 57%|█████▋ | 4570/8000 [04:06<08:53, 6.43it/s, lr: 6.0e-05 loss: 2.353e-02] jess: 57%|█████▋ | 4571/8000 [04:06<13:37, 4.20it/s, lr: 6.0e-05 loss: 2.353e-02] jess: 57%|█████▋ | 4571/8000 [04:06<13:37, 4.20it/s, lr: 6.0e-05 loss: 1.544e-01] jess: 57%|█████▋ | 4572/8000 [04:06<09:58, 5.73it/s, lr: 6.0e-05 loss: 1.544e-01] jess: 57%|█████▋ | 4572/8000 [04:06<09:58, 5.73it/s, lr: 6.0e-05 loss: 1.427e-01] jess: 57%|█████▋ | 4573/8000 [04:06<13:51, 4.12it/s, lr: 6.0e-05 loss: 1.427e-01] jess: 57%|█████▋ | 4573/8000 [04:06<13:51, 4.12it/s, lr: 6.0e-05 loss: 6.395e-02] jess: 57%|█████▋ | 4574/8000 [04:06<10:02, 5.69it/s, lr: 6.0e-05 loss: 6.395e-02] jess: 57%|█████▋ | 4574/8000 [04:07<10:02, 5.69it/s, lr: 6.0e-05 loss: 4.092e-02] jess: 57%|█████▋ | 4575/8000 [04:07<08:52, 6.43it/s, lr: 6.0e-05 loss: 4.092e-02] jess: 57%|█████▋ | 4575/8000 [04:07<08:52, 6.43it/s, lr: 6.0e-05 loss: 1.557e-01] jess: 57%|█████▋ | 4576/8000 [04:07<23:55, 2.39it/s, lr: 6.0e-05 loss: 1.557e-01] jess: 57%|█████▋ | 4576/8000 [04:07<23:55, 2.39it/s, lr: 6.0e-05 loss: 9.960e-03] jess: 57%|█████▋ | 4577/8000 [04:07<13:06, 4.35it/s, lr: 6.0e-05 loss: 9.960e-03] jess: 57%|█████▋ | 4577/8000 [04:07<13:06, 4.35it/s, lr: 6.0e-05 loss: 3.291e-02] jess: 57%|█████▋ | 4578/8000 [04:07<09:52, 5.77it/s, lr: 6.0e-05 loss: 3.291e-02] jess: 57%|█████▋ | 4578/8000 [04:07<09:52, 5.77it/s, lr: 6.0e-05 loss: 3.176e-03] jess: 57%|█████▋ | 4579/8000 [04:07<08:49, 6.46it/s, lr: 6.0e-05 loss: 3.176e-03] jess: 57%|█████▋ | 4579/8000 [04:08<08:49, 6.46it/s, lr: 6.0e-05 loss: 1.461e-01] jess: 57%|█████▋ | 4580/8000 [04:08<13:28, 4.23it/s, lr: 6.0e-05 loss: 1.461e-01] jess: 57%|█████▋ | 4580/8000 [04:08<13:28, 4.23it/s, lr: 6.0e-05 loss: 6.811e-02] jess: 57%|█████▋ | 4581/8000 [04:08<10:22, 5.49it/s, lr: 6.0e-05 loss: 6.811e-02] jess: 57%|█████▋ | 4581/8000 [04:08<10:22, 5.49it/s, lr: 6.0e-05 loss: 2.131e-01] jess: 57%|█████▋ | 4582/8000 [04:08<13:47, 4.13it/s, lr: 6.0e-05 loss: 2.131e-01] jess: 57%|█████▋ | 4582/8000 [04:08<13:47, 4.13it/s, lr: 6.0e-05 loss: 2.259e-01] jess: 57%|█████▋ | 4583/8000 [04:08<10:01, 5.68it/s, lr: 6.0e-05 loss: 2.259e-01] jess: 57%|█████▋ | 4583/8000 [04:09<10:01, 5.68it/s, lr: 6.0e-05 loss: 1.313e-01] jess: 57%|█████▋ | 4584/8000 [04:09<13:46, 4.13it/s, lr: 6.0e-05 loss: 1.313e-01] jess: 57%|█████▋ | 4584/8000 [04:09<13:46, 4.13it/s, lr: 6.0e-05 loss: 8.514e-02] jess: 57%|█████▋ | 4585/8000 [04:09<10:01, 5.68it/s, lr: 6.0e-05 loss: 8.514e-02] jess: 57%|█████▋ | 4585/8000 [04:09<10:01, 5.68it/s, lr: 6.0e-05 loss: 1.617e-01] jess: 57%|█████▋ | 4586/8000 [04:09<08:49, 6.44it/s, lr: 6.0e-05 loss: 1.617e-01] jess: 57%|█████▋ | 4586/8000 [04:09<08:49, 6.44it/s, lr: 6.0e-05 loss: 5.340e-03] jess: 57%|█████▋ | 4587/8000 [04:09<08:29, 6.70it/s, lr: 6.0e-05 loss: 5.340e-03] jess: 57%|█████▋ | 4587/8000 [04:09<08:29, 6.70it/s, lr: 6.0e-05 loss: 4.731e-02] jess: 57%|█████▋ | 4588/8000 [04:09<08:24, 6.76it/s, lr: 6.0e-05 loss: 4.731e-02] jess: 57%|█████▋ | 4588/8000 [04:09<08:24, 6.76it/s, lr: 6.0e-05 loss: 5.171e-02] jess: 57%|█████▋ | 4589/8000 [04:09<13:20, 4.26it/s, lr: 6.0e-05 loss: 5.171e-02] jess: 57%|█████▋ | 4589/8000 [04:10<13:20, 4.26it/s, lr: 6.0e-05 loss: 2.073e-01] jess: 57%|█████▋ | 4590/8000 [04:10<14:48, 3.84it/s, lr: 6.0e-05 loss: 2.073e-01] jess: 57%|█████▋ | 4590/8000 [04:10<14:48, 3.84it/s, lr: 6.0e-05 loss: 1.371e-01] jess: 57%|█████▋ | 4591/8000 [04:10<15:12, 3.74it/s, lr: 6.0e-05 loss: 1.371e-01] jess: 57%|█████▋ | 4591/8000 [04:10<15:12, 3.74it/s, lr: 6.0e-05 loss: 1.785e-02] jess: 57%|█████▋ | 4592/8000 [04:10<14:14, 3.99it/s, lr: 6.0e-05 loss: 1.785e-02] jess: 57%|█████▋ | 4592/8000 [04:10<14:14, 3.99it/s, lr: 6.0e-05 loss: 7.483e-02] jess: 57%|█████▋ | 4593/8000 [04:10<10:09, 5.59it/s, lr: 6.0e-05 loss: 7.483e-02] jess: 57%|█████▋ | 4593/8000 [04:11<10:09, 5.59it/s, lr: 6.0e-05 loss: 7.882e-02] jess: 57%|█████▋ | 4594/8000 [04:11<13:51, 4.10it/s, lr: 6.0e-05 loss: 7.882e-02] jess: 57%|█████▋ | 4594/8000 [04:11<13:51, 4.10it/s, lr: 6.0e-05 loss: 6.546e-02] jess: 57%|█████▋ | 4595/8000 [04:11<14:53, 3.81it/s, lr: 6.0e-05 loss: 6.546e-02] jess: 57%|█████▋ | 4595/8000 [04:11<14:53, 3.81it/s, lr: 6.0e-05 loss: 1.092e-01] jess: 57%|█████▋ | 4596/8000 [04:11<10:18, 5.50it/s, lr: 6.0e-05 loss: 1.092e-01] jess: 57%|█████▋ | 4596/8000 [04:11<10:18, 5.50it/s, lr: 6.0e-05 loss: 1.103e-01] jess: 57%|█████▋ | 4597/8000 [04:11<13:54, 4.08it/s, lr: 6.0e-05 loss: 1.103e-01] jess: 57%|█████▋ | 4597/8000 [04:12<13:54, 4.08it/s, lr: 6.0e-05 loss: 1.943e-01] jess: 57%|█████▋ | 4598/8000 [04:12<10:01, 5.65it/s, lr: 6.0e-05 loss: 1.943e-01] jess: 57%|█████▋ | 4598/8000 [04:12<10:01, 5.65it/s, lr: 6.0e-05 loss: 6.192e-02] jess: 57%|█████▋ | 4599/8000 [04:12<09:09, 6.19it/s, lr: 6.0e-05 loss: 6.192e-02] jess: 57%|█████▋ | 4599/8000 [04:12<09:09, 6.19it/s, lr: 6.0e-05 loss: 1.001e-01] jess: 57%|█████▊ | 4600/8000 [04:12<19:40, 2.88it/s, lr: 6.0e-05 loss: 1.001e-01] jess: 57%|█████▊ | 4600/8000 [04:12<19:40, 2.88it/s, lr: 6.0e-05 loss: 1.320e-02]
Saving at step 4600
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004600.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.33s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.32s/it]
Generating Images: 100%|██████████| 3/3 [00:03<00:00, 1.32s/it]

jess: 58%|█████▊ | 4601/8000 [04:13<19:39, 2.88it/s, lr: 6.0e-05 loss: 5.189e-02] jess: 58%|█████▊ | 4602/8000 [04:13<16:02, 3.53it/s, lr: 6.0e-05 loss: 5.189e-02] jess: 58%|█████▊ | 4602/8000 [04:13<16:02, 3.53it/s, lr: 6.0e-05 loss: 1.086e-01] jess: 58%|█████▊ | 4603/8000 [04:13<15:42, 3.60it/s, lr: 6.0e-05 loss: 1.086e-01] jess: 58%|█████▊ | 4603/8000 [04:13<15:42, 3.60it/s, lr: 6.0e-05 loss: 2.209e-02] jess: 58%|█████▊ | 4604/8000 [04:13<15:28, 3.66it/s, lr: 6.0e-05 loss: 2.209e-02] jess: 58%|█████▊ | 4604/8000 [04:13<15:28, 3.66it/s, lr: 6.0e-05 loss: 1.589e-01] jess: 58%|█████▊ | 4605/8000 [04:13<10:36, 5.33it/s, lr: 6.0e-05 loss: 1.589e-01] jess: 58%|█████▊ | 4605/8000 [04:14<10:36, 5.33it/s, lr: 6.0e-05 loss: 8.432e-02] jess: 58%|█████▊ | 4606/8000 [04:14<08:59, 6.29it/s, lr: 6.0e-05 loss: 8.432e-02] jess: 58%|█████▊ | 4606/8000 [04:14<08:59, 6.29it/s, lr: 6.0e-05 loss: 1.189e-03] jess: 58%|█████▊ | 4607/8000 [04:14<08:29, 6.66it/s, lr: 6.0e-05 loss: 1.189e-03] jess: 58%|█████▊ | 4607/8000 [04:14<08:29, 6.66it/s, lr: 6.0e-05 loss: 1.654e-01] jess: 58%|█████▊ | 4608/8000 [04:14<13:15, 4.27it/s, lr: 6.0e-05 loss: 1.654e-01] jess: 58%|█████▊ | 4608/8000 [04:14<13:15, 4.27it/s, lr: 6.0e-05 loss: 1.616e-01] jess: 58%|█████▊ | 4609/8000 [04:14<10:13, 5.52it/s, lr: 6.0e-05 loss: 1.616e-01] jess: 58%|█████▊ | 4609/8000 [04:14<10:13, 5.52it/s, lr: 6.0e-05 loss: 8.544e-03] jess: 58%|█████▊ | 4610/8000 [04:14<13:41, 4.13it/s, lr: 6.0e-05 loss: 8.544e-03] jess: 58%|█████▊ | 4610/8000 [04:15<13:41, 4.13it/s, lr: 6.0e-05 loss: 1.212e-01] jess: 58%|█████▊ | 4611/8000 [04:15<14:56, 3.78it/s, lr: 6.0e-05 loss: 1.212e-01] jess: 58%|█████▊ | 4611/8000 [04:15<14:56, 3.78it/s, lr: 6.0e-05 loss: 4.303e-02] jess: 58%|█████▊ | 4612/8000 [04:15<14:57, 3.78it/s, lr: 6.0e-05 loss: 4.303e-02] jess: 58%|█████▊ | 4612/8000 [04:15<14:57, 3.78it/s, lr: 6.0e-05 loss: 1.463e-01] jess: 58%|█████▊ | 4613/8000 [04:15<15:05, 3.74it/s, lr: 6.0e-05 loss: 1.463e-01] jess: 58%|█████▊ | 4613/8000 [04:15<15:05, 3.74it/s, lr: 6.0e-05 loss: 6.640e-02] jess: 58%|█████▊ | 4614/8000 [04:15<10:20, 5.46it/s, lr: 6.0e-05 loss: 6.640e-02] jess: 58%|█████▊ | 4614/8000 [04:16<10:20, 5.46it/s, lr: 6.0e-05 loss: 9.511e-02] jess: 58%|█████▊ | 4615/8000 [04:16<13:48, 4.09it/s, lr: 6.0e-05 loss: 9.511e-02] jess: 58%|█████▊ | 4615/8000 [04:16<13:48, 4.09it/s, lr: 6.0e-05 loss: 6.162e-02] jess: 58%|█████▊ | 4616/8000 [04:16<14:55, 3.78it/s, lr: 6.0e-05 loss: 6.162e-02] jess: 58%|█████▊ | 4616/8000 [04:16<14:55, 3.78it/s, lr: 6.0e-05 loss: 1.268e-01] jess: 58%|█████▊ | 4617/8000 [04:16<10:40, 5.28it/s, lr: 6.0e-05 loss: 1.268e-01] jess: 58%|█████▊ | 4617/8000 [04:16<10:40, 5.28it/s, lr: 6.0e-05 loss: 1.126e-01] jess: 58%|█████▊ | 4618/8000 [04:16<13:44, 4.10it/s, lr: 6.0e-05 loss: 1.126e-01] jess: 58%|█████▊ | 4618/8000 [04:17<13:44, 4.10it/s, lr: 6.0e-05 loss: 7.490e-02] jess: 58%|█████▊ | 4619/8000 [04:17<09:56, 5.67it/s, lr: 6.0e-05 loss: 7.490e-02] jess: 58%|█████▊ | 4619/8000 [04:17<09:56, 5.67it/s, lr: 6.0e-05 loss: 4.141e-02] jess: 58%|█████▊ | 4620/8000 [04:17<13:40, 4.12it/s, lr: 6.0e-05 loss: 4.141e-02] jess: 58%|█████▊ | 4620/8000 [04:17<13:40, 4.12it/s, lr: 6.0e-05 loss: 4.202e-02] jess: 58%|█████▊ | 4621/8000 [04:17<09:59, 5.64it/s, lr: 6.0e-05 loss: 4.202e-02] jess: 58%|█████▊ | 4621/8000 [04:17<09:59, 5.64it/s, lr: 6.0e-05 loss: 7.953e-02] jess: 58%|█████▊ | 4622/8000 [04:17<13:40, 4.12it/s, lr: 6.0e-05 loss: 7.953e-02] jess: 58%|█████▊ | 4622/8000 [04:17<13:40, 4.12it/s, lr: 6.0e-05 loss: 7.246e-02] jess: 58%|█████▊ | 4623/8000 [04:17<14:48, 3.80it/s, lr: 6.0e-05 loss: 7.246e-02] jess: 58%|█████▊ | 4623/8000 [04:18<14:48, 3.80it/s, lr: 6.0e-05 loss: 8.153e-02] jess: 58%|█████▊ | 4624/8000 [04:18<15:09, 3.71it/s, lr: 6.0e-05 loss: 8.153e-02] jess: 58%|█████▊ | 4624/8000 [04:18<15:09, 3.71it/s, lr: 6.0e-05 loss: 9.796e-02] jess: 58%|█████▊ | 4625/8000 [04:18<10:26, 5.39it/s, lr: 6.0e-05 loss: 9.796e-02] jess: 58%|█████▊ | 4625/8000 [04:18<10:26, 5.39it/s, lr: 6.0e-05 loss: 1.526e-02] jess: 58%|█████▊ | 4626/8000 [04:18<23:09, 2.43it/s, lr: 6.0e-05 loss: 1.526e-02] jess: 58%|█████▊ | 4626/8000 [04:19<23:09, 2.43it/s, lr: 6.0e-05 loss: 2.024e-01] jess: 58%|█████▊ | 4627/8000 [04:19<17:46, 3.16it/s, lr: 6.0e-05 loss: 2.024e-01] jess: 58%|█████▊ | 4627/8000 [04:19<17:46, 3.16it/s, lr: 6.0e-05 loss: 1.834e-01] jess: 58%|█████▊ | 4628/8000 [04:19<11:08, 5.04it/s, lr: 6.0e-05 loss: 1.834e-01] jess: 58%|█████▊ | 4628/8000 [04:19<11:08, 5.04it/s, lr: 6.0e-05 loss: 4.624e-02] jess: 58%|█████▊ | 4629/8000 [04:19<09:05, 6.18it/s, lr: 6.0e-05 loss: 4.624e-02] jess: 58%|█████▊ | 4629/8000 [04:19<09:05, 6.18it/s, lr: 6.0e-05 loss: 5.760e-02] jess: 58%|█████▊ | 4630/8000 [04:19<08:28, 6.63it/s, lr: 6.0e-05 loss: 5.760e-02] jess: 58%|█████▊ | 4630/8000 [04:19<08:28, 6.63it/s, lr: 6.0e-05 loss: 1.115e-01] jess: 58%|█████▊ | 4631/8000 [04:19<13:13, 4.24it/s, lr: 6.0e-05 loss: 1.115e-01] jess: 58%|█████▊ | 4631/8000 [04:20<13:13, 4.24it/s, lr: 6.0e-05 loss: 1.241e-01] jess: 58%|█████▊ | 4632/8000 [04:20<14:33, 3.86it/s, lr: 6.0e-05 loss: 1.241e-01] jess: 58%|█████▊ | 4632/8000 [04:20<14:33, 3.86it/s, lr: 6.0e-05 loss: 8.479e-02] jess: 58%|█████▊ | 4633/8000 [04:20<14:59, 3.74it/s, lr: 6.0e-05 loss: 8.479e-02] jess: 58%|█████▊ | 4633/8000 [04:20<14:59, 3.74it/s, lr: 6.0e-05 loss: 1.603e-02] jess: 58%|█████▊ | 4634/8000 [04:20<10:20, 5.43it/s, lr: 6.0e-05 loss: 1.603e-02] jess: 58%|█████▊ | 4634/8000 [04:20<10:20, 5.43it/s, lr: 6.0e-05 loss: 2.366e-01] jess: 58%|█████▊ | 4635/8000 [04:20<13:44, 4.08it/s, lr: 6.0e-05 loss: 2.366e-01] jess: 58%|█████▊ | 4635/8000 [04:21<13:44, 4.08it/s, lr: 6.0e-05 loss: 2.174e-01] jess: 58%|█████▊ | 4636/8000 [04:21<09:52, 5.67it/s, lr: 6.0e-05 loss: 2.174e-01] jess: 58%|█████▊ | 4636/8000 [04:21<09:52, 5.67it/s, lr: 6.0e-05 loss: 8.257e-02] jess: 58%|█████▊ | 4637/8000 [04:21<08:43, 6.42it/s, lr: 6.0e-05 loss: 8.257e-02] jess: 58%|█████▊ | 4637/8000 [04:21<08:43, 6.42it/s, lr: 6.0e-05 loss: 6.108e-02] jess: 58%|█████▊ | 4638/8000 [04:21<13:08, 4.26it/s, lr: 6.0e-05 loss: 6.108e-02] jess: 58%|█████▊ | 4638/8000 [04:21<13:08, 4.26it/s, lr: 6.0e-05 loss: 1.071e-01] jess: 58%|█████▊ | 4639/8000 [04:21<09:44, 5.75it/s, lr: 6.0e-05 loss: 1.071e-01] jess: 58%|█████▊ | 4639/8000 [04:21<09:44, 5.75it/s, lr: 6.0e-05 loss: 1.950e-01] jess: 58%|█████▊ | 4640/8000 [04:21<13:33, 4.13it/s, lr: 6.0e-05 loss: 1.950e-01] jess: 58%|█████▊ | 4640/8000 [04:22<13:33, 4.13it/s, lr: 6.0e-05 loss: 2.225e-01] jess: 58%|█████▊ | 4641/8000 [04:22<14:38, 3.82it/s, lr: 6.0e-05 loss: 2.225e-01] jess: 58%|█████▊ | 4641/8000 [04:22<14:38, 3.82it/s, lr: 6.0e-05 loss: 5.094e-03] jess: 58%|█████▊ | 4642/8000 [04:22<10:08, 5.52it/s, lr: 6.0e-05 loss: 5.094e-03] jess: 58%|█████▊ | 4642/8000 [04:22<10:08, 5.52it/s, lr: 6.0e-05 loss: 1.383e-02] jess: 58%|█████▊ | 4643/8000 [04:22<13:39, 4.10it/s, lr: 6.0e-05 loss: 1.383e-02] jess: 58%|█████▊ | 4643/8000 [04:22<13:39, 4.10it/s, lr: 6.0e-05 loss: 6.503e-02] jess: 58%|█████▊ | 4644/8000 [04:22<14:42, 3.80it/s, lr: 6.0e-05 loss: 6.503e-02] jess: 58%|█████▊ | 4644/8000 [04:23<14:42, 3.80it/s, lr: 6.0e-05 loss: 1.857e-01] jess: 58%|█████▊ | 4645/8000 [04:23<14:56, 3.74it/s, lr: 6.0e-05 loss: 1.857e-01] jess: 58%|█████▊ | 4645/8000 [04:23<14:56, 3.74it/s, lr: 6.0e-05 loss: 1.198e-01] jess: 58%|█████▊ | 4646/8000 [04:23<15:06, 3.70it/s, lr: 6.0e-05 loss: 1.198e-01] jess: 58%|█████▊ | 4646/8000 [04:23<15:06, 3.70it/s, lr: 6.0e-05 loss: 1.851e-01] jess: 58%|█████▊ | 4647/8000 [04:23<10:15, 5.44it/s, lr: 6.0e-05 loss: 1.851e-01] jess: 58%|█████▊ | 4647/8000 [04:23<10:15, 5.44it/s, lr: 6.0e-05 loss: 3.533e-02] jess: 58%|█████▊ | 4648/8000 [04:23<08:50, 6.31it/s, lr: 6.0e-05 loss: 3.533e-02] jess: 58%|█████▊ | 4648/8000 [04:23<08:50, 6.31it/s, lr: 6.0e-05 loss: 1.030e-02] jess: 58%|█████▊ | 4649/8000 [04:23<13:01, 4.29it/s, lr: 6.0e-05 loss: 1.030e-02] jess: 58%|█████▊ | 4649/8000 [04:24<13:01, 4.29it/s, lr: 6.0e-05 loss: 1.805e-01] jess: 58%|█████▊ | 4650/8000 [04:24<20:38, 2.70it/s, lr: 6.0e-05 loss: 1.805e-01] jess: 58%|█████▊ | 4650/8000 [04:24<20:38, 2.70it/s, lr: 6.0e-05 loss: 3.404e-02]
Saving at step 4650
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004650.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
jess: 58%|█████▊ | 4651/8000 [04:24<20:38, 2.70it/s, lr: 6.0e-05 loss: 7.189e-03] jess: 58%|█████▊ | 4652/8000 [04:24<13:19, 4.19it/s, lr: 6.0e-05 loss: 7.189e-03] jess: 58%|█████▊ | 4652/8000 [04:25<13:19, 4.19it/s, lr: 6.0e-05 loss: 1.144e-01] jess: 58%|█████▊ | 4653/8000 [04:25<10:47, 5.17it/s, lr: 6.0e-05 loss: 1.144e-01] jess: 58%|█████▊ | 4653/8000 [04:25<10:47, 5.17it/s, lr: 6.0e-05 loss: 1.637e-01] jess: 58%|█████▊ | 4654/8000 [04:25<09:02, 6.16it/s, lr: 6.0e-05 loss: 1.637e-01] jess: 58%|█████▊ | 4654/8000 [04:25<09:02, 6.16it/s, lr: 6.0e-05 loss: 1.658e-01] jess: 58%|█████▊ | 4655/8000 [04:25<08:28, 6.57it/s, lr: 6.0e-05 loss: 1.658e-01] jess: 58%|█████▊ | 4655/8000 [04:25<08:28, 6.57it/s, lr: 6.0e-05 loss: 5.158e-02] jess: 58%|█████▊ | 4656/8000 [04:25<08:11, 6.80it/s, lr: 6.0e-05 loss: 5.158e-02] jess: 58%|█████▊ | 4656/8000 [04:25<08:11, 6.80it/s, lr: 6.0e-05 loss: 8.107e-03] jess: 58%|█████▊ | 4657/8000 [04:25<08:08, 6.84it/s, lr: 6.0e-05 loss: 8.107e-03] jess: 58%|█████▊ | 4657/8000 [04:25<08:08, 6.84it/s, lr: 6.0e-05 loss: 1.075e-01] jess: 58%|█████▊ | 4658/8000 [04:25<13:02, 4.27it/s, lr: 6.0e-05 loss: 1.075e-01] jess: 58%|█████▊ | 4658/8000 [04:26<13:02, 4.27it/s, lr: 6.0e-05 loss: 6.540e-02] jess: 58%|█████▊ | 4659/8000 [04:26<09:37, 5.78it/s, lr: 6.0e-05 loss: 6.540e-02] jess: 58%|█████▊ | 4659/8000 [04:26<09:37, 5.78it/s, lr: 6.0e-05 loss: 1.025e-01] jess: 58%|█████▊ | 4660/8000 [04:26<08:38, 6.44it/s, lr: 6.0e-05 loss: 1.025e-01] jess: 58%|█████▊ | 4660/8000 [04:26<08:38, 6.44it/s, lr: 6.0e-05 loss: 1.358e-01] jess: 58%|█████▊ | 4661/8000 [04:26<08:23, 6.63it/s, lr: 6.0e-05 loss: 1.358e-01] jess: 58%|█████▊ | 4661/8000 [04:26<08:23, 6.63it/s, lr: 6.0e-05 loss: 1.034e-01] jess: 58%|█████▊ | 4662/8000 [04:26<13:05, 4.25it/s, lr: 6.0e-05 loss: 1.034e-01] jess: 58%|█████▊ | 4662/8000 [04:26<13:05, 4.25it/s, lr: 6.0e-05 loss: 2.042e-01] jess: 58%|█████▊ | 4663/8000 [04:26<14:25, 3.86it/s, lr: 6.0e-05 loss: 2.042e-01] jess: 58%|█████▊ | 4663/8000 [04:27<14:25, 3.86it/s, lr: 6.0e-05 loss: 1.749e-01] jess: 58%|█████▊ | 4664/8000 [04:27<14:54, 3.73it/s, lr: 6.0e-05 loss: 1.749e-01] jess: 58%|█████▊ | 4664/8000 [04:27<14:54, 3.73it/s, lr: 6.0e-05 loss: 7.965e-02] jess: 58%|█████▊ | 4665/8000 [04:27<15:02, 3.70it/s, lr: 6.0e-05 loss: 7.965e-02] jess: 58%|█████▊ | 4665/8000 [04:27<15:02, 3.70it/s, lr: 6.0e-05 loss: 1.431e-01] jess: 58%|█████▊ | 4666/8000 [04:27<15:04, 3.69it/s, lr: 6.0e-05 loss: 1.431e-01] jess: 58%|█████▊ | 4666/8000 [04:27<15:04, 3.69it/s, lr: 6.0e-05 loss: 1.509e-01] jess: 58%|█████▊ | 4667/8000 [04:27<15:05, 3.68it/s, lr: 6.0e-05 loss: 1.509e-01] jess: 58%|█████▊ | 4667/8000 [04:28<15:05, 3.68it/s, lr: 6.0e-05 loss: 4.370e-02] jess: 58%|█████▊ | 4668/8000 [04:28<10:12, 5.44it/s, lr: 6.0e-05 loss: 4.370e-02] jess: 58%|█████▊ | 4668/8000 [04:28<10:12, 5.44it/s, lr: 6.0e-05 loss: 7.076e-02] jess: 58%|█████▊ | 4669/8000 [04:28<08:46, 6.32it/s, lr: 6.0e-05 loss: 7.076e-02] jess: 58%|█████▊ | 4669/8000 [04:28<08:46, 6.32it/s, lr: 6.0e-05 loss: 1.189e-01] jess: 58%|█████▊ | 4670/8000 [04:28<13:11, 4.21it/s, lr: 6.0e-05 loss: 1.189e-01] jess: 58%|█████▊ | 4670/8000 [04:28<13:11, 4.21it/s, lr: 6.0e-05 loss: 5.606e-02] jess: 58%|█████▊ | 4671/8000 [04:28<14:30, 3.82it/s, lr: 6.0e-05 loss: 5.606e-02] jess: 58%|█████▊ | 4671/8000 [04:29<14:30, 3.82it/s, lr: 6.0e-05 loss: 2.192e-01] jess: 58%|█████▊ | 4672/8000 [04:29<14:53, 3.73it/s, lr: 6.0e-05 loss: 2.192e-01] jess: 58%|█████▊ | 4672/8000 [04:29<14:53, 3.73it/s, lr: 6.0e-05 loss: 1.059e-02] jess: 58%|█████▊ | 4673/8000 [04:29<10:11, 5.44it/s, lr: 6.0e-05 loss: 1.059e-02] jess: 58%|█████▊ | 4673/8000 [04:29<10:11, 5.44it/s, lr: 6.0e-05 loss: 3.762e-02] jess: 58%|█████▊ | 4674/8000 [04:29<08:46, 6.31it/s, lr: 6.0e-05 loss: 3.762e-02] jess: 58%|█████▊ | 4674/8000 [04:29<08:46, 6.31it/s, lr: 6.0e-05 loss: 7.765e-02] jess: 58%|█████▊ | 4675/8000 [04:29<13:03, 4.24it/s, lr: 6.0e-05 loss: 7.765e-02] jess: 58%|█████▊ | 4675/8000 [04:30<13:03, 4.24it/s, lr: 6.0e-05 loss: 2.654e-02] jess: 58%|█████▊ | 4676/8000 [04:30<24:29, 2.26it/s, lr: 6.0e-05 loss: 2.654e-02] jess: 58%|█████▊ | 4676/8000 [04:30<24:29, 2.26it/s, lr: 6.0e-05 loss: 1.220e-01] jess: 58%|█████▊ | 4677/8000 [04:30<17:51, 3.10it/s, lr: 6.0e-05 loss: 1.220e-01] jess: 58%|█████▊ | 4677/8000 [04:30<17:51, 3.10it/s, lr: 6.0e-05 loss: 1.693e-02] jess: 58%|█████▊ | 4678/8000 [04:30<15:56, 3.47it/s, lr: 6.0e-05 loss: 1.693e-02] jess: 58%|█████▊ | 4678/8000 [04:30<15:56, 3.47it/s, lr: 6.0e-05 loss: 7.383e-03] jess: 58%|█████▊ | 4679/8000 [04:30<14:14, 3.88it/s, lr: 6.0e-05 loss: 7.383e-03] jess: 58%|█████▊ | 4679/8000 [04:31<14:14, 3.88it/s, lr: 6.0e-05 loss: 8.487e-02] jess: 58%|█████▊ | 4680/8000 [04:31<14:48, 3.74it/s, lr: 6.0e-05 loss: 8.487e-02] jess: 58%|█████▊ | 4680/8000 [04:31<14:48, 3.74it/s, lr: 6.0e-05 loss: 5.789e-02] jess: 59%|█████▊ | 4681/8000 [04:31<10:12, 5.42it/s, lr: 6.0e-05 loss: 5.789e-02] jess: 59%|█████▊ | 4681/8000 [04:31<10:12, 5.42it/s, lr: 6.0e-05 loss: 6.430e-02] jess: 59%|█████▊ | 4682/8000 [04:31<08:42, 6.35it/s, lr: 6.0e-05 loss: 6.430e-02] jess: 59%|█████▊ | 4682/8000 [04:31<08:42, 6.35it/s, lr: 6.0e-05 loss: 7.382e-02] jess: 59%|█████▊ | 4683/8000 [04:31<13:00, 4.25it/s, lr: 6.0e-05 loss: 7.382e-02] jess: 59%|█████▊ | 4683/8000 [04:32<13:00, 4.25it/s, lr: 6.0e-05 loss: 3.829e-02] jess: 59%|█████▊ | 4684/8000 [04:32<14:11, 3.89it/s, lr: 6.0e-05 loss: 3.829e-02] jess: 59%|█████▊ | 4684/8000 [04:32<14:11, 3.89it/s, lr: 6.0e-05 loss: 5.353e-02] jess: 59%|█████▊ | 4685/8000 [04:32<09:56, 5.56it/s, lr: 6.0e-05 loss: 5.353e-02] jess: 59%|█████▊ | 4685/8000 [04:32<09:56, 5.56it/s, lr: 6.0e-05 loss: 9.068e-02] jess: 59%|█████▊ | 4686/8000 [04:32<08:40, 6.37it/s, lr: 6.0e-05 loss: 9.068e-02] jess: 59%|█████▊ | 4686/8000 [04:32<08:40, 6.37it/s, lr: 6.0e-05 loss: 1.003e-01] jess: 59%|█████▊ | 4687/8000 [04:32<13:05, 4.22it/s, lr: 6.0e-05 loss: 1.003e-01] jess: 59%|█████▊ | 4687/8000 [04:32<13:05, 4.22it/s, lr: 6.0e-05 loss: 1.202e-01] jess: 59%|█████▊ | 4688/8000 [04:32<09:38, 5.72it/s, lr: 6.0e-05 loss: 1.202e-01] jess: 59%|█████▊ | 4688/8000 [04:33<09:38, 5.72it/s, lr: 6.0e-05 loss: 2.864e-01] jess: 59%|█████▊ | 4689/8000 [04:33<13:46, 4.01it/s, lr: 6.0e-05 loss: 2.864e-01] jess: 59%|█████▊ | 4689/8000 [04:33<13:46, 4.01it/s, lr: 6.0e-05 loss: 9.314e-02] jess: 59%|█████▊ | 4690/8000 [04:33<09:46, 5.65it/s, lr: 6.0e-05 loss: 9.314e-02] jess: 59%|█████▊ | 4690/8000 [04:33<09:46, 5.65it/s, lr: 6.0e-05 loss: 1.312e-01] jess: 59%|█████▊ | 4691/8000 [04:33<08:49, 6.25it/s, lr: 6.0e-05 loss: 1.312e-01] jess: 59%|█████▊ | 4691/8000 [04:33<08:49, 6.25it/s, lr: 6.0e-05 loss: 9.213e-02] jess: 59%|█████▊ | 4692/8000 [04:33<13:06, 4.21it/s, lr: 6.0e-05 loss: 9.213e-02] jess: 59%|█████▊ | 4692/8000 [04:33<13:06, 4.21it/s, lr: 6.0e-05 loss: 1.447e-01] jess: 59%|█████▊ | 4693/8000 [04:33<09:38, 5.71it/s, lr: 6.0e-05 loss: 1.447e-01] jess: 59%|█████▊ | 4693/8000 [04:34<09:38, 5.71it/s, lr: 6.0e-05 loss: 1.444e-01] jess: 59%|█████▊ | 4694/8000 [04:34<13:22, 4.12it/s, lr: 6.0e-05 loss: 1.444e-01] jess: 59%|█████▊ | 4694/8000 [04:34<13:22, 4.12it/s, lr: 6.0e-05 loss: 9.263e-02] jess: 59%|█████▊ | 4695/8000 [04:34<14:24, 3.82it/s, lr: 6.0e-05 loss: 9.263e-02] jess: 59%|█████▊ | 4695/8000 [04:34<14:24, 3.82it/s, lr: 6.0e-05 loss: 1.303e-01] jess: 59%|█████▊ | 4696/8000 [04:34<09:59, 5.51it/s, lr: 6.0e-05 loss: 1.303e-01] jess: 59%|█████▊ | 4696/8000 [04:34<09:59, 5.51it/s, lr: 6.0e-05 loss: 7.517e-02] jess: 59%|█████▊ | 4697/8000 [04:34<13:28, 4.08it/s, lr: 6.0e-05 loss: 7.517e-02] jess: 59%|█████▊ | 4697/8000 [04:34<13:28, 4.08it/s, lr: 6.0e-05 loss: 1.816e-01] jess: 59%|█████▊ | 4698/8000 [04:34<09:41, 5.68it/s, lr: 6.0e-05 loss: 1.816e-01] jess: 59%|█████▊ | 4698/8000 [04:35<09:41, 5.68it/s, lr: 6.0e-05 loss: 2.819e-03] jess: 59%|█████▊ | 4699/8000 [04:35<08:31, 6.46it/s, lr: 6.0e-05 loss: 2.819e-03] jess: 59%|█████▊ | 4699/8000 [04:35<08:31, 6.46it/s, lr: 6.0e-05 loss: 5.735e-02] jess: 59%|█████▉ | 4700/8000 [04:35<23:07, 2.38it/s, lr: 6.0e-05 loss: 5.735e-02] jess: 59%|█████▉ | 4700/8000 [04:35<23:07, 2.38it/s, lr: 6.0e-05 loss: 1.068e-01]
Saving at step 4700
Saved checkpoint to /app/ai-toolkit/output/jess/jess_000004700.safetensors
Saved optimizer to /app/ai-toolkit/output/jess/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:01<00:02, 1.33s/it]
Generating Images: 67%|██████▋ | 2/3 [00:02<00:01, 1.33s/it]

========================================
Job stopped
========================================

Xet Storage Details

Size:
247 kB
·
Xet hash:
c9d5c394b18e9f11ba4541462bcf237cb7a114f15f1182e01cf243cda4bd063c

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.