Quantumbraid's picture
download
raw
75.1 kB
Running 1 job
{
"type": "diffusion_trainer",
"training_folder": "/app/ai-toolkit/output",
"sqlite_db_path": "/app/ai-toolkit/aitk_db.db",
"device": "cuda",
"trigger_word": null,
"performance_log_every": 10,
"network": {
"type": "lora",
"linear": 32,
"linear_alpha": 32,
"conv": 16,
"conv_alpha": 16,
"lokr_full_rank": true,
"lokr_factor": -1,
"network_kwargs": {
"ignore_if_contains": []
}
},
"save": {
"dtype": "bf16",
"save_every": 20,
"max_step_saves_to_keep": 300,
"save_format": "diffusers",
"push_to_hub": false
},
"datasets": [
{
"folder_path": "/app/ai-toolkit/datasets/amber",
"mask_path": null,
"mask_min_value": 0.1,
"default_caption": "amber",
"caption_ext": "txt",
"caption_dropout_rate": 0.05,
"cache_latents_to_disk": false,
"is_reg": false,
"network_weight": 1,
"resolution": [
512,
256
],
"controls": [],
"shrink_video_to_frames": true,
"num_frames": 1,
"flip_x": false,
"flip_y": false,
"num_repeats": 1
}
],
"train": {
"batch_size": 7,
"bypass_guidance_embedding": false,
"steps": 8000,
"gradient_accumulation": 1,
"train_unet": true,
"train_text_encoder": false,
"gradient_checkpointing": true,
"noise_scheduler": "ddpm",
"optimizer": "adamw8bit",
"timestep_type": "sigmoid",
"content_or_style": "balanced",
"optimizer_params": {
"weight_decay": 0.0001
},
"unload_text_encoder": false,
"cache_text_embeddings": false,
"lr": 4e-05,
"ema_config": {
"use_ema": false,
"ema_decay": 0.99
},
"skip_first_sample": false,
"force_first_sample": false,
"disable_sampling": false,
"dtype": "bf16",
"diff_output_preservation": false,
"diff_output_preservation_multiplier": 1,
"diff_output_preservation_class": "person",
"switch_boundary_every": 1,
"loss_type": "mse"
},
"logging": {
"log_every": 1,
"use_ui_logger": true
},
"model": {
"name_or_path": "stable-diffusion-v1-5/stable-diffusion-v1-5",
"quantize": false,
"qtype": "qfloat8",
"quantize_te": false,
"qtype_te": "qfloat8",
"arch": "sd1",
"low_vram": false,
"model_kwargs": {},
"compile": false
},
"sample": {
"sampler": "ddpm",
"sample_every": 40,
"sample_start_step": 0,
"width": 512,
"height": 512,
"samples": [
{
"prompt": "amber standing nude in the kitchen, morning coffee, bent over counter"
},
{
"prompt": "amber standing nude, in a hot shower, steam"
},
{
"prompt": "amber standing in the street nude, neon reflects off skin, light rain"
}
],
"neg": "",
"seed": 42,
"walk_seed": false,
"guidance_scale": 6,
"sample_steps": 30,
"num_frames": 1,
"fps": 1
}
}
Using SQLite database at /app/ai-toolkit/aitk_db.db
Job ID: "d87a3d89-af47-4335-861a-9f65095ff520"
#############################################
# Running job: amber
#############################################
Running 1 process
Loading pipeline components...: 0%| | 0/6 [00:00<?, ?it/s] Loading pipeline components...: 83%|████████▎ | 5/6 [00:00<00:00, 43.24it/s]
Loading weights: 0%| | 0/196 [00:00<?, ?it/s] Loading weights: 100%|██████████| 196/196 [00:00<00:00, 10910.63it/s]
Loading pipeline components...: 100%|██████████| 6/6 [00:00<00:00, 37.39it/s]
create LoRA network. base dim (rank): 32, alpha: 32
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
apply LoRA to Conv2d with kernel size (3,3). dim (rank): 16, alpha: 16
create LoRA for Text Encoder: 0 modules.
create LoRA for U-Net: 278 modules.
enable LoRA for U-Net
#### IMPORTANT RESUMING FROM /app/ai-toolkit/output/amber/amber_000001850.safetensors ####
Loading from /app/ai-toolkit/output/amber/amber_000001850.safetensors
Missing keys: []
Found step 1850 in metadata, starting from there
Loading optimizer state from /app/ai-toolkit/output/amber/optimizer.pt
Updating optimizer LR from params
Dataset: /app/ai-toolkit/datasets/amber
- Preprocessing image dimensions
0%| | 0/109 [00:00<?, ?it/s] 100%|██████████| 109/109 [00:00<00:00, 58269.07it/s]
- Found 109 images
Bucket sizes for /app/ai-toolkit/datasets/amber:
416x624: 80 files
512x512: 7 files
672x384: 6 files
624x416: 9 files
240x256: 1 files
448x448: 1 files
640x400: 1 files
576x448: 3 files
528x496: 1 files
9 buckets made
Dataset: /app/ai-toolkit/datasets/amber
- Preprocessing image dimensions
0%| | 0/109 [00:00<?, ?it/s] 100%|██████████| 109/109 [00:00<00:00, 89205.69it/s]
- Found 109 images
Bucket sizes for /app/ai-toolkit/datasets/amber:
208x304: 80 files
256x256: 9 files
336x192: 7 files
304x208: 9 files
240x256: 1 files
288x224: 3 files
6 buckets made
amber: 23%|██▎ | 1850/8000 [00:00<?, ?it/s] amber: 23%|██▎ | 1850/8000 [00:00<?, ?it/s, lr: 4.0e-05 loss: 3.199e-03] amber: 23%|██▎ | 1851/8000 [00:00<1:28:47, 1.15it/s, lr: 4.0e-05 loss: 3.199e-03] amber: 23%|██▎ | 1851/8000 [00:01<1:28:47, 1.15it/s, lr: 4.0e-05 loss: 4.877e-03] amber: 23%|██▎ | 1852/8000 [00:01<34:02, 3.01it/s, lr: 4.0e-05 loss: 4.877e-03] amber: 23%|██▎ | 1852/8000 [00:01<34:02, 3.01it/s, lr: 4.0e-05 loss: 2.519e-01] amber: 23%|██▎ | 1853/8000 [00:01<38:43, 2.65it/s, lr: 4.0e-05 loss: 2.519e-01] amber: 23%|██▎ | 1853/8000 [00:01<38:43, 2.65it/s, lr: 4.0e-05 loss: 1.285e-01] amber: 23%|██▎ | 1854/8000 [00:01<30:15, 3.39it/s, lr: 4.0e-05 loss: 1.285e-01] amber: 23%|██▎ | 1854/8000 [00:01<30:15, 3.39it/s, lr: 4.0e-05 loss: 5.380e-02] amber: 23%|██▎ | 1855/8000 [00:01<26:24, 3.88it/s, lr: 4.0e-05 loss: 5.380e-02] amber: 23%|██▎ | 1855/8000 [00:02<26:24, 3.88it/s, lr: 4.0e-05 loss: 1.692e-01] amber: 23%|██▎ | 1856/8000 [00:02<24:36, 4.16it/s, lr: 4.0e-05 loss: 1.692e-01] amber: 23%|██▎ | 1856/8000 [00:02<24:36, 4.16it/s, lr: 4.0e-05 loss: 7.423e-02] amber: 23%|██▎ | 1857/8000 [00:02<31:42, 3.23it/s, lr: 4.0e-05 loss: 7.423e-02] amber: 23%|██▎ | 1857/8000 [00:02<31:42, 3.23it/s, lr: 4.0e-05 loss: 1.378e-01] amber: 23%|██▎ | 1858/8000 [00:02<36:54, 2.77it/s, lr: 4.0e-05 loss: 1.378e-01] amber: 23%|██▎ | 1858/8000 [00:03<36:54, 2.77it/s, lr: 4.0e-05 loss: 6.093e-02] amber: 23%|██▎ | 1859/8000 [00:03<36:47, 2.78it/s, lr: 4.0e-05 loss: 6.093e-02] amber: 23%|██▎ | 1859/8000 [00:03<36:47, 2.78it/s, lr: 4.0e-05 loss: 1.005e-01] amber: 23%|██▎ | 1860/8000 [00:03<32:10, 3.18it/s, lr: 4.0e-05 loss: 1.005e-01] amber: 23%|██▎ | 1860/8000 [00:03<32:10, 3.18it/s, lr: 4.0e-05 loss: 1.869e-02]
Saving at step 1860
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000001860.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 23%|██▎ | 1861/8000 [00:04<32:10, 3.18it/s, lr: 4.0e-05 loss: 3.974e-02] amber: 23%|██▎ | 1862/8000 [00:04<27:11, 3.76it/s, lr: 4.0e-05 loss: 3.974e-02] amber: 23%|██▎ | 1862/8000 [00:04<27:11, 3.76it/s, lr: 4.0e-05 loss: 6.398e-02] amber: 23%|██▎ | 1863/8000 [00:04<33:23, 3.06it/s, lr: 4.0e-05 loss: 6.398e-02] amber: 23%|██▎ | 1863/8000 [00:05<33:23, 3.06it/s, lr: 4.0e-05 loss: 1.035e-01] amber: 23%|██▎ | 1864/8000 [00:05<34:14, 2.99it/s, lr: 4.0e-05 loss: 1.035e-01] amber: 23%|██▎ | 1864/8000 [00:05<34:14, 2.99it/s, lr: 4.0e-05 loss: 1.648e-01] amber: 23%|██▎ | 1865/8000 [00:05<37:12, 2.75it/s, lr: 4.0e-05 loss: 1.648e-01] amber: 23%|██▎ | 1865/8000 [00:05<37:12, 2.75it/s, lr: 4.0e-05 loss: 1.645e-01] amber: 23%|██▎ | 1866/8000 [00:05<26:27, 3.86it/s, lr: 4.0e-05 loss: 1.645e-01] amber: 23%|██▎ | 1866/8000 [00:06<26:27, 3.86it/s, lr: 4.0e-05 loss: 9.131e-02] amber: 23%|██▎ | 1867/8000 [00:06<34:24, 2.97it/s, lr: 4.0e-05 loss: 9.131e-02] amber: 23%|██▎ | 1867/8000 [00:06<34:24, 2.97it/s, lr: 4.0e-05 loss: 1.490e-01] amber: 23%|██▎ | 1868/8000 [00:06<27:51, 3.67it/s, lr: 4.0e-05 loss: 1.490e-01] amber: 23%|██▎ | 1868/8000 [00:06<27:51, 3.67it/s, lr: 4.0e-05 loss: 1.535e-01] amber: 23%|██▎ | 1869/8000 [00:06<23:43, 4.31it/s, lr: 4.0e-05 loss: 1.535e-01] amber: 23%|██▎ | 1869/8000 [00:06<23:43, 4.31it/s, lr: 4.0e-05 loss: 2.011e-01] amber: 23%|██▎ | 1870/8000 [00:06<32:23, 3.15it/s, lr: 4.0e-05 loss: 2.011e-01] amber: 23%|██▎ | 1870/8000 [00:07<32:23, 3.15it/s, lr: 4.0e-05 loss: 1.399e-01] amber: 23%|██▎ | 1871/8000 [00:07<36:08, 2.83it/s, lr: 4.0e-05 loss: 1.399e-01] amber: 23%|██▎ | 1871/8000 [00:07<36:08, 2.83it/s, lr: 4.0e-05 loss: 8.062e-02] amber: 23%|██▎ | 1872/8000 [00:07<23:53, 4.28it/s, lr: 4.0e-05 loss: 8.062e-02] amber: 23%|██▎ | 1872/8000 [00:07<23:53, 4.28it/s, lr: 4.0e-05 loss: 1.529e-01] amber: 23%|██▎ | 1873/8000 [00:07<21:10, 4.82it/s, lr: 4.0e-05 loss: 1.529e-01] amber: 23%|██▎ | 1873/8000 [00:07<21:10, 4.82it/s, lr: 4.0e-05 loss: 1.870e-02] amber: 23%|██▎ | 1874/8000 [00:08<32:26, 3.15it/s, lr: 4.0e-05 loss: 1.870e-02] amber: 23%|██▎ | 1874/8000 [00:08<32:26, 3.15it/s, lr: 4.0e-05 loss: 1.232e-01] amber: 23%|██▎ | 1875/8000 [00:08<25:47, 3.96it/s, lr: 4.0e-05 loss: 1.232e-01] amber: 23%|██▎ | 1875/8000 [00:08<25:47, 3.96it/s, lr: 4.0e-05 loss: 1.125e-01] amber: 23%|██▎ | 1876/8000 [00:08<34:27, 2.96it/s, lr: 4.0e-05 loss: 1.125e-01] amber: 23%|██▎ | 1876/8000 [00:08<34:27, 2.96it/s, lr: 4.0e-05 loss: 1.536e-01] amber: 23%|██▎ | 1877/8000 [00:08<26:01, 3.92it/s, lr: 4.0e-05 loss: 1.536e-01] amber: 23%|██▎ | 1877/8000 [00:09<26:01, 3.92it/s, lr: 4.0e-05 loss: 1.437e-01] amber: 23%|██▎ | 1878/8000 [00:09<23:06, 4.41it/s, lr: 4.0e-05 loss: 1.437e-01] amber: 23%|██▎ | 1878/8000 [00:09<23:06, 4.41it/s, lr: 4.0e-05 loss: 8.771e-02] amber: 23%|██▎ | 1879/8000 [00:09<22:50, 4.47it/s, lr: 4.0e-05 loss: 8.771e-02] amber: 23%|██▎ | 1879/8000 [00:09<22:50, 4.47it/s, lr: 4.0e-05 loss: 6.495e-02] amber: 24%|██▎ | 1880/8000 [00:09<33:38, 3.03it/s, lr: 4.0e-05 loss: 6.495e-02] amber: 24%|██▎ | 1880/8000 [00:09<33:38, 3.03it/s, lr: 4.0e-05 loss: 8.532e-02]
Saving at step 1880
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000001880.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:00<00:01, 1.21it/s]
Generating Images: 67%|██████▋ | 2/3 [00:01<00:00, 1.25it/s]
Generating Images: 100%|██████████| 3/3 [00:02<00:00, 1.24it/s]

amber: 24%|██▎ | 1881/8000 [00:10<33:37, 3.03it/s, lr: 4.0e-05 loss: 7.968e-02] amber: 24%|██▎ | 1882/8000 [00:10<30:50, 3.31it/s, lr: 4.0e-05 loss: 7.968e-02] amber: 24%|██▎ | 1882/8000 [00:10<30:50, 3.31it/s, lr: 4.0e-05 loss: 1.340e-01] amber: 24%|██▎ | 1883/8000 [00:10<27:14, 3.74it/s, lr: 4.0e-05 loss: 1.340e-01] amber: 24%|██▎ | 1883/8000 [00:11<27:14, 3.74it/s, lr: 4.0e-05 loss: 6.611e-02] amber: 24%|██▎ | 1884/8000 [00:11<22:11, 4.59it/s, lr: 4.0e-05 loss: 6.611e-02] amber: 24%|██▎ | 1884/8000 [00:11<22:11, 4.59it/s, lr: 4.0e-05 loss: 8.834e-02] amber: 24%|██▎ | 1885/8000 [00:11<31:50, 3.20it/s, lr: 4.0e-05 loss: 8.834e-02] amber: 24%|██▎ | 1885/8000 [00:11<31:50, 3.20it/s, lr: 4.0e-05 loss: 1.801e-01] amber: 24%|██▎ | 1886/8000 [00:11<34:44, 2.93it/s, lr: 4.0e-05 loss: 1.801e-01] amber: 24%|██▎ | 1886/8000 [00:12<34:44, 2.93it/s, lr: 4.0e-05 loss: 1.796e-01] amber: 24%|██▎ | 1887/8000 [00:12<35:06, 2.90it/s, lr: 4.0e-05 loss: 1.796e-01] amber: 24%|██▎ | 1887/8000 [00:12<35:06, 2.90it/s, lr: 4.0e-05 loss: 1.807e-01] amber: 24%|██▎ | 1888/8000 [00:12<30:25, 3.35it/s, lr: 4.0e-05 loss: 1.807e-01] amber: 24%|██▎ | 1888/8000 [00:12<30:25, 3.35it/s, lr: 4.0e-05 loss: 1.106e-01] amber: 24%|██▎ | 1889/8000 [00:12<20:32, 4.96it/s, lr: 4.0e-05 loss: 1.106e-01] amber: 24%|██▎ | 1889/8000 [00:12<20:32, 4.96it/s, lr: 4.0e-05 loss: 9.994e-02] amber: 24%|██▎ | 1890/8000 [00:12<31:00, 3.28it/s, lr: 4.0e-05 loss: 9.994e-02] amber: 24%|██▎ | 1890/8000 [00:13<31:00, 3.28it/s, lr: 4.0e-05 loss: 1.304e-01] amber: 24%|██▎ | 1891/8000 [00:13<47:52, 2.13it/s, lr: 4.0e-05 loss: 1.304e-01] amber: 24%|██▎ | 1891/8000 [00:13<47:52, 2.13it/s, lr: 4.0e-05 loss: 1.591e-01] amber: 24%|██▎ | 1892/8000 [00:13<31:55, 3.19it/s, lr: 4.0e-05 loss: 1.591e-01] amber: 24%|██▎ | 1892/8000 [00:13<31:55, 3.19it/s, lr: 4.0e-05 loss: 2.093e-01] amber: 24%|██▎ | 1893/8000 [00:13<34:17, 2.97it/s, lr: 4.0e-05 loss: 2.093e-01] amber: 24%|██▎ | 1893/8000 [00:14<34:17, 2.97it/s, lr: 4.0e-05 loss: 6.410e-02] amber: 24%|██▎ | 1894/8000 [00:14<26:26, 3.85it/s, lr: 4.0e-05 loss: 6.410e-02] amber: 24%|██▎ | 1894/8000 [00:14<26:26, 3.85it/s, lr: 4.0e-05 loss: 1.678e-01] amber: 24%|██▎ | 1895/8000 [00:14<32:11, 3.16it/s, lr: 4.0e-05 loss: 1.678e-01] amber: 24%|██▎ | 1895/8000 [00:14<32:11, 3.16it/s, lr: 4.0e-05 loss: 1.718e-01] amber: 24%|██▎ | 1896/8000 [00:14<26:03, 3.90it/s, lr: 4.0e-05 loss: 1.718e-01] amber: 24%|██▎ | 1896/8000 [00:15<26:03, 3.90it/s, lr: 4.0e-05 loss: 1.311e-01] amber: 24%|██▎ | 1897/8000 [00:15<23:42, 4.29it/s, lr: 4.0e-05 loss: 1.311e-01] amber: 24%|██▎ | 1897/8000 [00:15<23:42, 4.29it/s, lr: 4.0e-05 loss: 3.651e-02] amber: 24%|██▎ | 1898/8000 [00:15<22:46, 4.47it/s, lr: 4.0e-05 loss: 3.651e-02] amber: 24%|██▎ | 1898/8000 [00:15<22:46, 4.47it/s, lr: 4.0e-05 loss: 1.090e-01] amber: 24%|██▎ | 1899/8000 [00:15<20:36, 4.93it/s, lr: 4.0e-05 loss: 1.090e-01] amber: 24%|██▎ | 1899/8000 [00:15<20:36, 4.93it/s, lr: 4.0e-05 loss: 9.221e-02] amber: 24%|██▍ | 1900/8000 [00:15<20:47, 4.89it/s, lr: 4.0e-05 loss: 9.221e-02] amber: 24%|██▍ | 1900/8000 [00:15<20:47, 4.89it/s, lr: 4.0e-05 loss: 1.355e-01]
Saving at step 1900
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000001900.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 24%|██▍ | 1901/8000 [00:16<20:47, 4.89it/s, lr: 4.0e-05 loss: 1.249e-01] amber: 24%|██▍ | 1902/8000 [00:16<30:09, 3.37it/s, lr: 4.0e-05 loss: 1.249e-01] amber: 24%|██▍ | 1902/8000 [00:16<30:09, 3.37it/s, lr: 4.0e-05 loss: 9.817e-02] amber: 24%|██▍ | 1903/8000 [00:16<34:18, 2.96it/s, lr: 4.0e-05 loss: 9.817e-02] amber: 24%|██▍ | 1903/8000 [00:17<34:18, 2.96it/s, lr: 4.0e-05 loss: 2.104e-01] amber: 24%|██▍ | 1904/8000 [00:17<34:45, 2.92it/s, lr: 4.0e-05 loss: 2.104e-01] amber: 24%|██▍ | 1904/8000 [00:17<34:45, 2.92it/s, lr: 4.0e-05 loss: 5.209e-03] amber: 24%|██▍ | 1905/8000 [00:17<26:32, 3.83it/s, lr: 4.0e-05 loss: 5.209e-03] amber: 24%|██▍ | 1905/8000 [00:17<26:32, 3.83it/s, lr: 4.0e-05 loss: 1.928e-01] amber: 24%|██▍ | 1906/8000 [00:17<33:17, 3.05it/s, lr: 4.0e-05 loss: 1.928e-01] amber: 24%|██▍ | 1906/8000 [00:18<33:17, 3.05it/s, lr: 4.0e-05 loss: 8.253e-02] amber: 24%|██▍ | 1907/8000 [00:18<36:37, 2.77it/s, lr: 4.0e-05 loss: 8.253e-02] amber: 24%|██▍ | 1907/8000 [00:18<36:37, 2.77it/s, lr: 4.0e-05 loss: 1.456e-01] amber: 24%|██▍ | 1908/8000 [00:18<25:18, 4.01it/s, lr: 4.0e-05 loss: 1.456e-01] amber: 24%|██▍ | 1908/8000 [00:18<25:18, 4.01it/s, lr: 4.0e-05 loss: 1.212e-02] amber: 24%|██▍ | 1909/8000 [00:18<21:20, 4.76it/s, lr: 4.0e-05 loss: 1.212e-02] amber: 24%|██▍ | 1909/8000 [00:19<21:20, 4.76it/s, lr: 4.0e-05 loss: 2.005e-01] amber: 24%|██▍ | 1910/8000 [00:19<30:35, 3.32it/s, lr: 4.0e-05 loss: 2.005e-01] amber: 24%|██▍ | 1910/8000 [00:19<30:35, 3.32it/s, lr: 4.0e-05 loss: 9.541e-02] amber: 24%|██▍ | 1911/8000 [00:19<24:17, 4.18it/s, lr: 4.0e-05 loss: 9.541e-02] amber: 24%|██▍ | 1911/8000 [00:19<24:17, 4.18it/s, lr: 4.0e-05 loss: 2.182e-01] amber: 24%|██▍ | 1912/8000 [00:19<26:36, 3.81it/s, lr: 4.0e-05 loss: 2.182e-01] amber: 24%|██▍ | 1912/8000 [00:19<26:36, 3.81it/s, lr: 4.0e-05 loss: 2.178e-01] amber: 24%|██▍ | 1913/8000 [00:19<21:13, 4.78it/s, lr: 4.0e-05 loss: 2.178e-01] amber: 24%|██▍ | 1913/8000 [00:19<21:13, 4.78it/s, lr: 4.0e-05 loss: 2.215e-01] amber: 24%|██▍ | 1914/8000 [00:19<20:39, 4.91it/s, lr: 4.0e-05 loss: 2.215e-01] amber: 24%|██▍ | 1914/8000 [00:20<20:39, 4.91it/s, lr: 4.0e-05 loss: 4.587e-02] amber: 24%|██▍ | 1915/8000 [00:20<30:20, 3.34it/s, lr: 4.0e-05 loss: 4.587e-02] amber: 24%|██▍ | 1915/8000 [00:20<30:20, 3.34it/s, lr: 4.0e-05 loss: 1.064e-01] amber: 24%|██▍ | 1916/8000 [00:20<24:47, 4.09it/s, lr: 4.0e-05 loss: 1.064e-01] amber: 24%|██▍ | 1916/8000 [00:20<24:47, 4.09it/s, lr: 4.0e-05 loss: 9.203e-02] amber: 24%|██▍ | 1917/8000 [00:20<32:58, 3.08it/s, lr: 4.0e-05 loss: 9.203e-02] amber: 24%|██▍ | 1917/8000 [00:21<32:58, 3.08it/s, lr: 4.0e-05 loss: 7.985e-03] amber: 24%|██▍ | 1918/8000 [00:21<24:40, 4.11it/s, lr: 4.0e-05 loss: 7.985e-03] amber: 24%|██▍ | 1918/8000 [00:21<24:40, 4.11it/s, lr: 4.0e-05 loss: 1.275e-01] amber: 24%|██▍ | 1919/8000 [00:21<23:09, 4.37it/s, lr: 4.0e-05 loss: 1.275e-01] amber: 24%|██▍ | 1919/8000 [00:21<23:09, 4.37it/s, lr: 4.0e-05 loss: 2.258e-01] amber: 24%|██▍ | 1920/8000 [00:21<32:43, 3.10it/s, lr: 4.0e-05 loss: 2.258e-01] amber: 24%|██▍ | 1920/8000 [00:21<32:43, 3.10it/s, lr: 4.0e-05 loss: 1.876e-01]
Saving at step 1920
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000001920.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:00<00:01, 1.15it/s]
Generating Images: 67%|██████▋ | 2/3 [00:01<00:00, 1.11it/s]
Generating Images: 100%|██████████| 3/3 [00:02<00:00, 1.08it/s]

amber: 24%|██▍ | 1921/8000 [00:22<32:43, 3.10it/s, lr: 4.0e-05 loss: 1.460e-01] amber: 24%|██▍ | 1922/8000 [00:22<38:40, 2.62it/s, lr: 4.0e-05 loss: 1.460e-01] amber: 24%|██▍ | 1922/8000 [00:22<38:40, 2.62it/s, lr: 4.0e-05 loss: 9.958e-02] amber: 24%|██▍ | 1923/8000 [00:22<29:23, 3.45it/s, lr: 4.0e-05 loss: 9.958e-02] amber: 24%|██▍ | 1923/8000 [00:23<29:23, 3.45it/s, lr: 4.0e-05 loss: 6.142e-02] amber: 24%|██▍ | 1924/8000 [00:23<25:33, 3.96it/s, lr: 4.0e-05 loss: 6.142e-02] amber: 24%|██▍ | 1924/8000 [00:23<25:33, 3.96it/s, lr: 4.0e-05 loss: 1.783e-01] amber: 24%|██▍ | 1925/8000 [00:23<31:51, 3.18it/s, lr: 4.0e-05 loss: 1.783e-01] amber: 24%|██▍ | 1925/8000 [00:23<31:51, 3.18it/s, lr: 4.0e-05 loss: 1.265e-01] amber: 24%|██▍ | 1926/8000 [00:23<36:05, 2.81it/s, lr: 4.0e-05 loss: 1.265e-01] amber: 24%|██▍ | 1926/8000 [00:24<36:05, 2.81it/s, lr: 4.0e-05 loss: 2.247e-02] amber: 24%|██▍ | 1927/8000 [00:24<35:12, 2.87it/s, lr: 4.0e-05 loss: 2.247e-02] amber: 24%|██▍ | 1927/8000 [00:24<35:12, 2.87it/s, lr: 4.0e-05 loss: 3.064e-02] amber: 24%|██▍ | 1928/8000 [00:24<30:29, 3.32it/s, lr: 4.0e-05 loss: 3.064e-02] amber: 24%|██▍ | 1928/8000 [00:24<30:29, 3.32it/s, lr: 4.0e-05 loss: 1.123e-01] amber: 24%|██▍ | 1929/8000 [00:24<35:30, 2.85it/s, lr: 4.0e-05 loss: 1.123e-01] amber: 24%|██▍ | 1929/8000 [00:25<35:30, 2.85it/s, lr: 4.0e-05 loss: 1.011e-01] amber: 24%|██▍ | 1930/8000 [00:25<34:19, 2.95it/s, lr: 4.0e-05 loss: 1.011e-01] amber: 24%|██▍ | 1930/8000 [00:25<34:19, 2.95it/s, lr: 4.0e-05 loss: 1.790e-01] amber: 24%|██▍ | 1931/8000 [00:25<48:01, 2.11it/s, lr: 4.0e-05 loss: 1.790e-01] amber: 24%|██▍ | 1931/8000 [00:26<48:01, 2.11it/s, lr: 4.0e-05 loss: 5.296e-02] amber: 24%|██▍ | 1932/8000 [00:26<30:11, 3.35it/s, lr: 4.0e-05 loss: 5.296e-02] amber: 24%|██▍ | 1932/8000 [00:26<30:11, 3.35it/s, lr: 4.0e-05 loss: 9.351e-02] amber: 24%|██▍ | 1933/8000 [00:26<33:26, 3.02it/s, lr: 4.0e-05 loss: 9.351e-02] amber: 24%|██▍ | 1933/8000 [00:26<33:26, 3.02it/s, lr: 4.0e-05 loss: 5.321e-02] amber: 24%|██▍ | 1934/8000 [00:26<25:23, 3.98it/s, lr: 4.0e-05 loss: 5.321e-02] amber: 24%|██▍ | 1934/8000 [00:26<25:23, 3.98it/s, lr: 4.0e-05 loss: 7.450e-02] amber: 24%|██▍ | 1935/8000 [00:26<33:49, 2.99it/s, lr: 4.0e-05 loss: 7.450e-02] amber: 24%|██▍ | 1935/8000 [00:27<33:49, 2.99it/s, lr: 4.0e-05 loss: 9.584e-02] amber: 24%|██▍ | 1936/8000 [00:27<34:26, 2.93it/s, lr: 4.0e-05 loss: 9.584e-02] amber: 24%|██▍ | 1936/8000 [00:27<34:26, 2.93it/s, lr: 4.0e-05 loss: 9.839e-02] amber: 24%|██▍ | 1937/8000 [00:27<36:03, 2.80it/s, lr: 4.0e-05 loss: 9.839e-02] amber: 24%|██▍ | 1937/8000 [00:27<36:03, 2.80it/s, lr: 4.0e-05 loss: 8.440e-03] amber: 24%|██▍ | 1938/8000 [00:27<33:33, 3.01it/s, lr: 4.0e-05 loss: 8.440e-03] amber: 24%|██▍ | 1938/8000 [00:28<33:33, 3.01it/s, lr: 4.0e-05 loss: 8.154e-02] amber: 24%|██▍ | 1939/8000 [00:28<26:35, 3.80it/s, lr: 4.0e-05 loss: 8.154e-02] amber: 24%|██▍ | 1939/8000 [00:28<26:35, 3.80it/s, lr: 4.0e-05 loss: 3.983e-02] amber: 24%|██▍ | 1940/8000 [00:28<32:48, 3.08it/s, lr: 4.0e-05 loss: 3.983e-02] amber: 24%|██▍ | 1940/8000 [00:28<32:48, 3.08it/s, lr: 4.0e-05 loss: 1.100e-01]
Saving at step 1940
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000001940.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 24%|██▍ | 1941/8000 [00:29<32:47, 3.08it/s, lr: 4.0e-05 loss: 1.394e-01] amber: 24%|██▍ | 1942/8000 [00:29<37:26, 2.70it/s, lr: 4.0e-05 loss: 1.394e-01] amber: 24%|██▍ | 1942/8000 [00:30<37:26, 2.70it/s, lr: 4.0e-05 loss: 5.106e-02] amber: 24%|██▍ | 1943/8000 [00:30<35:35, 2.84it/s, lr: 4.0e-05 loss: 5.106e-02] amber: 24%|██▍ | 1943/8000 [00:30<35:35, 2.84it/s, lr: 4.0e-05 loss: 2.320e-03] amber: 24%|██▍ | 1944/8000 [00:30<27:11, 3.71it/s, lr: 4.0e-05 loss: 2.320e-03] amber: 24%|██▍ | 1944/8000 [00:30<27:11, 3.71it/s, lr: 4.0e-05 loss: 5.541e-02] amber: 24%|██▍ | 1945/8000 [00:30<22:49, 4.42it/s, lr: 4.0e-05 loss: 5.541e-02] amber: 24%|██▍ | 1945/8000 [00:30<22:49, 4.42it/s, lr: 4.0e-05 loss: 1.863e-01] amber: 24%|██▍ | 1946/8000 [00:30<21:55, 4.60it/s, lr: 4.0e-05 loss: 1.863e-01] amber: 24%|██▍ | 1946/8000 [00:31<21:55, 4.60it/s, lr: 4.0e-05 loss: 7.328e-02] amber: 24%|██▍ | 1947/8000 [00:31<31:00, 3.25it/s, lr: 4.0e-05 loss: 7.328e-02] amber: 24%|██▍ | 1947/8000 [00:31<31:00, 3.25it/s, lr: 4.0e-05 loss: 1.553e-01] amber: 24%|██▍ | 1948/8000 [00:31<25:31, 3.95it/s, lr: 4.0e-05 loss: 1.553e-01] amber: 24%|██▍ | 1948/8000 [00:31<25:31, 3.95it/s, lr: 4.0e-05 loss: 6.427e-02] amber: 24%|██▍ | 1949/8000 [00:31<32:16, 3.12it/s, lr: 4.0e-05 loss: 6.427e-02] amber: 24%|██▍ | 1949/8000 [00:31<32:16, 3.12it/s, lr: 4.0e-05 loss: 1.657e-01] amber: 24%|██▍ | 1950/8000 [00:31<27:20, 3.69it/s, lr: 4.0e-05 loss: 1.657e-01] amber: 24%|██▍ | 1950/8000 [00:32<27:20, 3.69it/s, lr: 4.0e-05 loss: 1.704e-01] amber: 24%|██▍ | 1951/8000 [00:32<23:34, 4.28it/s, lr: 4.0e-05 loss: 1.704e-01] amber: 24%|██▍ | 1951/8000 [00:32<23:34, 4.28it/s, lr: 4.0e-05 loss: 7.785e-02] amber: 24%|██▍ | 1952/8000 [00:32<22:28, 4.48it/s, lr: 4.0e-05 loss: 7.785e-02] amber: 24%|██▍ | 1952/8000 [00:32<22:28, 4.48it/s, lr: 4.0e-05 loss: 1.957e-01] amber: 24%|██▍ | 1953/8000 [00:32<30:45, 3.28it/s, lr: 4.0e-05 loss: 1.957e-01] amber: 24%|██▍ | 1953/8000 [00:32<30:45, 3.28it/s, lr: 4.0e-05 loss: 1.632e-01] amber: 24%|██▍ | 1954/8000 [00:32<35:03, 2.87it/s, lr: 4.0e-05 loss: 1.632e-01] amber: 24%|██▍ | 1954/8000 [00:33<35:03, 2.87it/s, lr: 4.0e-05 loss: 1.054e-01] amber: 24%|██▍ | 1955/8000 [00:33<34:27, 2.92it/s, lr: 4.0e-05 loss: 1.054e-01] amber: 24%|██▍ | 1955/8000 [00:33<34:27, 2.92it/s, lr: 4.0e-05 loss: 1.404e-01] amber: 24%|██▍ | 1956/8000 [00:33<36:11, 2.78it/s, lr: 4.0e-05 loss: 1.404e-01] amber: 24%|██▍ | 1956/8000 [00:33<36:11, 2.78it/s, lr: 4.0e-05 loss: 3.551e-02] amber: 24%|██▍ | 1957/8000 [00:33<28:13, 3.57it/s, lr: 4.0e-05 loss: 3.551e-02] amber: 24%|██▍ | 1957/8000 [00:34<28:13, 3.57it/s, lr: 4.0e-05 loss: 6.882e-02] amber: 24%|██▍ | 1958/8000 [00:34<24:15, 4.15it/s, lr: 4.0e-05 loss: 6.882e-02] amber: 24%|██▍ | 1958/8000 [00:34<24:15, 4.15it/s, lr: 4.0e-05 loss: 8.941e-02] amber: 24%|██▍ | 1959/8000 [00:34<21:17, 4.73it/s, lr: 4.0e-05 loss: 8.941e-02] amber: 24%|██▍ | 1959/8000 [00:34<21:17, 4.73it/s, lr: 4.0e-05 loss: 8.968e-02] amber: 24%|██▍ | 1960/8000 [00:34<30:41, 3.28it/s, lr: 4.0e-05 loss: 8.968e-02] amber: 24%|██▍ | 1960/8000 [00:34<30:41, 3.28it/s, lr: 4.0e-05 loss: 7.167e-02]
Saving at step 1960
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000001960.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:00<00:01, 1.08it/s]
Generating Images: 67%|██████▋ | 2/3 [00:01<00:00, 1.05it/s]
Generating Images: 100%|██████████| 3/3 [00:02<00:00, 1.09it/s]

amber: 25%|██▍ | 1961/8000 [00:35<30:40, 3.28it/s, lr: 4.0e-05 loss: 1.628e-01] amber: 25%|██▍ | 1962/8000 [00:35<32:07, 3.13it/s, lr: 4.0e-05 loss: 1.628e-01] amber: 25%|██▍ | 1962/8000 [00:35<32:07, 3.13it/s, lr: 4.0e-05 loss: 1.357e-01] amber: 25%|██▍ | 1963/8000 [00:35<36:16, 2.77it/s, lr: 4.0e-05 loss: 1.357e-01] amber: 25%|██▍ | 1963/8000 [00:36<36:16, 2.77it/s, lr: 4.0e-05 loss: 8.914e-02] amber: 25%|██▍ | 1964/8000 [00:36<25:15, 3.98it/s, lr: 4.0e-05 loss: 8.914e-02] amber: 25%|██▍ | 1964/8000 [00:36<25:15, 3.98it/s, lr: 4.0e-05 loss: 4.864e-03] amber: 25%|██▍ | 1965/8000 [00:36<20:49, 4.83it/s, lr: 4.0e-05 loss: 4.864e-03] amber: 25%|██▍ | 1965/8000 [00:36<20:49, 4.83it/s, lr: 4.0e-05 loss: 1.341e-01] amber: 25%|██▍ | 1966/8000 [00:36<30:07, 3.34it/s, lr: 4.0e-05 loss: 1.341e-01] amber: 25%|██▍ | 1966/8000 [00:36<30:07, 3.34it/s, lr: 4.0e-05 loss: 1.928e-01] amber: 25%|██▍ | 1967/8000 [00:36<23:33, 4.27it/s, lr: 4.0e-05 loss: 1.928e-01] amber: 25%|██▍ | 1967/8000 [00:37<23:33, 4.27it/s, lr: 4.0e-05 loss: 8.046e-02] amber: 25%|██▍ | 1968/8000 [00:37<31:55, 3.15it/s, lr: 4.0e-05 loss: 8.046e-02] amber: 25%|██▍ | 1968/8000 [00:37<31:55, 3.15it/s, lr: 4.0e-05 loss: 2.710e-01] amber: 25%|██▍ | 1969/8000 [00:37<23:56, 4.20it/s, lr: 4.0e-05 loss: 2.710e-01] amber: 25%|██▍ | 1969/8000 [00:37<23:56, 4.20it/s, lr: 4.0e-05 loss: 7.108e-02] amber: 25%|██▍ | 1970/8000 [00:37<20:52, 4.81it/s, lr: 4.0e-05 loss: 7.108e-02] amber: 25%|██▍ | 1970/8000 [00:38<20:52, 4.81it/s, lr: 4.0e-05 loss: 2.216e-02] amber: 25%|██▍ | 1971/8000 [00:38<36:07, 2.78it/s, lr: 4.0e-05 loss: 2.216e-02] amber: 25%|██▍ | 1971/8000 [00:38<36:07, 2.78it/s, lr: 4.0e-05 loss: 8.008e-02] amber: 25%|██▍ | 1972/8000 [00:38<23:22, 4.30it/s, lr: 4.0e-05 loss: 8.008e-02] amber: 25%|██▍ | 1972/8000 [00:38<23:22, 4.30it/s, lr: 4.0e-05 loss: 2.002e-01] amber: 25%|██▍ | 1973/8000 [00:38<31:34, 3.18it/s, lr: 4.0e-05 loss: 2.002e-01] amber: 25%|██▍ | 1973/8000 [00:38<31:34, 3.18it/s, lr: 4.0e-05 loss: 1.494e-01] amber: 25%|██▍ | 1974/8000 [00:38<35:58, 2.79it/s, lr: 4.0e-05 loss: 1.494e-01] amber: 25%|██▍ | 1974/8000 [00:39<35:58, 2.79it/s, lr: 4.0e-05 loss: 1.686e-01] amber: 25%|██▍ | 1975/8000 [00:39<27:23, 3.67it/s, lr: 4.0e-05 loss: 1.686e-01] amber: 25%|██▍ | 1975/8000 [00:39<27:23, 3.67it/s, lr: 4.0e-05 loss: 9.893e-02] amber: 25%|██▍ | 1976/8000 [00:39<33:29, 3.00it/s, lr: 4.0e-05 loss: 9.893e-02] amber: 25%|██▍ | 1976/8000 [00:39<33:29, 3.00it/s, lr: 4.0e-05 loss: 1.226e-01] amber: 25%|██▍ | 1977/8000 [00:39<23:10, 4.33it/s, lr: 4.0e-05 loss: 1.226e-01] amber: 25%|██▍ | 1977/8000 [00:39<23:10, 4.33it/s, lr: 4.0e-05 loss: 6.847e-02] amber: 25%|██▍ | 1978/8000 [00:39<20:58, 4.78it/s, lr: 4.0e-05 loss: 6.847e-02] amber: 25%|██▍ | 1978/8000 [00:40<20:58, 4.78it/s, lr: 4.0e-05 loss: 1.219e-01] amber: 25%|██▍ | 1979/8000 [00:40<20:47, 4.83it/s, lr: 4.0e-05 loss: 1.219e-01] amber: 25%|██▍ | 1979/8000 [00:40<20:47, 4.83it/s, lr: 4.0e-05 loss: 3.896e-02] amber: 25%|██▍ | 1980/8000 [00:40<29:47, 3.37it/s, lr: 4.0e-05 loss: 3.896e-02] amber: 25%|██▍ | 1980/8000 [00:40<29:47, 3.37it/s, lr: 4.0e-05 loss: 2.290e-01]
Saving at step 1980
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000001980.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 25%|██▍ | 1981/8000 [00:41<29:47, 3.37it/s, lr: 4.0e-05 loss: 1.094e-01] amber: 25%|██▍ | 1982/8000 [00:41<35:30, 2.82it/s, lr: 4.0e-05 loss: 1.094e-01] amber: 25%|██▍ | 1982/8000 [00:41<35:30, 2.82it/s, lr: 4.0e-05 loss: 5.053e-02] amber: 25%|██▍ | 1983/8000 [00:41<27:55, 3.59it/s, lr: 4.0e-05 loss: 5.053e-02] amber: 25%|██▍ | 1983/8000 [00:42<27:55, 3.59it/s, lr: 4.0e-05 loss: 1.809e-01] amber: 25%|██▍ | 1984/8000 [00:42<23:12, 4.32it/s, lr: 4.0e-05 loss: 1.809e-01] amber: 25%|██▍ | 1984/8000 [00:42<23:12, 4.32it/s, lr: 4.0e-05 loss: 1.635e-01] amber: 25%|██▍ | 1985/8000 [00:42<31:07, 3.22it/s, lr: 4.0e-05 loss: 1.635e-01] amber: 25%|██▍ | 1985/8000 [00:42<31:07, 3.22it/s, lr: 4.0e-05 loss: 1.474e-01] amber: 25%|██▍ | 1986/8000 [00:42<25:10, 3.98it/s, lr: 4.0e-05 loss: 1.474e-01] amber: 25%|██▍ | 1986/8000 [00:42<25:10, 3.98it/s, lr: 4.0e-05 loss: 8.887e-03] amber: 25%|██▍ | 1987/8000 [00:42<23:22, 4.29it/s, lr: 4.0e-05 loss: 8.887e-03] amber: 25%|██▍ | 1987/8000 [00:43<23:22, 4.29it/s, lr: 4.0e-05 loss: 9.898e-02] amber: 25%|██▍ | 1988/8000 [00:43<31:15, 3.21it/s, lr: 4.0e-05 loss: 9.898e-02] amber: 25%|██▍ | 1988/8000 [00:43<31:15, 3.21it/s, lr: 4.0e-05 loss: 1.016e-01] amber: 25%|██▍ | 1989/8000 [00:43<34:56, 2.87it/s, lr: 4.0e-05 loss: 1.016e-01] amber: 25%|██▍ | 1989/8000 [00:43<34:56, 2.87it/s, lr: 4.0e-05 loss: 1.228e-01] amber: 25%|██▍ | 1990/8000 [00:43<24:56, 4.01it/s, lr: 4.0e-05 loss: 1.228e-01] amber: 25%|██▍ | 1990/8000 [00:44<24:56, 4.01it/s, lr: 4.0e-05 loss: 1.307e-01] amber: 25%|██▍ | 1991/8000 [00:44<31:57, 3.13it/s, lr: 4.0e-05 loss: 1.307e-01] amber: 25%|██▍ | 1991/8000 [00:44<31:57, 3.13it/s, lr: 4.0e-05 loss: 1.118e-01] amber: 25%|██▍ | 1992/8000 [00:44<34:48, 2.88it/s, lr: 4.0e-05 loss: 1.118e-01] amber: 25%|██▍ | 1992/8000 [00:44<34:48, 2.88it/s, lr: 4.0e-05 loss: 9.800e-02] amber: 25%|██▍ | 1993/8000 [00:44<34:22, 2.91it/s, lr: 4.0e-05 loss: 9.800e-02] amber: 25%|██▍ | 1993/8000 [00:45<34:22, 2.91it/s, lr: 4.0e-05 loss: 8.639e-02] amber: 25%|██▍ | 1994/8000 [00:45<35:32, 2.82it/s, lr: 4.0e-05 loss: 8.639e-02] amber: 25%|██▍ | 1994/8000 [00:45<35:32, 2.82it/s, lr: 4.0e-05 loss: 1.114e-01] amber: 25%|██▍ | 1995/8000 [00:45<23:06, 4.33it/s, lr: 4.0e-05 loss: 1.114e-01] amber: 25%|██▍ | 1995/8000 [00:45<23:06, 4.33it/s, lr: 4.0e-05 loss: 4.177e-02] amber: 25%|██▍ | 1996/8000 [00:45<30:40, 3.26it/s, lr: 4.0e-05 loss: 4.177e-02] amber: 25%|██▍ | 1996/8000 [00:46<30:40, 3.26it/s, lr: 4.0e-05 loss: 9.985e-02] amber: 25%|██▍ | 1997/8000 [00:46<34:41, 2.88it/s, lr: 4.0e-05 loss: 9.985e-02] amber: 25%|██▍ | 1997/8000 [00:46<34:41, 2.88it/s, lr: 4.0e-05 loss: 9.185e-03] amber: 25%|██▍ | 1998/8000 [00:46<23:55, 4.18it/s, lr: 4.0e-05 loss: 9.185e-03] amber: 25%|██▍ | 1998/8000 [00:46<23:55, 4.18it/s, lr: 4.0e-05 loss: 8.035e-02] amber: 25%|██▍ | 1999/8000 [00:46<30:31, 3.28it/s, lr: 4.0e-05 loss: 8.035e-02] amber: 25%|██▍ | 1999/8000 [00:46<30:31, 3.28it/s, lr: 4.0e-05 loss: 5.009e-02] amber: 25%|██▌ | 2000/8000 [00:46<35:21, 2.83it/s, lr: 4.0e-05 loss: 5.009e-02] amber: 25%|██▌ | 2000/8000 [00:47<35:21, 2.83it/s, lr: 4.0e-05 loss: 3.544e-02]
Saving at step 2000
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002000.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:00<00:01, 1.18it/s]
Generating Images: 67%|██████▋ | 2/3 [00:01<00:00, 1.15it/s]
Generating Images: 100%|██████████| 3/3 [00:02<00:00, 1.16it/s]

amber: 25%|██▌ | 2001/8000 [00:48<35:20, 2.83it/s, lr: 4.0e-05 loss: 1.426e-01] amber: 25%|██▌ | 2002/8000 [00:48<35:21, 2.83it/s, lr: 4.0e-05 loss: 1.426e-01] amber: 25%|██▌ | 2002/8000 [00:48<35:21, 2.83it/s, lr: 4.0e-05 loss: 1.090e-01] amber: 25%|██▌ | 2003/8000 [00:48<26:23, 3.79it/s, lr: 4.0e-05 loss: 1.090e-01] amber: 25%|██▌ | 2003/8000 [00:48<26:23, 3.79it/s, lr: 4.0e-05 loss: 1.443e-01] amber: 25%|██▌ | 2004/8000 [00:48<21:29, 4.65it/s, lr: 4.0e-05 loss: 1.443e-01] amber: 25%|██▌ | 2004/8000 [00:48<21:29, 4.65it/s, lr: 4.0e-05 loss: 1.063e-01] amber: 25%|██▌ | 2005/8000 [00:48<20:43, 4.82it/s, lr: 4.0e-05 loss: 1.063e-01] amber: 25%|██▌ | 2005/8000 [00:48<20:43, 4.82it/s, lr: 4.0e-05 loss: 6.498e-02] amber: 25%|██▌ | 2006/8000 [00:48<20:01, 4.99it/s, lr: 4.0e-05 loss: 6.498e-02] amber: 25%|██▌ | 2006/8000 [00:49<20:01, 4.99it/s, lr: 4.0e-05 loss: 1.044e-01] amber: 25%|██▌ | 2007/8000 [00:49<21:13, 4.71it/s, lr: 4.0e-05 loss: 1.044e-01] amber: 25%|██▌ | 2007/8000 [00:49<21:13, 4.71it/s, lr: 4.0e-05 loss: 6.096e-02] amber: 25%|██▌ | 2008/8000 [00:49<18:58, 5.26it/s, lr: 4.0e-05 loss: 6.096e-02] amber: 25%|██▌ | 2008/8000 [00:49<18:58, 5.26it/s, lr: 4.0e-05 loss: 4.601e-03] amber: 25%|██▌ | 2009/8000 [00:49<19:02, 5.24it/s, lr: 4.0e-05 loss: 4.601e-03] amber: 25%|██▌ | 2009/8000 [00:49<19:02, 5.24it/s, lr: 4.0e-05 loss: 1.077e-01] amber: 25%|██▌ | 2010/8000 [00:49<20:44, 4.82it/s, lr: 4.0e-05 loss: 1.077e-01] amber: 25%|██▌ | 2010/8000 [00:50<20:44, 4.82it/s, lr: 4.0e-05 loss: 2.060e-02] amber: 25%|██▌ | 2011/8000 [00:50<39:02, 2.56it/s, lr: 4.0e-05 loss: 2.060e-02] amber: 25%|██▌ | 2011/8000 [00:50<39:02, 2.56it/s, lr: 4.0e-05 loss: 5.982e-03] amber: 25%|██▌ | 2012/8000 [00:50<33:06, 3.01it/s, lr: 4.0e-05 loss: 5.982e-03] amber: 25%|██▌ | 2012/8000 [00:50<33:06, 3.01it/s, lr: 4.0e-05 loss: 5.834e-02] amber: 25%|██▌ | 2013/8000 [00:50<35:01, 2.85it/s, lr: 4.0e-05 loss: 5.834e-02] amber: 25%|██▌ | 2013/8000 [00:50<35:01, 2.85it/s, lr: 4.0e-05 loss: 3.884e-02] amber: 25%|██▌ | 2014/8000 [00:50<26:31, 3.76it/s, lr: 4.0e-05 loss: 3.884e-02] amber: 25%|██▌ | 2014/8000 [00:51<26:31, 3.76it/s, lr: 4.0e-05 loss: 8.473e-02] amber: 25%|██▌ | 2015/8000 [00:51<23:43, 4.20it/s, lr: 4.0e-05 loss: 8.473e-02] amber: 25%|██▌ | 2015/8000 [00:51<23:43, 4.20it/s, lr: 4.0e-05 loss: 6.994e-02] amber: 25%|██▌ | 2016/8000 [00:51<31:08, 3.20it/s, lr: 4.0e-05 loss: 6.994e-02] amber: 25%|██▌ | 2016/8000 [00:51<31:08, 3.20it/s, lr: 4.0e-05 loss: 1.789e-01] amber: 25%|██▌ | 2017/8000 [00:51<35:07, 2.84it/s, lr: 4.0e-05 loss: 1.789e-01] amber: 25%|██▌ | 2017/8000 [00:52<35:07, 2.84it/s, lr: 4.0e-05 loss: 1.979e-01] amber: 25%|██▌ | 2018/8000 [00:52<25:01, 3.98it/s, lr: 4.0e-05 loss: 1.979e-01] amber: 25%|██▌ | 2018/8000 [00:52<25:01, 3.98it/s, lr: 4.0e-05 loss: 7.844e-02] amber: 25%|██▌ | 2019/8000 [00:52<21:52, 4.56it/s, lr: 4.0e-05 loss: 7.844e-02] amber: 25%|██▌ | 2019/8000 [00:52<21:52, 4.56it/s, lr: 4.0e-05 loss: 5.801e-02] amber: 25%|██▌ | 2020/8000 [00:52<20:06, 4.95it/s, lr: 4.0e-05 loss: 5.801e-02] amber: 25%|██▌ | 2020/8000 [00:52<20:06, 4.95it/s, lr: 4.0e-05 loss: 1.169e-01]
Saving at step 2020
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002020.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 25%|██▌ | 2021/8000 [00:53<20:06, 4.95it/s, lr: 4.0e-05 loss: 1.525e-01] amber: 25%|██▌ | 2022/8000 [00:53<36:10, 2.75it/s, lr: 4.0e-05 loss: 1.525e-01] amber: 25%|██▌ | 2022/8000 [00:54<36:10, 2.75it/s, lr: 4.0e-05 loss: 1.194e-01] amber: 25%|██▌ | 2023/8000 [00:54<36:04, 2.76it/s, lr: 4.0e-05 loss: 1.194e-01] amber: 25%|██▌ | 2023/8000 [00:54<36:04, 2.76it/s, lr: 4.0e-05 loss: 2.074e-01] amber: 25%|██▌ | 2024/8000 [00:54<34:55, 2.85it/s, lr: 4.0e-05 loss: 2.074e-01] amber: 25%|██▌ | 2024/8000 [00:54<34:55, 2.85it/s, lr: 4.0e-05 loss: 1.897e-01] amber: 25%|██▌ | 2025/8000 [00:54<35:25, 2.81it/s, lr: 4.0e-05 loss: 1.897e-01] amber: 25%|██▌ | 2025/8000 [00:54<35:25, 2.81it/s, lr: 4.0e-05 loss: 2.307e-01] amber: 25%|██▌ | 2026/8000 [00:54<24:07, 4.13it/s, lr: 4.0e-05 loss: 2.307e-01] amber: 25%|██▌ | 2026/8000 [00:55<24:07, 4.13it/s, lr: 4.0e-05 loss: 6.733e-02] amber: 25%|██▌ | 2027/8000 [00:55<20:33, 4.84it/s, lr: 4.0e-05 loss: 6.733e-02] amber: 25%|██▌ | 2027/8000 [00:55<20:33, 4.84it/s, lr: 4.0e-05 loss: 4.323e-02] amber: 25%|██▌ | 2028/8000 [00:55<21:01, 4.73it/s, lr: 4.0e-05 loss: 4.323e-02] amber: 25%|██▌ | 2028/8000 [00:55<21:01, 4.73it/s, lr: 4.0e-05 loss: 1.016e-01] amber: 25%|██▌ | 2029/8000 [00:55<29:58, 3.32it/s, lr: 4.0e-05 loss: 1.016e-01] amber: 25%|██▌ | 2029/8000 [00:56<29:58, 3.32it/s, lr: 4.0e-05 loss: 7.190e-02] amber: 25%|██▌ | 2030/8000 [00:56<34:19, 2.90it/s, lr: 4.0e-05 loss: 7.190e-02] amber: 25%|██▌ | 2030/8000 [00:56<34:19, 2.90it/s, lr: 4.0e-05 loss: 2.160e-01] amber: 25%|██▌ | 2031/8000 [00:56<23:39, 4.20it/s, lr: 4.0e-05 loss: 2.160e-01] amber: 25%|██▌ | 2031/8000 [00:56<23:39, 4.20it/s, lr: 4.0e-05 loss: 2.573e-01] amber: 25%|██▌ | 2032/8000 [00:56<20:55, 4.75it/s, lr: 4.0e-05 loss: 2.573e-01] amber: 25%|██▌ | 2032/8000 [00:56<20:55, 4.75it/s, lr: 4.0e-05 loss: 7.502e-02] amber: 25%|██▌ | 2033/8000 [00:56<29:52, 3.33it/s, lr: 4.0e-05 loss: 7.502e-02] amber: 25%|██▌ | 2033/8000 [00:56<29:52, 3.33it/s, lr: 4.0e-05 loss: 2.010e-01] amber: 25%|██▌ | 2034/8000 [00:56<25:03, 3.97it/s, lr: 4.0e-05 loss: 2.010e-01] amber: 25%|██▌ | 2034/8000 [00:57<25:03, 3.97it/s, lr: 4.0e-05 loss: 2.018e-01] amber: 25%|██▌ | 2035/8000 [00:57<23:27, 4.24it/s, lr: 4.0e-05 loss: 2.018e-01] amber: 25%|██▌ | 2035/8000 [00:57<23:27, 4.24it/s, lr: 4.0e-05 loss: 8.124e-02] amber: 25%|██▌ | 2036/8000 [00:57<31:02, 3.20it/s, lr: 4.0e-05 loss: 8.124e-02] amber: 25%|██▌ | 2036/8000 [00:57<31:02, 3.20it/s, lr: 4.0e-05 loss: 1.077e-01] amber: 25%|██▌ | 2037/8000 [00:57<25:39, 3.87it/s, lr: 4.0e-05 loss: 1.077e-01] amber: 25%|██▌ | 2037/8000 [00:58<25:39, 3.87it/s, lr: 4.0e-05 loss: 8.081e-02] amber: 25%|██▌ | 2038/8000 [00:58<23:05, 4.30it/s, lr: 4.0e-05 loss: 8.081e-02] amber: 25%|██▌ | 2038/8000 [00:58<23:05, 4.30it/s, lr: 4.0e-05 loss: 1.668e-01] amber: 25%|██▌ | 2039/8000 [00:58<30:39, 3.24it/s, lr: 4.0e-05 loss: 1.668e-01] amber: 25%|██▌ | 2039/8000 [00:58<30:39, 3.24it/s, lr: 4.0e-05 loss: 1.955e-01] amber: 26%|██▌ | 2040/8000 [00:58<34:37, 2.87it/s, lr: 4.0e-05 loss: 1.955e-01] amber: 26%|██▌ | 2040/8000 [00:58<34:37, 2.87it/s, lr: 4.0e-05 loss: 1.165e-01]
Saving at step 2040
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002040.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:00<00:01, 1.09it/s]
Generating Images: 67%|██████▋ | 2/3 [00:01<00:00, 1.10it/s]
Generating Images: 100%|██████████| 3/3 [00:02<00:00, 1.09it/s]

amber: 26%|██▌ | 2041/8000 [00:59<34:37, 2.87it/s, lr: 4.0e-05 loss: 1.612e-01] amber: 26%|██▌ | 2042/8000 [00:59<37:48, 2.63it/s, lr: 4.0e-05 loss: 1.612e-01] amber: 26%|██▌ | 2042/8000 [01:00<37:48, 2.63it/s, lr: 4.0e-05 loss: 8.940e-02] amber: 26%|██▌ | 2043/8000 [01:00<35:17, 2.81it/s, lr: 4.0e-05 loss: 8.940e-02] amber: 26%|██▌ | 2043/8000 [01:00<35:17, 2.81it/s, lr: 4.0e-05 loss: 9.794e-02] amber: 26%|██▌ | 2044/8000 [01:00<35:38, 2.78it/s, lr: 4.0e-05 loss: 9.794e-02] amber: 26%|██▌ | 2044/8000 [01:00<35:38, 2.78it/s, lr: 4.0e-05 loss: 1.102e-02] amber: 26%|██▌ | 2045/8000 [01:00<24:59, 3.97it/s, lr: 4.0e-05 loss: 1.102e-02] amber: 26%|██▌ | 2045/8000 [01:00<24:59, 3.97it/s, lr: 4.0e-05 loss: 4.759e-03] amber: 26%|██▌ | 2046/8000 [01:00<21:02, 4.71it/s, lr: 4.0e-05 loss: 4.759e-03] amber: 26%|██▌ | 2046/8000 [01:01<21:02, 4.71it/s, lr: 4.0e-05 loss: 1.554e-01] amber: 26%|██▌ | 2047/8000 [01:01<30:11, 3.29it/s, lr: 4.0e-05 loss: 1.554e-01] amber: 26%|██▌ | 2047/8000 [01:01<30:11, 3.29it/s, lr: 4.0e-05 loss: 8.086e-02] amber: 26%|██▌ | 2048/8000 [01:01<24:04, 4.12it/s, lr: 4.0e-05 loss: 8.086e-02] amber: 26%|██▌ | 2048/8000 [01:01<24:04, 4.12it/s, lr: 4.0e-05 loss: 6.062e-02] amber: 26%|██▌ | 2049/8000 [01:01<30:50, 3.22it/s, lr: 4.0e-05 loss: 6.062e-02] amber: 26%|██▌ | 2049/8000 [01:01<30:50, 3.22it/s, lr: 4.0e-05 loss: 1.079e-01] amber: 26%|██▌ | 2050/8000 [01:01<23:03, 4.30it/s, lr: 4.0e-05 loss: 1.079e-01] amber: 26%|██▌ | 2050/8000 [01:02<23:03, 4.30it/s, lr: 4.0e-05 loss: 1.285e-01] amber: 26%|██▌ | 2051/8000 [01:02<42:26, 2.34it/s, lr: 4.0e-05 loss: 1.285e-01] amber: 26%|██▌ | 2051/8000 [01:02<42:26, 2.34it/s, lr: 4.0e-05 loss: 1.044e-01] amber: 26%|██▌ | 2052/8000 [01:02<25:50, 3.84it/s, lr: 4.0e-05 loss: 1.044e-01] amber: 26%|██▌ | 2052/8000 [01:02<25:50, 3.84it/s, lr: 4.0e-05 loss: 1.005e-01] amber: 26%|██▌ | 2053/8000 [01:02<31:29, 3.15it/s, lr: 4.0e-05 loss: 1.005e-01] amber: 26%|██▌ | 2053/8000 [01:03<31:29, 3.15it/s, lr: 4.0e-05 loss: 1.263e-01] amber: 26%|██▌ | 2054/8000 [01:03<24:59, 3.97it/s, lr: 4.0e-05 loss: 1.263e-01] amber: 26%|██▌ | 2054/8000 [01:03<24:59, 3.97it/s, lr: 4.0e-05 loss: 7.761e-02] amber: 26%|██▌ | 2055/8000 [01:03<22:26, 4.42it/s, lr: 4.0e-05 loss: 7.761e-02] amber: 26%|██▌ | 2055/8000 [01:03<22:26, 4.42it/s, lr: 4.0e-05 loss: 2.239e-02] amber: 26%|██▌ | 2056/8000 [01:03<30:35, 3.24it/s, lr: 4.0e-05 loss: 2.239e-02] amber: 26%|██▌ | 2056/8000 [01:03<30:35, 3.24it/s, lr: 4.0e-05 loss: 2.033e-01] amber: 26%|██▌ | 2057/8000 [01:03<25:51, 3.83it/s, lr: 4.0e-05 loss: 2.033e-01] amber: 26%|██▌ | 2057/8000 [01:04<25:51, 3.83it/s, lr: 4.0e-05 loss: 1.505e-01] amber: 26%|██▌ | 2058/8000 [01:04<22:42, 4.36it/s, lr: 4.0e-05 loss: 1.505e-01] amber: 26%|██▌ | 2058/8000 [01:04<22:42, 4.36it/s, lr: 4.0e-05 loss: 1.697e-01] amber: 26%|██▌ | 2059/8000 [01:04<30:24, 3.26it/s, lr: 4.0e-05 loss: 1.697e-01] amber: 26%|██▌ | 2059/8000 [01:04<30:24, 3.26it/s, lr: 4.0e-05 loss: 1.704e-01] amber: 26%|██▌ | 2060/8000 [01:04<35:13, 2.81it/s, lr: 4.0e-05 loss: 1.704e-01] amber: 26%|██▌ | 2060/8000 [01:05<35:13, 2.81it/s, lr: 4.0e-05 loss: 1.099e-01]
Saving at step 2060
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002060.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 26%|██▌ | 2061/8000 [01:05<35:13, 2.81it/s, lr: 4.0e-05 loss: 5.211e-02] amber: 26%|██▌ | 2062/8000 [01:05<28:53, 3.43it/s, lr: 4.0e-05 loss: 5.211e-02] amber: 26%|██▌ | 2062/8000 [01:05<28:53, 3.43it/s, lr: 4.0e-05 loss: 2.110e-01] amber: 26%|██▌ | 2063/8000 [01:05<22:23, 4.42it/s, lr: 4.0e-05 loss: 2.110e-01] amber: 26%|██▌ | 2063/8000 [01:06<22:23, 4.42it/s, lr: 4.0e-05 loss: 1.115e-01] amber: 26%|██▌ | 2064/8000 [01:06<30:27, 3.25it/s, lr: 4.0e-05 loss: 1.115e-01] amber: 26%|██▌ | 2064/8000 [01:06<30:27, 3.25it/s, lr: 4.0e-05 loss: 1.025e-01] amber: 26%|██▌ | 2065/8000 [01:06<33:04, 2.99it/s, lr: 4.0e-05 loss: 1.025e-01] amber: 26%|██▌ | 2065/8000 [01:07<33:04, 2.99it/s, lr: 4.0e-05 loss: 1.742e-01] amber: 26%|██▌ | 2066/8000 [01:07<34:09, 2.90it/s, lr: 4.0e-05 loss: 1.742e-01] amber: 26%|██▌ | 2066/8000 [01:07<34:09, 2.90it/s, lr: 4.0e-05 loss: 1.446e-01] amber: 26%|██▌ | 2067/8000 [01:07<36:10, 2.73it/s, lr: 4.0e-05 loss: 1.446e-01] amber: 26%|██▌ | 2067/8000 [01:07<36:10, 2.73it/s, lr: 4.0e-05 loss: 8.856e-02] amber: 26%|██▌ | 2068/8000 [01:07<24:46, 3.99it/s, lr: 4.0e-05 loss: 8.856e-02] amber: 26%|██▌ | 2068/8000 [01:07<24:46, 3.99it/s, lr: 4.0e-05 loss: 8.445e-03] amber: 26%|██▌ | 2069/8000 [01:07<21:11, 4.67it/s, lr: 4.0e-05 loss: 8.445e-03] amber: 26%|██▌ | 2069/8000 [01:08<21:11, 4.67it/s, lr: 4.0e-05 loss: 1.898e-01] amber: 26%|██▌ | 2070/8000 [01:08<21:38, 4.57it/s, lr: 4.0e-05 loss: 1.898e-01] amber: 26%|██▌ | 2070/8000 [01:08<21:38, 4.57it/s, lr: 4.0e-05 loss: 6.244e-02] amber: 26%|██▌ | 2071/8000 [01:08<29:53, 3.31it/s, lr: 4.0e-05 loss: 6.244e-02] amber: 26%|██▌ | 2071/8000 [01:08<29:53, 3.31it/s, lr: 4.0e-05 loss: 5.093e-02] amber: 26%|██▌ | 2072/8000 [01:08<23:53, 4.14it/s, lr: 4.0e-05 loss: 5.093e-02] amber: 26%|██▌ | 2072/8000 [01:08<23:53, 4.14it/s, lr: 4.0e-05 loss: 8.127e-02] amber: 26%|██▌ | 2073/8000 [01:08<31:10, 3.17it/s, lr: 4.0e-05 loss: 8.127e-02] amber: 26%|██▌ | 2073/8000 [01:09<31:10, 3.17it/s, lr: 4.0e-05 loss: 3.082e-02] amber: 26%|██▌ | 2074/8000 [01:09<26:16, 3.76it/s, lr: 4.0e-05 loss: 3.082e-02] amber: 26%|██▌ | 2074/8000 [01:09<26:16, 3.76it/s, lr: 4.0e-05 loss: 1.509e-01] amber: 26%|██▌ | 2075/8000 [01:09<23:24, 4.22it/s, lr: 4.0e-05 loss: 1.509e-01] amber: 26%|██▌ | 2075/8000 [01:09<23:24, 4.22it/s, lr: 4.0e-05 loss: 8.395e-02] amber: 26%|██▌ | 2076/8000 [01:09<20:52, 4.73it/s, lr: 4.0e-05 loss: 8.395e-02] amber: 26%|██▌ | 2076/8000 [01:09<20:52, 4.73it/s, lr: 4.0e-05 loss: 5.587e-03] amber: 26%|██▌ | 2077/8000 [01:09<19:40, 5.02it/s, lr: 4.0e-05 loss: 5.587e-03] amber: 26%|██▌ | 2077/8000 [01:10<19:40, 5.02it/s, lr: 4.0e-05 loss: 4.107e-02] amber: 26%|██▌ | 2078/8000 [01:10<29:02, 3.40it/s, lr: 4.0e-05 loss: 4.107e-02] amber: 26%|██▌ | 2078/8000 [01:10<29:02, 3.40it/s, lr: 4.0e-05 loss: 1.664e-01] amber: 26%|██▌ | 2079/8000 [01:10<23:13, 4.25it/s, lr: 4.0e-05 loss: 1.664e-01] amber: 26%|██▌ | 2079/8000 [01:10<23:13, 4.25it/s, lr: 4.0e-05 loss: 1.709e-01] amber: 26%|██▌ | 2080/8000 [01:10<19:25, 5.08it/s, lr: 4.0e-05 loss: 1.709e-01] amber: 26%|██▌ | 2080/8000 [01:10<19:25, 5.08it/s, lr: 4.0e-05 loss: 1.269e-01]
Saving at step 2080
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002080.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:00<00:01, 1.11it/s]
Generating Images: 67%|██████▋ | 2/3 [00:01<00:00, 1.09it/s]
Generating Images: 100%|██████████| 3/3 [00:02<00:00, 1.08it/s]

amber: 26%|██▌ | 2081/8000 [01:11<19:25, 5.08it/s, lr: 4.0e-05 loss: 6.502e-02] amber: 26%|██▌ | 2082/8000 [01:11<35:22, 2.79it/s, lr: 4.0e-05 loss: 6.502e-02] amber: 26%|██▌ | 2082/8000 [01:11<35:22, 2.79it/s, lr: 4.0e-05 loss: 1.075e-01] amber: 26%|██▌ | 2083/8000 [01:11<34:17, 2.88it/s, lr: 4.0e-05 loss: 1.075e-01] amber: 26%|██▌ | 2083/8000 [01:12<34:17, 2.88it/s, lr: 4.0e-05 loss: 1.100e-01] amber: 26%|██▌ | 2084/8000 [01:12<24:27, 4.03it/s, lr: 4.0e-05 loss: 1.100e-01] amber: 26%|██▌ | 2084/8000 [01:12<24:27, 4.03it/s, lr: 4.0e-05 loss: 1.608e-01] amber: 26%|██▌ | 2085/8000 [01:12<19:06, 5.16it/s, lr: 4.0e-05 loss: 1.608e-01] amber: 26%|██▌ | 2085/8000 [01:12<19:06, 5.16it/s, lr: 4.0e-05 loss: 1.636e-01] amber: 26%|██▌ | 2086/8000 [01:12<31:56, 3.09it/s, lr: 4.0e-05 loss: 1.636e-01] amber: 26%|██▌ | 2086/8000 [01:12<31:56, 3.09it/s, lr: 4.0e-05 loss: 7.658e-02] amber: 26%|██▌ | 2087/8000 [01:12<20:49, 4.73it/s, lr: 4.0e-05 loss: 7.658e-02] amber: 26%|██▌ | 2087/8000 [01:13<20:49, 4.73it/s, lr: 4.0e-05 loss: 6.005e-02] amber: 26%|██▌ | 2088/8000 [01:13<29:36, 3.33it/s, lr: 4.0e-05 loss: 6.005e-02] amber: 26%|██▌ | 2088/8000 [01:13<29:36, 3.33it/s, lr: 4.0e-05 loss: 1.628e-01] amber: 26%|██▌ | 2089/8000 [01:13<32:17, 3.05it/s, lr: 4.0e-05 loss: 1.628e-01] amber: 26%|██▌ | 2089/8000 [01:13<32:17, 3.05it/s, lr: 4.0e-05 loss: 1.245e-01] amber: 26%|██▌ | 2090/8000 [01:13<33:53, 2.91it/s, lr: 4.0e-05 loss: 1.245e-01] amber: 26%|██▌ | 2090/8000 [01:14<33:53, 2.91it/s, lr: 4.0e-05 loss: 1.820e-01] amber: 26%|██▌ | 2091/8000 [01:14<41:47, 2.36it/s, lr: 4.0e-05 loss: 1.820e-01] amber: 26%|██▌ | 2091/8000 [01:14<41:47, 2.36it/s, lr: 4.0e-05 loss: 1.125e-01] amber: 26%|██▌ | 2092/8000 [01:14<28:35, 3.44it/s, lr: 4.0e-05 loss: 1.125e-01] amber: 26%|██▌ | 2092/8000 [01:14<28:35, 3.44it/s, lr: 4.0e-05 loss: 3.663e-02] amber: 26%|██▌ | 2093/8000 [01:14<22:15, 4.42it/s, lr: 4.0e-05 loss: 3.663e-02] amber: 26%|██▌ | 2093/8000 [01:15<22:15, 4.42it/s, lr: 4.0e-05 loss: 8.886e-02] amber: 26%|██▌ | 2094/8000 [01:15<30:24, 3.24it/s, lr: 4.0e-05 loss: 8.886e-02] amber: 26%|██▌ | 2094/8000 [01:15<30:24, 3.24it/s, lr: 4.0e-05 loss: 2.898e-02] amber: 26%|██▌ | 2095/8000 [01:15<23:08, 4.25it/s, lr: 4.0e-05 loss: 2.898e-02] amber: 26%|██▌ | 2095/8000 [01:15<23:08, 4.25it/s, lr: 4.0e-05 loss: 2.922e-01] amber: 26%|██▌ | 2096/8000 [01:15<31:54, 3.08it/s, lr: 4.0e-05 loss: 2.922e-01] amber: 26%|██▌ | 2096/8000 [01:15<31:54, 3.08it/s, lr: 4.0e-05 loss: 1.989e-01] amber: 26%|██▌ | 2097/8000 [01:15<32:54, 2.99it/s, lr: 4.0e-05 loss: 1.989e-01] amber: 26%|██▌ | 2097/8000 [01:16<32:54, 2.99it/s, lr: 4.0e-05 loss: 8.650e-02] amber: 26%|██▌ | 2098/8000 [01:16<24:31, 4.01it/s, lr: 4.0e-05 loss: 8.650e-02] amber: 26%|██▌ | 2098/8000 [01:16<24:31, 4.01it/s, lr: 4.0e-05 loss: 1.999e-02] amber: 26%|██▌ | 2099/8000 [01:16<20:42, 4.75it/s, lr: 4.0e-05 loss: 1.999e-02] amber: 26%|██▌ | 2099/8000 [01:16<20:42, 4.75it/s, lr: 4.0e-05 loss: 1.315e-01] amber: 26%|██▋ | 2100/8000 [01:16<30:11, 3.26it/s, lr: 4.0e-05 loss: 1.315e-01] amber: 26%|██▋ | 2100/8000 [01:16<30:11, 3.26it/s, lr: 4.0e-05 loss: 8.510e-02]
Saving at step 2100
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002100.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 26%|██▋ | 2101/8000 [01:17<30:11, 3.26it/s, lr: 4.0e-05 loss: 3.968e-02] amber: 26%|██▋ | 2102/8000 [01:17<30:30, 3.22it/s, lr: 4.0e-05 loss: 3.968e-02] amber: 26%|██▋ | 2102/8000 [01:17<30:30, 3.22it/s, lr: 4.0e-05 loss: 4.500e-02] amber: 26%|██▋ | 2103/8000 [01:17<33:36, 2.93it/s, lr: 4.0e-05 loss: 4.500e-02] amber: 26%|██▋ | 2103/8000 [01:18<33:36, 2.93it/s, lr: 4.0e-05 loss: 2.074e-01] amber: 26%|██▋ | 2104/8000 [01:18<23:41, 4.15it/s, lr: 4.0e-05 loss: 2.074e-01] amber: 26%|██▋ | 2104/8000 [01:18<23:41, 4.15it/s, lr: 4.0e-05 loss: 5.697e-02] amber: 26%|██▋ | 2105/8000 [01:18<19:53, 4.94it/s, lr: 4.0e-05 loss: 5.697e-02] amber: 26%|██▋ | 2105/8000 [01:18<19:53, 4.94it/s, lr: 4.0e-05 loss: 1.172e-01] amber: 26%|██▋ | 2106/8000 [01:18<29:19, 3.35it/s, lr: 4.0e-05 loss: 1.172e-01] amber: 26%|██▋ | 2106/8000 [01:18<29:19, 3.35it/s, lr: 4.0e-05 loss: 1.351e-01] amber: 26%|██▋ | 2107/8000 [01:18<34:26, 2.85it/s, lr: 4.0e-05 loss: 1.351e-01] amber: 26%|██▋ | 2107/8000 [01:19<34:26, 2.85it/s, lr: 4.0e-05 loss: 9.223e-02] amber: 26%|██▋ | 2108/8000 [01:19<29:36, 3.32it/s, lr: 4.0e-05 loss: 9.223e-02] amber: 26%|██▋ | 2108/8000 [01:19<29:36, 3.32it/s, lr: 4.0e-05 loss: 1.487e-01] amber: 26%|██▋ | 2109/8000 [01:19<32:27, 3.02it/s, lr: 4.0e-05 loss: 1.487e-01] amber: 26%|██▋ | 2109/8000 [01:19<32:27, 3.02it/s, lr: 4.0e-05 loss: 9.912e-02] amber: 26%|██▋ | 2110/8000 [01:19<27:00, 3.63it/s, lr: 4.0e-05 loss: 9.912e-02] amber: 26%|██▋ | 2110/8000 [01:20<27:00, 3.63it/s, lr: 4.0e-05 loss: 8.016e-02] amber: 26%|██▋ | 2111/8000 [01:20<23:29, 4.18it/s, lr: 4.0e-05 loss: 8.016e-02] amber: 26%|██▋ | 2111/8000 [01:20<23:29, 4.18it/s, lr: 4.0e-05 loss: 1.252e-01] amber: 26%|██▋ | 2112/8000 [01:20<31:10, 3.15it/s, lr: 4.0e-05 loss: 1.252e-01] amber: 26%|██▋ | 2112/8000 [01:20<31:10, 3.15it/s, lr: 4.0e-05 loss: 8.604e-02] amber: 26%|██▋ | 2113/8000 [01:20<34:53, 2.81it/s, lr: 4.0e-05 loss: 8.604e-02] amber: 26%|██▋ | 2113/8000 [01:21<34:53, 2.81it/s, lr: 4.0e-05 loss: 8.397e-02] amber: 26%|██▋ | 2114/8000 [01:21<36:13, 2.71it/s, lr: 4.0e-05 loss: 8.397e-02] amber: 26%|██▋ | 2114/8000 [01:21<36:13, 2.71it/s, lr: 4.0e-05 loss: 1.158e-01] amber: 26%|██▋ | 2115/8000 [01:21<34:33, 2.84it/s, lr: 4.0e-05 loss: 1.158e-01] amber: 26%|██▋ | 2115/8000 [01:21<34:33, 2.84it/s, lr: 4.0e-05 loss: 8.874e-02] amber: 26%|██▋ | 2116/8000 [01:21<24:59, 3.92it/s, lr: 4.0e-05 loss: 8.874e-02] amber: 26%|██▋ | 2116/8000 [01:21<24:59, 3.92it/s, lr: 4.0e-05 loss: 1.056e-01] amber: 26%|██▋ | 2117/8000 [01:21<22:49, 4.30it/s, lr: 4.0e-05 loss: 1.056e-01] amber: 26%|██▋ | 2117/8000 [01:22<22:49, 4.30it/s, lr: 4.0e-05 loss: 8.279e-02] amber: 26%|██▋ | 2118/8000 [01:22<31:34, 3.11it/s, lr: 4.0e-05 loss: 8.279e-02] amber: 26%|██▋ | 2118/8000 [01:22<31:34, 3.11it/s, lr: 4.0e-05 loss: 8.825e-02] amber: 26%|██▋ | 2119/8000 [01:22<33:59, 2.88it/s, lr: 4.0e-05 loss: 8.825e-02] amber: 26%|██▋ | 2119/8000 [01:22<33:59, 2.88it/s, lr: 4.0e-05 loss: 1.504e-01] amber: 26%|██▋ | 2120/8000 [01:23<33:25, 2.93it/s, lr: 4.0e-05 loss: 1.504e-01] amber: 26%|██▋ | 2120/8000 [01:23<33:25, 2.93it/s, lr: 4.0e-05 loss: 7.506e-02]
Saving at step 2120
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002120.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:00<00:01, 1.14it/s]
Generating Images: 67%|██████▋ | 2/3 [00:01<00:00, 1.15it/s]
Generating Images: 100%|██████████| 3/3 [00:02<00:00, 1.15it/s]

amber: 27%|██▋ | 2121/8000 [01:24<33:25, 2.93it/s, lr: 4.0e-05 loss: 1.343e-01] amber: 27%|██▋ | 2122/8000 [01:24<39:10, 2.50it/s, lr: 4.0e-05 loss: 1.343e-01] amber: 27%|██▋ | 2122/8000 [01:24<39:10, 2.50it/s, lr: 4.0e-05 loss: 7.052e-02] amber: 27%|██▋ | 2123/8000 [01:24<37:28, 2.61it/s, lr: 4.0e-05 loss: 7.052e-02] amber: 27%|██▋ | 2123/8000 [01:24<37:28, 2.61it/s, lr: 4.0e-05 loss: 3.392e-01] amber: 27%|██▋ | 2124/8000 [01:24<27:26, 3.57it/s, lr: 4.0e-05 loss: 3.392e-01] amber: 27%|██▋ | 2124/8000 [01:24<27:26, 3.57it/s, lr: 4.0e-05 loss: 5.131e-03] amber: 27%|██▋ | 2125/8000 [01:24<23:02, 4.25it/s, lr: 4.0e-05 loss: 5.131e-03] amber: 27%|██▋ | 2125/8000 [01:25<23:02, 4.25it/s, lr: 4.0e-05 loss: 1.059e-01] amber: 27%|██▋ | 2126/8000 [01:25<20:47, 4.71it/s, lr: 4.0e-05 loss: 1.059e-01] amber: 27%|██▋ | 2126/8000 [01:25<20:47, 4.71it/s, lr: 4.0e-05 loss: 1.472e-01] amber: 27%|██▋ | 2127/8000 [01:25<17:03, 5.74it/s, lr: 4.0e-05 loss: 1.472e-01] amber: 27%|██▋ | 2127/8000 [01:25<17:03, 5.74it/s, lr: 4.0e-05 loss: 1.069e-01] amber: 27%|██▋ | 2128/8000 [01:25<27:55, 3.51it/s, lr: 4.0e-05 loss: 1.069e-01] amber: 27%|██▋ | 2128/8000 [01:25<27:55, 3.51it/s, lr: 4.0e-05 loss: 9.698e-02] amber: 27%|██▋ | 2129/8000 [01:25<19:18, 5.07it/s, lr: 4.0e-05 loss: 9.698e-02] amber: 27%|██▋ | 2129/8000 [01:26<19:18, 5.07it/s, lr: 4.0e-05 loss: 1.355e-01] amber: 27%|██▋ | 2130/8000 [01:26<17:00, 5.75it/s, lr: 4.0e-05 loss: 1.355e-01] amber: 27%|██▋ | 2130/8000 [01:26<17:00, 5.75it/s, lr: 4.0e-05 loss: 8.170e-02] amber: 27%|██▋ | 2131/8000 [01:26<37:57, 2.58it/s, lr: 4.0e-05 loss: 8.170e-02] amber: 27%|██▋ | 2131/8000 [01:26<37:57, 2.58it/s, lr: 4.0e-05 loss: 1.298e-01] amber: 27%|██▋ | 2132/8000 [01:26<25:05, 3.90it/s, lr: 4.0e-05 loss: 1.298e-01] amber: 27%|██▋ | 2132/8000 [01:26<25:05, 3.90it/s, lr: 4.0e-05 loss: 4.209e-02] amber: 27%|██▋ | 2133/8000 [01:26<28:12, 3.47it/s, lr: 4.0e-05 loss: 4.209e-02] amber: 27%|██▋ | 2133/8000 [01:27<28:12, 3.47it/s, lr: 4.0e-05 loss: 2.857e-02] amber: 27%|██▋ | 2134/8000 [01:27<32:54, 2.97it/s, lr: 4.0e-05 loss: 2.857e-02] amber: 27%|██▋ | 2134/8000 [01:27<32:54, 2.97it/s, lr: 4.0e-05 loss: 1.836e-01] amber: 27%|██▋ | 2135/8000 [01:27<20:16, 4.82it/s, lr: 4.0e-05 loss: 1.836e-01] amber: 27%|██▋ | 2135/8000 [01:27<20:16, 4.82it/s, lr: 4.0e-05 loss: 2.715e-02] amber: 27%|██▋ | 2136/8000 [01:27<30:06, 3.25it/s, lr: 4.0e-05 loss: 2.715e-02] amber: 27%|██▋ | 2136/8000 [01:28<30:06, 3.25it/s, lr: 4.0e-05 loss: 8.799e-02] amber: 27%|██▋ | 2137/8000 [01:28<21:23, 4.57it/s, lr: 4.0e-05 loss: 8.799e-02] amber: 27%|██▋ | 2137/8000 [01:28<21:23, 4.57it/s, lr: 4.0e-05 loss: 5.327e-02] amber: 27%|██▋ | 2138/8000 [01:28<29:26, 3.32it/s, lr: 4.0e-05 loss: 5.327e-02] amber: 27%|██▋ | 2138/8000 [01:28<29:26, 3.32it/s, lr: 4.0e-05 loss: 1.415e-01] amber: 27%|██▋ | 2139/8000 [01:28<21:51, 4.47it/s, lr: 4.0e-05 loss: 1.415e-01] amber: 27%|██▋ | 2139/8000 [01:28<21:51, 4.47it/s, lr: 4.0e-05 loss: 6.185e-02] amber: 27%|██▋ | 2140/8000 [01:28<20:38, 4.73it/s, lr: 4.0e-05 loss: 6.185e-02] amber: 27%|██▋ | 2140/8000 [01:29<20:38, 4.73it/s, lr: 4.0e-05 loss: 1.029e-02]
Saving at step 2140
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002140.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 27%|██▋ | 2141/8000 [01:29<20:37, 4.73it/s, lr: 4.0e-05 loss: 9.400e-03] amber: 27%|██▋ | 2142/8000 [01:29<25:16, 3.86it/s, lr: 4.0e-05 loss: 9.400e-03] amber: 27%|██▋ | 2142/8000 [01:29<25:16, 3.86it/s, lr: 4.0e-05 loss: 8.218e-02] amber: 27%|██▋ | 2143/8000 [01:29<29:33, 3.30it/s, lr: 4.0e-05 loss: 8.218e-02] amber: 27%|██▋ | 2143/8000 [01:30<29:33, 3.30it/s, lr: 4.0e-05 loss: 2.341e-02] amber: 27%|██▋ | 2144/8000 [01:30<31:31, 3.10it/s, lr: 4.0e-05 loss: 2.341e-02] amber: 27%|██▋ | 2144/8000 [01:30<31:31, 3.10it/s, lr: 4.0e-05 loss: 2.357e-01] amber: 27%|██▋ | 2145/8000 [01:30<33:01, 2.95it/s, lr: 4.0e-05 loss: 2.357e-01] amber: 27%|██▋ | 2145/8000 [01:30<33:01, 2.95it/s, lr: 4.0e-05 loss: 8.780e-02] amber: 27%|██▋ | 2146/8000 [01:30<21:29, 4.54it/s, lr: 4.0e-05 loss: 8.780e-02] amber: 27%|██▋ | 2146/8000 [01:31<21:29, 4.54it/s, lr: 4.0e-05 loss: 1.411e-01] amber: 27%|██▋ | 2147/8000 [01:31<31:11, 3.13it/s, lr: 4.0e-05 loss: 1.411e-01] amber: 27%|██▋ | 2147/8000 [01:31<31:11, 3.13it/s, lr: 4.0e-05 loss: 2.480e-01] amber: 27%|██▋ | 2148/8000 [01:31<34:51, 2.80it/s, lr: 4.0e-05 loss: 2.480e-01] amber: 27%|██▋ | 2148/8000 [01:31<34:51, 2.80it/s, lr: 4.0e-05 loss: 3.107e-02] amber: 27%|██▋ | 2149/8000 [01:31<23:48, 4.10it/s, lr: 4.0e-05 loss: 3.107e-02] amber: 27%|██▋ | 2149/8000 [01:31<23:48, 4.10it/s, lr: 4.0e-05 loss: 2.218e-01] amber: 27%|██▋ | 2150/8000 [01:31<30:20, 3.21it/s, lr: 4.0e-05 loss: 2.218e-01] amber: 27%|██▋ | 2150/8000 [01:32<30:20, 3.21it/s, lr: 4.0e-05 loss: 8.544e-02] amber: 27%|██▋ | 2151/8000 [01:32<33:51, 2.88it/s, lr: 4.0e-05 loss: 8.544e-02] amber: 27%|██▋ | 2151/8000 [01:32<33:51, 2.88it/s, lr: 4.0e-05 loss: 5.861e-02] amber: 27%|██▋ | 2152/8000 [01:32<23:35, 4.13it/s, lr: 4.0e-05 loss: 5.861e-02] amber: 27%|██▋ | 2152/8000 [01:32<23:35, 4.13it/s, lr: 4.0e-05 loss: 1.691e-01] amber: 27%|██▋ | 2153/8000 [01:32<20:22, 4.78it/s, lr: 4.0e-05 loss: 1.691e-01] amber: 27%|██▋ | 2153/8000 [01:32<20:22, 4.78it/s, lr: 4.0e-05 loss: 1.743e-01] amber: 27%|██▋ | 2154/8000 [01:32<20:09, 4.84it/s, lr: 4.0e-05 loss: 1.743e-01] amber: 27%|██▋ | 2154/8000 [01:33<20:09, 4.84it/s, lr: 4.0e-05 loss: 1.293e-01] amber: 27%|██▋ | 2155/8000 [01:33<29:21, 3.32it/s, lr: 4.0e-05 loss: 1.293e-01] amber: 27%|██▋ | 2155/8000 [01:33<29:21, 3.32it/s, lr: 4.0e-05 loss: 1.113e-01] amber: 27%|██▋ | 2156/8000 [01:33<32:46, 2.97it/s, lr: 4.0e-05 loss: 1.113e-01] amber: 27%|██▋ | 2156/8000 [01:33<32:46, 2.97it/s, lr: 4.0e-05 loss: 6.117e-02] amber: 27%|██▋ | 2157/8000 [01:33<33:05, 2.94it/s, lr: 4.0e-05 loss: 6.117e-02] amber: 27%|██▋ | 2157/8000 [01:34<33:05, 2.94it/s, lr: 4.0e-05 loss: 9.228e-02] amber: 27%|██▋ | 2158/8000 [01:34<25:36, 3.80it/s, lr: 4.0e-05 loss: 9.228e-02] amber: 27%|██▋ | 2158/8000 [01:34<25:36, 3.80it/s, lr: 4.0e-05 loss: 1.899e-01] amber: 27%|██▋ | 2159/8000 [01:34<19:39, 4.95it/s, lr: 4.0e-05 loss: 1.899e-01] amber: 27%|██▋ | 2159/8000 [01:34<19:39, 4.95it/s, lr: 4.0e-05 loss: 2.771e-02] amber: 27%|██▋ | 2160/8000 [01:34<31:12, 3.12it/s, lr: 4.0e-05 loss: 2.771e-02] amber: 27%|██▋ | 2160/8000 [01:34<31:12, 3.12it/s, lr: 4.0e-05 loss: 2.430e-01]
Saving at step 2160
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002160.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
Generating Images: 0%| | 0/3 [00:00<?, ?it/s]
Generating Images: 33%|███▎ | 1/3 [00:00<00:01, 1.10it/s]
Generating Images: 67%|██████▋ | 2/3 [00:01<00:00, 1.12it/s]
Generating Images: 100%|██████████| 3/3 [00:02<00:00, 1.13it/s]

amber: 27%|██▋ | 2161/8000 [01:35<31:12, 3.12it/s, lr: 4.0e-05 loss: 1.057e-01] amber: 27%|██▋ | 2162/8000 [01:35<26:13, 3.71it/s, lr: 4.0e-05 loss: 1.057e-01] amber: 27%|██▋ | 2162/8000 [01:35<26:13, 3.71it/s, lr: 4.0e-05 loss: 2.019e-01] amber: 27%|██▋ | 2163/8000 [01:35<29:57, 3.25it/s, lr: 4.0e-05 loss: 2.019e-01] amber: 27%|██▋ | 2163/8000 [01:36<29:57, 3.25it/s, lr: 4.0e-05 loss: 7.531e-02] amber: 27%|██▋ | 2164/8000 [01:36<34:14, 2.84it/s, lr: 4.0e-05 loss: 7.531e-02] amber: 27%|██▋ | 2164/8000 [01:36<34:14, 2.84it/s, lr: 4.0e-05 loss: 1.506e-01] amber: 27%|██▋ | 2165/8000 [01:36<22:30, 4.32it/s, lr: 4.0e-05 loss: 1.506e-01] amber: 27%|██▋ | 2165/8000 [01:36<22:30, 4.32it/s, lr: 4.0e-05 loss: 6.016e-02] amber: 27%|██▋ | 2166/8000 [01:36<18:53, 5.15it/s, lr: 4.0e-05 loss: 6.016e-02] amber: 27%|██▋ | 2166/8000 [01:36<18:53, 5.15it/s, lr: 4.0e-05 loss: 3.239e-02] amber: 27%|██▋ | 2167/8000 [01:36<19:21, 5.02it/s, lr: 4.0e-05 loss: 3.239e-02] amber: 27%|██▋ | 2167/8000 [01:37<19:21, 5.02it/s, lr: 4.0e-05 loss: 1.573e-01] amber: 27%|██▋ | 2168/8000 [01:37<29:06, 3.34it/s, lr: 4.0e-05 loss: 1.573e-01] amber: 27%|██▋ | 2168/8000 [01:37<29:06, 3.34it/s, lr: 4.0e-05 loss: 5.499e-02] amber: 27%|██▋ | 2169/8000 [01:37<20:01, 4.85it/s, lr: 4.0e-05 loss: 5.499e-02] amber: 27%|██▋ | 2169/8000 [01:37<20:01, 4.85it/s, lr: 4.0e-05 loss: 1.203e-01] amber: 27%|██▋ | 2170/8000 [01:37<30:07, 3.23it/s, lr: 4.0e-05 loss: 1.203e-01] amber: 27%|██▋ | 2170/8000 [01:38<30:07, 3.23it/s, lr: 4.0e-05 loss: 1.448e-01] amber: 27%|██▋ | 2171/8000 [01:38<45:54, 2.12it/s, lr: 4.0e-05 loss: 1.448e-01] amber: 27%|██▋ | 2171/8000 [01:38<45:54, 2.12it/s, lr: 4.0e-05 loss: 1.333e-01] amber: 27%|██▋ | 2172/8000 [01:38<37:32, 2.59it/s, lr: 4.0e-05 loss: 1.333e-01] amber: 27%|██▋ | 2172/8000 [01:38<37:32, 2.59it/s, lr: 4.0e-05 loss: 1.534e-01] amber: 27%|██▋ | 2173/8000 [01:38<22:29, 4.32it/s, lr: 4.0e-05 loss: 1.534e-01] amber: 27%|██▋ | 2173/8000 [01:38<22:29, 4.32it/s, lr: 4.0e-05 loss: 1.432e-01] amber: 27%|██▋ | 2174/8000 [01:38<19:58, 4.86it/s, lr: 4.0e-05 loss: 1.432e-01] amber: 27%|██▋ | 2174/8000 [01:39<19:58, 4.86it/s, lr: 4.0e-05 loss: 1.751e-01] amber: 27%|██▋ | 2175/8000 [01:39<22:47, 4.26it/s, lr: 4.0e-05 loss: 1.751e-01] amber: 27%|██▋ | 2175/8000 [01:39<22:47, 4.26it/s, lr: 4.0e-05 loss: 2.378e-01] amber: 27%|██▋ | 2176/8000 [01:39<29:48, 3.26it/s, lr: 4.0e-05 loss: 2.378e-01] amber: 27%|██▋ | 2176/8000 [01:39<29:48, 3.26it/s, lr: 4.0e-05 loss: 1.577e-01] amber: 27%|██▋ | 2177/8000 [01:39<22:46, 4.26it/s, lr: 4.0e-05 loss: 1.577e-01] amber: 27%|██▋ | 2177/8000 [01:39<22:46, 4.26it/s, lr: 4.0e-05 loss: 5.564e-02] amber: 27%|██▋ | 2178/8000 [01:39<19:55, 4.87it/s, lr: 4.0e-05 loss: 5.564e-02] amber: 27%|██▋ | 2178/8000 [01:40<19:55, 4.87it/s, lr: 4.0e-05 loss: 9.955e-02] amber: 27%|██▋ | 2179/8000 [01:40<29:09, 3.33it/s, lr: 4.0e-05 loss: 9.955e-02] amber: 27%|██▋ | 2179/8000 [01:40<29:09, 3.33it/s, lr: 4.0e-05 loss: 5.410e-02] amber: 27%|██▋ | 2180/8000 [01:40<31:40, 3.06it/s, lr: 4.0e-05 loss: 5.410e-02] amber: 27%|██▋ | 2180/8000 [01:40<31:40, 3.06it/s, lr: 4.0e-05 loss: 9.116e-02]
Saving at step 2180
Saved checkpoint to /app/ai-toolkit/output/amber/amber_000002180.safetensors
Saved optimizer to /app/ai-toolkit/output/amber/optimizer.pt
amber: 27%|██▋ | 2181/8000 [01:41<31:40, 3.06it/s, lr: 4.0e-05 loss: 2.897e-02] amber: 27%|██▋ | 2182/8000 [01:41<33:46, 2.87it/s, lr: 4.0e-05 loss: 2.897e-02] amber: 27%|██▋ | 2182/8000 [01:41<33:46, 2.87it/s, lr: 4.0e-05 loss: 2.612e-02] amber: 27%|██▋ | 2183/8000 [01:41<24:03, 4.03it/s, lr: 4.0e-05 loss: 2.612e-02] amber: 27%|██▋ | 2183/8000 [01:41<24:03, 4.03it/s, lr: 4.0e-05 loss: 6.114e-02] amber: 27%|██▋ | 2184/8000 [01:41<30:12, 3.21it/s, lr: 4.0e-05 loss: 6.114e-02] amber: 27%|██▋ | 2184/8000 [01:42<30:12, 3.21it/s, lr: 4.0e-05 loss: 1.407e-01] amber: 27%|██▋ | 2185/8000 [01:42<19:34, 4.95it/s, lr: 4.0e-05 loss: 1.407e-01] amber: 27%|██▋ | 2185/8000 [01:42<19:34, 4.95it/s, lr: 4.0e-05 loss: 1.398e-01] amber: 27%|██▋ | 2186/8000 [01:42<28:44, 3.37it/s, lr: 4.0e-05 loss: 1.398e-01] amber: 27%|██▋ | 2186/8000 [01:42<28:44, 3.37it/s, lr: 4.0e-05 loss: 7.118e-02] amber: 27%|██▋ | 2187/8000 [01:42<31:33, 3.07it/s, lr: 4.0e-05 loss: 7.118e-02] amber: 27%|██▋ | 2187/8000 [01:43<31:33, 3.07it/s, lr: 4.0e-05 loss: 1.577e-01] amber: 27%|██▋ | 2188/8000 [01:43<33:14, 2.91it/s, lr: 4.0e-05 loss: 1.577e-01] amber: 27%|██▋ | 2188/8000 [01:43<33:14, 2.91it/s, lr: 4.0e-05 loss: 8.114e-02] amber: 27%|██▋ | 2189/8000 [01:43<32:50, 2.95it/s, lr: 4.0e-05 loss: 8.114e-02] amber: 27%|██▋ | 2189/8000 [01:43<32:50, 2.95it/s, lr: 4.0e-05 loss: 2.866e-03] amber: 27%|██▋ | 2190/8000 [01:43<23:35, 4.10it/s, lr: 4.0e-05 loss: 2.866e-03] amber: 27%|██▋ | 2190/8000 [01:44<23:35, 4.10it/s, lr: 4.0e-05 loss: 1.911e-01] amber: 27%|██▋ | 2191/8000 [01:44<31:25, 3.08it/s, lr: 4.0e-05 loss: 1.911e-01] amber: 27%|██▋ | 2191/8000 [01:44<31:25, 3.08it/s, lr: 4.0e-05 loss: 1.530e-01] amber: 27%|██▋ | 2192/8000 [01:44<20:42, 4.67it/s, lr: 4.0e-05 loss: 1.530e-01] amber: 27%|██▋ | 2192/8000 [01:44<20:42, 4.67it/s, lr: 4.0e-05 loss: 2.019e-01] amber: 27%|██▋ | 2193/8000 [01:44<30:23, 3.19it/s, lr: 4.0e-05 loss: 2.019e-01] amber: 27%|██▋ | 2193/8000 [01:44<30:23, 3.19it/s, lr: 4.0e-05 loss: 6.848e-02] amber: 27%|██▋ | 2194/8000 [01:44<23:04, 4.19it/s, lr: 4.0e-05 loss: 6.848e-02] amber: 27%|██▋ | 2194/8000 [01:45<23:04, 4.19it/s, lr: 4.0e-05 loss: 2.154e-02] amber: 27%|██▋ | 2195/8000 [01:45<28:23, 3.41it/s, lr: 4.0e-05 loss: 2.154e-02] amber: 27%|██▋ | 2195/8000 [01:45<28:23, 3.41it/s, lr: 4.0e-05 loss: 9.233e-02] amber: 27%|██▋ | 2196/8000 [01:45<32:22, 2.99it/s, lr: 4.0e-05 loss: 9.233e-02] amber: 27%|██▋ | 2196/8000 [01:45<32:22, 2.99it/s, lr: 4.0e-05 loss: 7.153e-03] amber: 27%|██▋ | 2197/8000 [01:45<21:56, 4.41it/s, lr: 4.0e-05 loss: 7.153e-03]
========================================
Job stopped
========================================

Xet Storage Details

Size:
75.1 kB
·
Xet hash:
0739a111959522a7af2333c6293819798f7a2065a9db55d31cb6b368598cf0e9

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.