| 22:14:28 | INFO | ============================================================ |
| 22:14:28 | INFO | Starting LoRA training for style 'pattachitra' |
| 22:14:28 | INFO | ============================================================ |
| 22:14:28 | INFO | Config: rank=32, lr=0.0001, steps=800, dataset=/home/nvidia/25BCE5746/indic-heritage-studio-v2/assets/datasets/pattachitra |
| 22:14:28 | INFO | Mixed precision: bf16 (dtype=torch.bfloat16) |
| 22:14:28 | INFO | Device: cuda (NVIDIA A100-SXM4-80GB) |
| 22:14:28 | INFO | Loading SDXL components... |
| 22:14:30 | INFO | Loading tokenizers... |
| 22:14:32 | INFO | Loading text encoders... |
| 22:15:56 | INFO | Loading VAE... |
| 22:16:01 | INFO | Loading UNet... |
| 22:16:03 | INFO | Injecting LoRA adapters (rank=32)... |
| 22:16:09 | INFO | Building dataset... |
| 22:16:09 | INFO | Dataset at /home/nvidia/25BCE5746/indic-heritage-studio-v2/assets/datasets/pattachitra: 40 images |
| 22:16:09 | INFO | Setting up optimizer... |
| 22:16:09 | INFO | Starting training loop: 800 steps |
| trainable params: 47,841,280 || all params: 2,615,304,964 || trainable%: 1.8293 |
| 22:16:14 | INFO | Step 1/800 loss=0.3038 lr=0.00e+00 |
| 22:16:26 | INFO | Step 20/800 loss=0.0620 lr=2.50e-06 |
| 22:16:36 | INFO | Step 40/800 loss=0.2460 lr=5.00e-06 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:16:48 | INFO | Step 60/800 loss=0.2788 lr=7.50e-06 |
| 22:16:58 | INFO | Step 80/800 loss=0.3875 lr=1.00e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:17:11 | INFO | Step 100/800 loss=0.1231 lr=1.25e-05 |
| 22:17:25 | INFO | Step 120/800 loss=0.0736 lr=1.50e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:17:37 | INFO | Step 140/800 loss=0.2712 lr=1.75e-05 |
| 22:17:48 | INFO | Step 160/800 loss=0.0240 lr=2.00e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:17:59 | INFO | Step 180/800 loss=0.4110 lr=2.25e-05 |
| 22:18:10 | INFO | Step 200/800 loss=0.0848 lr=2.50e-05 |
| 22:18:10 | INFO | 💾 Checkpoint saved: /home/nvidia/25BCE5746/indic-heritage-studio-v2/outputs/lora_training/pattachitra/step_200.safetensors |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:18:22 | INFO | Step 220/800 loss=0.0823 lr=2.75e-05 |
| 22:18:33 | INFO | Step 240/800 loss=0.4107 lr=3.00e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:18:44 | INFO | Step 260/800 loss=0.2020 lr=3.25e-05 |
| 22:18:55 | INFO | Step 280/800 loss=0.0067 lr=3.50e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:19:06 | INFO | Step 300/800 loss=0.1741 lr=3.75e-05 |
| 22:19:17 | INFO | Step 320/800 loss=0.1657 lr=4.00e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:19:28 | INFO | Step 340/800 loss=0.0288 lr=4.25e-05 |
| 22:19:39 | INFO | Step 360/800 loss=0.0079 lr=4.50e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:19:51 | INFO | Step 380/800 loss=0.0827 lr=4.75e-05 |
| 22:20:02 | INFO | Step 400/800 loss=0.0278 lr=5.00e-05 |
| 22:20:02 | INFO | 💾 Checkpoint saved: /home/nvidia/25BCE5746/indic-heritage-studio-v2/outputs/lora_training/pattachitra/step_400.safetensors |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:20:14 | INFO | Step 420/800 loss=0.1806 lr=5.25e-05 |
| 22:20:25 | INFO | Step 440/800 loss=0.3304 lr=5.50e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:20:36 | INFO | Step 460/800 loss=0.4454 lr=5.75e-05 |
| 22:20:47 | INFO | Step 480/800 loss=0.0649 lr=6.00e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:20:58 | INFO | Step 500/800 loss=0.3032 lr=6.25e-05 |
| 22:21:09 | INFO | Step 520/800 loss=0.2066 lr=6.50e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:21:20 | INFO | Step 540/800 loss=0.0157 lr=6.75e-05 |
| 22:21:32 | INFO | Step 560/800 loss=0.0680 lr=7.00e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:21:43 | INFO | Step 580/800 loss=0.0122 lr=7.25e-05 |
| 22:21:54 | INFO | Step 600/800 loss=0.3531 lr=7.50e-05 |
| 22:21:54 | INFO | 💾 Checkpoint saved: /home/nvidia/25BCE5746/indic-heritage-studio-v2/outputs/lora_training/pattachitra/step_600.safetensors |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:22:06 | INFO | Step 620/800 loss=0.2386 lr=7.75e-05 |
| 22:22:17 | INFO | Step 640/800 loss=0.0031 lr=8.00e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:22:28 | INFO | Step 660/800 loss=0.0228 lr=8.25e-05 |
| 22:22:39 | INFO | Step 680/800 loss=0.0370 lr=8.50e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:22:50 | INFO | Step 700/800 loss=0.0093 lr=8.75e-05 |
| 22:23:02 | INFO | Step 720/800 loss=0.2647 lr=9.00e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:23:14 | INFO | Step 740/800 loss=0.2184 lr=9.25e-05 |
| 22:23:25 | INFO | Step 760/800 loss=0.2230 lr=9.50e-05 |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| 22:23:37 | INFO | Step 780/800 loss=0.0045 lr=9.75e-05 |
| 22:23:47 | INFO | Step 800/800 loss=0.2005 lr=1.00e-04 |
| 22:23:48 | INFO | 💾 Checkpoint saved: /home/nvidia/25BCE5746/indic-heritage-studio-v2/outputs/lora_training/pattachitra/step_800.safetensors |
| 22:23:48 | INFO | Final LoRA saved: /home/nvidia/25BCE5746/indic-heritage-studio-v2/assets/loras/pattachitra.safetensors (91.4 MB) |
| 22:23:48 | INFO | ============================================================ |
| 22:23:48 | INFO | LoRA training complete! Saved to: /home/nvidia/25BCE5746/indic-heritage-studio-v2/assets/loras/pattachitra.safetensors |
| 22:23:48 | INFO | ============================================================ |
|
|
| ✅ Done. LoRA saved to: /home/nvidia/25BCE5746/indic-heritage-studio-v2/assets/loras/pattachitra.safetensors |
|
|
