End of training

.gitattributes

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+canola_12_1.png filter=lfs diff=lfs merge=lfs -text
+canola_12_2.png filter=lfs diff=lfs merge=lfs -text
+canola_12_35_1.png filter=lfs diff=lfs merge=lfs -text
+canola_12_3_1.png filter=lfs diff=lfs merge=lfs -text
+canola_12_45_1.png filter=lfs diff=lfs merge=lfs -text
+canola_12_4_1.png filter=lfs diff=lfs merge=lfs -text
+canola_12_5_1.png filter=lfs diff=lfs merge=lfs -text
+canola_12_8_1.png filter=lfs diff=lfs merge=lfs -text
+image_0.png filter=lfs diff=lfs merge=lfs -text
+image_1.png filter=lfs diff=lfs merge=lfs -text
+image_2.png filter=lfs diff=lfs merge=lfs -text
+image_3.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,99 @@
+---
+base_model: stabilityai/stable-diffusion-3-medium-diffusers
+library_name: diffusers
+license: other
+instance_prompt: a photo of a 17 day old canola plant with different sized leaves.
+  It is growing in a bright blue cup
+widget:
+- text: A photo of a 17 day old canola plant with different sized leaves. The leaves
+    are ruffled around the edges and spread beyond the cup. It is growing in nutrient-rich
+    soil in a smooth, bright blue cup on a bright blue background
+  output:
+    url: image_0.png
+- text: A photo of a 17 day old canola plant with different sized leaves. The leaves
+    are ruffled around the edges and spread beyond the cup. It is growing in nutrient-rich
+    soil in a smooth, bright blue cup on a bright blue background
+  output:
+    url: image_1.png
+- text: A photo of a 17 day old canola plant with different sized leaves. The leaves
+    are ruffled around the edges and spread beyond the cup. It is growing in nutrient-rich
+    soil in a smooth, bright blue cup on a bright blue background
+  output:
+    url: image_2.png
+- text: A photo of a 17 day old canola plant with different sized leaves. The leaves
+    are ruffled around the edges and spread beyond the cup. It is growing in nutrient-rich
+    soil in a smooth, bright blue cup on a bright blue background
+  output:
+    url: image_3.png
+tags:
+- text-to-image
+- diffusers-training
+- diffusers
+- lora
+- template:sd-lora
+- sd3
+- sd3-diffusers
+---
+
+<!-- This model card has been generated automatically according to the information the training script had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+
+# SD3 DreamBooth LoRA - rdeinla/test-can-1-2
+
+<Gallery />
+
+## Model description
+
+These are rdeinla/test-can-1-2 DreamBooth LoRA weights for stabilityai/stable-diffusion-3-medium-diffusers.
+
+The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [SD3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_sd3.md).
+
+Was LoRA for the text encoder enabled? False.
+
+## Trigger words
+
+You should use `a photo of a 17 day old canola plant with different sized leaves. It is growing in a bright blue cup` to trigger the image generation.
+
+## Download model
+
+[Download the *.safetensors LoRA](rdeinla/test-can-1-2/tree/main) in the Files & versions tab.
+
+## Use it with the [🧨 diffusers library](https://github.com/huggingface/diffusers)
+
+```py
+from diffusers import AutoPipelineForText2Image
+import torch
+pipeline = AutoPipelineForText2Image.from_pretrained(stabilityai/stable-diffusion-3-medium-diffusers, torch_dtype=torch.float16).to('cuda')
+pipeline.load_lora_weights('rdeinla/test-can-1-2', weight_name='pytorch_lora_weights.safetensors')
+image = pipeline('A photo of a 17 day old canola plant with different sized leaves. The leaves are ruffled around the edges and spread beyond the cup. It is growing in nutrient-rich soil in a smooth, bright blue cup on a bright blue background').images[0]
+```
+
+### Use it with UIs such as AUTOMATIC1111, Comfy UI, SD.Next, Invoke
+
+- **LoRA**: download **[`diffusers_lora_weights.safetensors` here 💾](/rdeinla/test-can-1-2/blob/main/diffusers_lora_weights.safetensors)**.
+    - Rename it and place it on your `models/Lora` folder.
+    - On AUTOMATIC1111, load the LoRA by adding `<lora:your_new_name:1>` to your prompt. On ComfyUI just [load it as a regular LoRA](https://comfyanonymous.github.io/ComfyUI_examples/lora/).
+
+For more details, including weighting, merging and fusing LoRAs, check the [documentation on loading LoRAs in diffusers](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters)
+
+## License
+
+Please adhere to the licensing terms as described [here](https://huggingface.co/stabilityai/stable-diffusion-3-medium/blob/main/LICENSE.md).
+
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+# TODO: add an example code snippet for running this diffusion pipeline
+```
+
+#### Limitations and bias
+
+[TODO: provide examples of latent issues and potential remediations]
+
+## Training details
+
+[TODO: describe the data used to train the model]

can_12_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:747b15dd1bf4bfb7ea45f6f0c02c7f746b72b81db1a75a85f27084fa01c0ca3e
+size 4743176

checkpoint-500/optimizer.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46a5f791112faa751b6896e5107ae6101da510fc68ed84fc4ff4739720a25671
+size 37872780

checkpoint-500/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4ae7e43f457f19299682b7c3f36340059d24ae9d9121659ec10b3a1e4735ea1
+size 18825720

checkpoint-500/random_states_0.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:afa4f0a4cd1713781e762818debbf68e357584e72ea936f1bff86a9d51ef21ab
+size 14864

checkpoint-500/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18b984273ea2d45b7ffb1d047bb359d93111e41fcad70d16a1b453fd38f72636
+size 988

checkpoint-500/scheduler.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e022c236c0a83a2ba601318153b5b4e6d5340e39a69ab58d42f304efbfb68a9d
+size 1000

inference.py

@@ -0,0 +1,16 @@
+import torch
+from diffusers import StableDiffusion3Pipeline
+from diffusers import AutoPipelineForText2Image
+from safetensors.torch import load_file
+
+torch.backends.cuda.enable_mem_efficient_sdp(False)
+torch.backends.cuda.enable_flash_sdp(False)
+
+pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-3-medium-diffusers", torch_dtype=torch.float16).to('cuda')
+
+pipeline.load_lora_weights("rdeinla/test-can-1-2", weight_name='pytorch_lora_weights.safetensors')
+
+image = pipeline("A photo of a young canola plant with medium-sized leaves. The leaves are ruffled around the edges and have prominent ridges. It is about 19 days old. It is growing in dark, nutrient-rich soil. It is contained within a smooth, bright blue cylindrical cup on a bright blue background",
+guidance_scale = 5.0, negative_prompt = "blurry").images[0]
+
+image.save("canola_12_5_1.png")

logs/dreambooth-sd3-lora/1743823161.0409586/events.out.tfevents.1743823161.g338.1131650.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7b580a6b11ab72f28851627e17b8fab0251be4dcc56bb7e0c3b539b327914bb
+size 3778

logs/dreambooth-sd3-lora/1743823161.045078/hparams.yml

@@ -0,0 +1,77 @@
+adam_beta1: 0.9
+adam_beta2: 0.999
+adam_epsilon: 1.0e-08
+adam_weight_decay: 0.0001
+adam_weight_decay_text_encoder: 0.001
+allow_tf32: false
+cache_dir: null
+cache_latents: false
+caption_column: null
+center_crop: false
+checkpointing_steps: 500
+checkpoints_total_limit: null
+class_data_dir: null
+class_prompt: null
+dataloader_num_workers: 0
+dataset_config_name: null
+dataset_name: null
+gradient_accumulation_steps: 1
+gradient_checkpointing: false
+hub_model_id: null
+hub_token: null
+image_column: image
+instance_data_dir: /project/6085367/SharedData/diffusers/examples/dreambooth/Canola/10to19days/10to19-train
+instance_prompt: a photo of a 17 day old canola plant with different sized leaves.
+  It is growing in a bright blue cup
+learning_rate: 0.00045
+local_rank: 0
+logging_dir: logs
+logit_mean: 0.0
+logit_std: 1.0
+lora_blocks: null
+lora_layers: null
+lr_num_cycles: 1
+lr_power: 1.0
+lr_scheduler: constant
+lr_warmup_steps: 100
+max_grad_norm: 1.0
+max_sequence_length: 77
+max_train_steps: 850
+mixed_precision: fp16
+mode_scale: 1.29
+num_class_images: 100
+num_train_epochs: 425
+num_validation_images: 4
+optimizer: AdamW
+output_dir: /project/6085367/SharedData/diffusers/examples/dreambooth/trained-sd3-outputs/test-can-1-2
+precondition_outputs: 1
+pretrained_model_name_or_path: stabilityai/stable-diffusion-3-medium-diffusers
+prior_generation_precision: null
+prior_loss_weight: 1.0
+prodigy_beta3: null
+prodigy_decouple: true
+prodigy_safeguard_warmup: true
+prodigy_use_bias_correction: true
+push_to_hub: true
+random_flip: false
+rank: 8
+repeats: 1
+report_to: tensorboard
+resolution: 768
+resume_from_checkpoint: null
+revision: null
+sample_batch_size: 4
+scale_lr: false
+seed: 42
+text_encoder_lr: 5.0e-06
+train_batch_size: 2
+train_text_encoder: false
+upcast_before_saving: false
+use_8bit_adam: false
+validation_epochs: 4
+validation_prompt: A photo of a 17 day old canola plant with different sized leaves.
+  The leaves are ruffled around the edges and spread beyond the cup. It is growing
+  in nutrient-rich soil in a smooth, bright blue cup on a bright blue background
+variant: null
+weighting_scheme: logit_normal
+with_prior_preservation: false

logs/dreambooth-sd3-lora/events.out.tfevents.1743823161.g338.1131650.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c6461e18e1371c6d4e0f1f951418410018bc5f77f972160d802e103462feb5e
+size 468230219

pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:376fb497b2d9d0e3772133764cf114e2284a6c2de53fcd88dad203d8df22cd43
+size 9437288

run-inference.sh

@@ -0,0 +1,18 @@
+#!/bin/sh
+
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=4
+#SBATCH --cpus-per-task=4
+#SBATCH --mem=15GB
+#SBATCH --time=0-10:00
+#SBATCH --partition=livi
+#SBATCH --mail-user=deinlar@myumanitoba.ca
+#SBATCH --mail-type=END
+#SBATCH --gpus=1
+
+module load cuda/12.4.1  arch/avx2  gcc/13.2.0
+module load python/3.11.11
+
+source /project/6085367/SharedData/my_newvenv/bin/activate
+
+python3 inference.py>log.txt

slurm-6020659.out

@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
  0%|          | 0/28 [00:00<?, ?it/s]
  4%|▎         | 1/28 [00:02<00:56,  2.08s/it]
  7%|▋         | 2/28 [00:03<00:39,  1.52s/it]
 11%|█         | 3/28 [00:05<00:42,  1.71s/it]
 14%|█▍        | 4/28 [00:07<00:43,  1.80s/it]
 18%|█▊        | 5/28 [00:09<00:42,  1.85s/it]
 21%|██▏       | 6/28 [00:10<00:41,  1.88s/it]
 25%|██▌       | 7/28 [00:12<00:39,  1.90s/it]
 29%|██▊       | 8/28 [00:14<00:38,  1.91s/it]
 32%|███▏      | 9/28 [00:16<00:36,  1.92s/it]
 36%|███▌      | 10/28 [00:18<00:34,  1.92s/it]
 39%|███▉      | 11/28 [00:20<00:32,  1.93s/it]
 43%|████▎     | 12/28 [00:22<00:30,  1.93s/it]
 46%|████▋     | 13/28 [00:24<00:28,  1.93s/it]
 50%|█████     | 14/28 [00:26<00:27,  1.93s/it]
 54%|█████▎    | 15/28 [00:28<00:25,  1.94s/it]
 57%|█████▋    | 16/28 [00:30<00:23,  1.94s/it]
 61%|██████    | 17/28 [00:32<00:21,  1.94s/it]
 64%|██████▍   | 18/28 [00:34<00:19,  1.93s/it]
 68%|██████▊   | 19/28 [00:36<00:17,  1.94s/it]
 71%|███████▏  | 20/28 [00:38<00:15,  1.94s/it]
 75%|███████▌  | 21/28 [00:40<00:13,  1.94s/it]
 79%|███████▊  | 22/28 [00:41<00:11,  1.94s/it]
 82%|████████▏ | 23/28 [00:43<00:09,  1.94s/it]
 86%|████████▌ | 24/28 [00:45<00:07,  1.94s/it]
 89%|████████▉ | 25/28 [00:47<00:05,  1.94s/it]
 93%|█████████▎| 26/28 [00:49<00:03,  1.95s/it]
 96%|█████████▋| 27/28 [00:51<00:01,  1.95s/it]

slurm-6020660.out

@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
  0%|          | 0/28 [00:00<?, ?it/s]
  4%|▎         | 1/28 [00:02<00:56,  2.08s/it]
  7%|▋         | 2/28 [00:03<00:39,  1.53s/it]
 11%|█         | 3/28 [00:05<00:42,  1.71s/it]
 14%|█▍        | 4/28 [00:07<00:43,  1.80s/it]
 18%|█▊        | 5/28 [00:09<00:42,  1.85s/it]
 21%|██▏       | 6/28 [00:10<00:41,  1.88s/it]
 25%|██▌       | 7/28 [00:12<00:39,  1.90s/it]
 29%|██▊       | 8/28 [00:14<00:38,  1.92s/it]
 32%|███▏      | 9/28 [00:16<00:36,  1.92s/it]
 36%|███▌      | 10/28 [00:18<00:34,  1.93s/it]
 39%|███▉      | 11/28 [00:20<00:32,  1.93s/it]
 43%|████▎     | 12/28 [00:22<00:30,  1.93s/it]
 46%|████▋     | 13/28 [00:24<00:29,  1.94s/it]
 50%|█████     | 14/28 [00:26<00:27,  1.94s/it]
 54%|█████▎    | 15/28 [00:28<00:25,  1.94s/it]
 57%|█████▋    | 16/28 [00:30<00:23,  1.94s/it]
 61%|██████    | 17/28 [00:32<00:21,  1.94s/it]
 64%|██████▍   | 18/28 [00:34<00:19,  1.94s/it]
 68%|██████▊   | 19/28 [00:36<00:17,  1.95s/it]
 71%|███████▏  | 20/28 [00:38<00:15,  1.94s/it]
 75%|███████▌  | 21/28 [00:40<00:13,  1.94s/it]
 79%|███████▊  | 22/28 [00:42<00:11,  1.94s/it]
 82%|████████▏ | 23/28 [00:44<00:09,  1.95s/it]
 86%|████████▌ | 24/28 [00:45<00:07,  1.94s/it]
 89%|████████▉ | 25/28 [00:47<00:05,  1.94s/it]
 93%|█████████▎| 26/28 [00:49<00:03,  1.94s/it]
 96%|█████████▋| 27/28 [00:51<00:01,  1.95s/it]

slurm-6020661.out

@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
  0%|          | 0/28 [00:00<?, ?it/s]
  4%|▎         | 1/28 [00:02<00:56,  2.11s/it]
  7%|▋         | 2/28 [00:03<00:40,  1.54s/it]
 11%|█         | 3/28 [00:05<00:43,  1.73s/it]
 14%|█▍        | 4/28 [00:07<00:43,  1.81s/it]
 18%|█▊        | 5/28 [00:09<00:42,  1.85s/it]
 21%|██▏       | 6/28 [00:10<00:41,  1.88s/it]
 25%|██▌       | 7/28 [00:12<00:39,  1.89s/it]
 29%|██▊       | 8/28 [00:14<00:38,  1.91s/it]
 32%|███▏      | 9/28 [00:16<00:36,  1.91s/it]
 36%|███▌      | 10/28 [00:18<00:34,  1.92s/it]
 39%|███▉      | 11/28 [00:20<00:32,  1.93s/it]
 43%|████▎     | 12/28 [00:22<00:30,  1.93s/it]
 46%|████▋     | 13/28 [00:24<00:29,  1.94s/it]
 50%|█████     | 14/28 [00:26<00:27,  1.94s/it]
 54%|█████▎    | 15/28 [00:28<00:25,  1.94s/it]
 57%|█████▋    | 16/28 [00:30<00:23,  1.94s/it]
 61%|██████    | 17/28 [00:32<00:21,  1.94s/it]
 64%|██████▍   | 18/28 [00:34<00:19,  1.94s/it]
 68%|██████▊   | 19/28 [00:36<00:17,  1.94s/it]
 71%|███████▏  | 20/28 [00:38<00:15,  1.94s/it]
 75%|███████▌  | 21/28 [00:40<00:13,  1.94s/it]
 79%|███████▊  | 22/28 [00:42<00:11,  1.94s/it]
 82%|████████▏ | 23/28 [00:43<00:09,  1.94s/it]
 86%|████████▌ | 24/28 [00:45<00:07,  1.95s/it]
 89%|████████▉ | 25/28 [00:47<00:05,  1.95s/it]
 93%|█████████▎| 26/28 [00:49<00:03,  1.95s/it]
 96%|█████████▋| 27/28 [00:51<00:01,  1.95s/it]

slurm-6021266.out

@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
  0%|          | 0/28 [00:00<?, ?it/s]
  4%|▎         | 1/28 [00:04<01:52,  4.16s/it]
  7%|▋         | 2/28 [00:05<01:02,  2.39s/it]
 11%|█         | 3/28 [00:07<00:54,  2.18s/it]
 14%|█▍        | 4/28 [00:09<00:50,  2.08s/it]
 18%|█▊        | 5/28 [00:11<00:46,  2.03s/it]
 21%|██▏       | 6/28 [00:13<00:43,  1.99s/it]
 25%|██▌       | 7/28 [00:14<00:41,  1.97s/it]
 29%|██▊       | 8/28 [00:16<00:39,  1.96s/it]
 32%|███▏      | 9/28 [00:18<00:37,  1.95s/it]
 36%|███▌      | 10/28 [00:20<00:35,  1.95s/it]
 39%|███▉      | 11/28 [00:22<00:32,  1.94s/it]
 43%|████▎     | 12/28 [00:24<00:31,  1.94s/it]
 46%|████▋     | 13/28 [00:26<00:29,  1.94s/it]
 50%|█████     | 14/28 [00:28<00:27,  1.94s/it]
 54%|█████▎    | 15/28 [00:30<00:25,  1.94s/it]
 57%|█████▋    | 16/28 [00:32<00:23,  1.94s/it]
 61%|██████    | 17/28 [00:34<00:21,  1.94s/it]
 64%|██████▍   | 18/28 [00:36<00:19,  1.95s/it]
 68%|██████▊   | 19/28 [00:38<00:17,  1.94s/it]
 71%|███████▏  | 20/28 [00:40<00:15,  1.94s/it]
 75%|███████▌  | 21/28 [00:42<00:13,  1.94s/it]
 79%|███████▊  | 22/28 [00:44<00:11,  1.94s/it]
 82%|████████▏ | 23/28 [00:46<00:09,  1.94s/it]
 86%|████████▌ | 24/28 [00:47<00:07,  1.95s/it]
 89%|████████▉ | 25/28 [00:49<00:05,  1.96s/it]
 93%|█████████▎| 26/28 [00:51<00:03,  1.96s/it]
 96%|█████████▋| 27/28 [00:53<00:01,  1.95s/it]

slurm-6021267.out

@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
  0%|          | 0/28 [00:00<?, ?it/s]
  4%|▎         | 1/28 [00:02<00:56,  2.09s/it]
  7%|▋         | 2/28 [00:03<00:39,  1.52s/it]
 11%|█         | 3/28 [00:05<00:42,  1.71s/it]
 14%|█▍        | 4/28 [00:07<00:43,  1.81s/it]
 18%|█▊        | 5/28 [00:09<00:42,  1.87s/it]
 21%|██▏       | 6/28 [00:11<00:41,  1.90s/it]
 25%|██▌       | 7/28 [00:12<00:40,  1.91s/it]
 29%|██▊       | 8/28 [00:14<00:38,  1.92s/it]
 32%|███▏      | 9/28 [00:16<00:36,  1.93s/it]
 36%|███▌      | 10/28 [00:18<00:34,  1.93s/it]
 39%|███▉      | 11/28 [00:20<00:32,  1.93s/it]
 43%|████▎     | 12/28 [00:22<00:30,  1.93s/it]
 46%|████▋     | 13/28 [00:24<00:28,  1.93s/it]
 50%|█████     | 14/28 [00:26<00:27,  1.93s/it]
 54%|█████▎    | 15/28 [00:28<00:25,  1.94s/it]
 57%|█████▋    | 16/28 [00:30<00:23,  1.94s/it]
 61%|██████    | 17/28 [00:32<00:21,  1.94s/it]
 64%|██████▍   | 18/28 [00:34<00:19,  1.95s/it]
 68%|██████▊   | 19/28 [00:36<00:17,  1.94s/it]
 71%|███████▏  | 20/28 [00:38<00:15,  1.94s/it]
 75%|███████▌  | 21/28 [00:40<00:13,  1.94s/it]
 79%|███████▊  | 22/28 [00:42<00:11,  1.94s/it]
 82%|████████▏ | 23/28 [00:44<00:09,  1.94s/it]
 86%|████████▌ | 24/28 [00:45<00:07,  1.94s/it]
 89%|████████▉ | 25/28 [00:47<00:05,  1.94s/it]
 93%|█████████▎| 26/28 [00:49<00:03,  1.94s/it]
 96%|█████████▋| 27/28 [00:51<00:01,  1.94s/it]

slurm-6021268.out

@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
  0%|          | 0/28 [00:00<?, ?it/s]
  4%|▎         | 1/28 [00:02<00:55,  2.07s/it]
  7%|▋         | 2/28 [00:03<00:39,  1.52s/it]
 11%|█         | 3/28 [00:05<00:42,  1.71s/it]
 14%|█▍        | 4/28 [00:07<00:43,  1.80s/it]
 18%|█▊        | 5/28 [00:09<00:42,  1.85s/it]
 21%|██▏       | 6/28 [00:10<00:41,  1.88s/it]
 25%|██▌       | 7/28 [00:12<00:40,  1.91s/it]
 29%|██▊       | 8/28 [00:14<00:38,  1.93s/it]
 32%|███▏      | 9/28 [00:16<00:36,  1.94s/it]
 36%|███▌      | 10/28 [00:18<00:34,  1.94s/it]
 39%|███▉      | 11/28 [00:20<00:32,  1.94s/it]
 43%|████▎     | 12/28 [00:22<00:30,  1.94s/it]
 46%|████▋     | 13/28 [00:24<00:29,  1.94s/it]
 50%|█████     | 14/28 [00:26<00:27,  1.94s/it]
 54%|█████▎    | 15/28 [00:28<00:25,  1.94s/it]
 57%|█████▋    | 16/28 [00:30<00:23,  1.94s/it]
 61%|██████    | 17/28 [00:32<00:21,  1.94s/it]
 64%|██████▍   | 18/28 [00:34<00:19,  1.94s/it]
 68%|██████▊   | 19/28 [00:36<00:17,  1.94s/it]
 71%|███████▏  | 20/28 [00:38<00:15,  1.94s/it]
 75%|███████▌  | 21/28 [00:40<00:13,  1.94s/it]
 79%|███████▊  | 22/28 [00:42<00:11,  1.95s/it]
 82%|████████▏ | 23/28 [00:44<00:09,  1.95s/it]
 86%|████████▌ | 24/28 [00:45<00:07,  1.94s/it]
 89%|████████▉ | 25/28 [00:47<00:05,  1.94s/it]
 93%|█████████▎| 26/28 [00:49<00:03,  1.94s/it]
 96%|█████████▋| 27/28 [00:51<00:01,  1.94s/it]

slurm-6021269.out

@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
  0%|          | 0/28 [00:00<?, ?it/s]
  4%|▎         | 1/28 [00:02<00:56,  2.08s/it]
  7%|▋         | 2/28 [00:03<00:39,  1.52s/it]
 11%|█         | 3/28 [00:05<00:42,  1.71s/it]
 14%|█▍        | 4/28 [00:07<00:43,  1.80s/it]
 18%|█▊        | 5/28 [00:09<00:42,  1.85s/it]
 21%|██▏       | 6/28 [00:10<00:41,  1.88s/it]
 25%|██▌       | 7/28 [00:12<00:39,  1.90s/it]
 29%|██▊       | 8/28 [00:14<00:38,  1.91s/it]
 32%|███▏      | 9/28 [00:16<00:36,  1.92s/it]
 36%|███▌      | 10/28 [00:18<00:34,  1.93s/it]
 39%|███▉      | 11/28 [00:20<00:32,  1.93s/it]
 43%|████▎     | 12/28 [00:22<00:31,  1.94s/it]
 46%|████▋     | 13/28 [00:24<00:29,  1.95s/it]
 50%|█████     | 14/28 [00:26<00:27,  1.95s/it]
 54%|█████▎    | 15/28 [00:28<00:25,  1.95s/it]
 57%|█████▋    | 16/28 [00:30<00:23,  1.95s/it]
 61%|██████    | 17/28 [00:32<00:21,  1.95s/it]
 64%|██████▍   | 18/28 [00:34<00:19,  1.95s/it]
 68%|██████▊   | 19/28 [00:36<00:17,  1.95s/it]
 71%|███████▏  | 20/28 [00:38<00:15,  1.94s/it]
 75%|███████▌  | 21/28 [00:40<00:13,  1.94s/it]
 79%|███████▊  | 22/28 [00:42<00:11,  1.94s/it]
 82%|████████▏ | 23/28 [00:44<00:09,  1.94s/it]
 86%|████████▌ | 24/28 [00:45<00:07,  1.95s/it]
 89%|████████▉ | 25/28 [00:47<00:05,  1.95s/it]
 93%|█████████▎| 26/28 [00:49<00:03,  1.94s/it]
 96%|█████████▋| 27/28 [00:51<00:01,  1.94s/it]

slurm-6021270.out

@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
  0%|          | 0/28 [00:00<?, ?it/s]
  4%|▎         | 1/28 [00:02<00:55,  2.07s/it]
  7%|▋         | 2/28 [00:03<00:39,  1.52s/it]
 11%|█         | 3/28 [00:05<00:42,  1.71s/it]
 14%|█▍        | 4/28 [00:07<00:43,  1.80s/it]
 18%|█▊        | 5/28 [00:09<00:42,  1.85s/it]
 21%|██▏       | 6/28 [00:10<00:41,  1.88s/it]
 25%|██▌       | 7/28 [00:12<00:39,  1.89s/it]
 29%|██▊       | 8/28 [00:14<00:38,  1.91s/it]
 32%|███▏      | 9/28 [00:16<00:36,  1.91s/it]
 36%|███▌      | 10/28 [00:18<00:34,  1.92s/it]
 39%|███▉      | 11/28 [00:20<00:32,  1.93s/it]
 43%|████▎     | 12/28 [00:22<00:31,  1.94s/it]
 46%|████▋     | 13/28 [00:24<00:29,  1.95s/it]
 50%|█████     | 14/28 [00:26<00:27,  1.95s/it]
 54%|█████▎    | 15/28 [00:28<00:25,  1.95s/it]
 57%|█████▋    | 16/28 [00:30<00:23,  1.94s/it]
 61%|██████    | 17/28 [00:32<00:21,  1.94s/it]
 64%|██████▍   | 18/28 [00:34<00:19,  1.95s/it]
 68%|██████▊   | 19/28 [00:36<00:17,  1.94s/it]
 71%|███████▏  | 20/28 [00:38<00:15,  1.94s/it]
 75%|███████▌  | 21/28 [00:40<00:13,  1.94s/it]
 79%|███████▊  | 22/28 [00:42<00:11,  1.94s/it]
 82%|████████▏ | 23/28 [00:43<00:09,  1.94s/it]
 86%|████████▌ | 24/28 [00:45<00:07,  1.94s/it]
 89%|████████▉ | 25/28 [00:47<00:05,  1.94s/it]
 93%|█████████▎| 26/28 [00:49<00:03,  1.94s/it]
 96%|█████████▋| 27/28 [00:51<00:01,  1.94s/it]