Instructions to use Scale-or-Reason/gemma3-4B_0_split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Scale-or-Reason/gemma3-4B_0_split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Scale-or-Reason/gemma3-4B_0_split")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Scale-or-Reason/gemma3-4B_0_split")
model = AutoModelForMultimodalLM.from_pretrained("Scale-or-Reason/gemma3-4B_0_split")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Scale-or-Reason/gemma3-4B_0_split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Scale-or-Reason/gemma3-4B_0_split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Scale-or-Reason/gemma3-4B_0_split",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Scale-or-Reason/gemma3-4B_0_split

SGLang

How to use Scale-or-Reason/gemma3-4B_0_split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Scale-or-Reason/gemma3-4B_0_split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Scale-or-Reason/gemma3-4B_0_split",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Scale-or-Reason/gemma3-4B_0_split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Scale-or-Reason/gemma3-4B_0_split",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Scale-or-Reason/gemma3-4B_0_split with Docker Model Runner:
```
docker model run hf.co/Scale-or-Reason/gemma3-4B_0_split
```

gemma3-4B_0_split / slurm.out

Nicolas-BZRD

Upload folder using huggingface_hub

870fbf5 verified 8 months ago

Raw

History Blame Contribute Delete

93.2 kB

	2: W1124 00:08:21.177000 270804 torch/distributed/run.py:792]
	2: W1124 00:08:21.177000 270804 torch/distributed/run.py:792] *****************************************
	2: W1124 00:08:21.177000 270804 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
	2: W1124 00:08:21.177000 270804 torch/distributed/run.py:792] *****************************************
	3: W1124 00:08:21.180000 1900294 torch/distributed/run.py:792]
	3: W1124 00:08:21.180000 1900294 torch/distributed/run.py:792] *****************************************
	3: W1124 00:08:21.180000 1900294 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
	3: W1124 00:08:21.180000 1900294 torch/distributed/run.py:792] *****************************************
	0: W1124 00:08:21.180000 1912798 torch/distributed/run.py:792]
	0: W1124 00:08:21.180000 1912798 torch/distributed/run.py:792] *****************************************
	0: W1124 00:08:21.180000 1912798 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
	0: W1124 00:08:21.180000 1912798 torch/distributed/run.py:792] *****************************************
	1: W1124 00:08:21.306000 434889 torch/distributed/run.py:792]
	1: W1124 00:08:21.306000 434889 torch/distributed/run.py:792] *****************************************
	1: W1124 00:08:21.306000 434889 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
	1: W1124 00:08:21.306000 434889 torch/distributed/run.py:792] *****************************************
	0: [2025-11-24 00:08:46,359] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:1912876] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`[39m
	0: [2025-11-24 00:08:46,359] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:1912876] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing[39m
	2: [2025-11-24 00:08:47,025] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:270879] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`[39m
	2: [2025-11-24 00:08:47,026] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:270879] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing[39m
	1: [2025-11-24 00:08:47,077] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:434964] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`[39m
	1: [2025-11-24 00:08:47,078] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:434964] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing[39m
	3: [2025-11-24 00:08:47,277] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:1900370] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`[39m
	3: [2025-11-24 00:08:47,277] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:1900370] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing[39m
	0: [33m[2025-11-24 00:08:49,792] [WARNING] [axolotl.utils.config.normalize_config:139] [PID:1912876] [RANK:0] Invalid value for save_steps (1.6666666666666667) from saves_per_epoch and/or num_epochs. Saving at training end only.[39m
	0: [2025-11-24 00:08:49,874] [INFO] [axolotl.cli.config.load_cfg:245] [PID:1912876] [RANK:0] config:
	0: {
	0: "activation_offloading": false,
	0: "auto_resume_from_checkpoints": true,
	0: "axolotl_config_path": "/lustre/fswork/projects/rech/dgo/udv55np/train/tmp/1763939290239020138.yaml",
	0: "base_model": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-4b",
	0: "base_model_config": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-4b",
	0: "batch_size": 16,
	0: "bf16": true,
	0: "capabilities": {
	0: "bf16": true,
	0: "compute_capability": "sm_90",
	0: "fp8": false,
	0: "n_gpu": 16,
	0: "n_node": 1
	0: },
	0: "chat_template": "gemma3",
	0: "context_parallel_size": 1,
	0: "dataloader_num_workers": 16,
	0: "dataloader_pin_memory": true,
	0: "dataloader_prefetch_factor": 256,
	0: "dataset_prepared_path": "/lustre/fswork/projects/rech/dgo/udv55np/dataset_gemma/Nemotron-Super-49B-v1_5/split_0",
	0: "dataset_processes": 192,
	0: "datasets": [
	0: {
	0: "chat_template": "tokenizer_default",
	0: "data_files": [
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0007.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0009.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0005.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0006.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0014.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0010.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0012.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0008.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0001.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0002.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0013.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0015.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0004.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0011.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0000.jsonl",
	0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0003.jsonl"
	0: ],
	0: "ds_type": "json",
	0: "field_messages": "conversations",
	0: "message_property_mappings": {
	0: "content": "content",
	0: "role": "role"
	0: },
	0: "path": "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking",
	0: "trust_remote_code": false,
	0: "type": "chat_template"
	0: }
	0: ],
	0: "ddp": true,
	0: "deepspeed": {
	0: "bf16": {
	0: "enabled": true
	0: },
	0: "gradient_accumulation_steps": "auto",
	0: "gradient_clipping": "auto",
	0: "train_batch_size": "auto",
	0: "train_micro_batch_size_per_gpu": "auto",
	0: "wall_clock_breakdown": false,
	0: "zero_optimization": {
	0: "contiguous_gradients": true,
	0: "overlap_comm": true,
	0: "reduce_bucket_size": "auto",
	0: "stage": 3,
	0: "stage3_gather_16bit_weights_on_model_save": true,
	0: "stage3_param_persistence_threshold": "auto",
	0: "stage3_prefetch_bucket_size": "auto",
	0: "sub_group_size": 0
	0: }
	0: },
	0: "device": "cuda:0",
	0: "device_map": {
	0: "": 0
	0: },
	0: "dion_rank_fraction": 1.0,
	0: "dion_rank_multiple_of": 1,
	0: "env_capabilities": {
	0: "torch_version": "2.6.0"
	0: },
	0: "eot_tokens": [
	0: "<end_of_turn>"
	0: ],
	0: "eval_batch_size": 1,
	0: "eval_causal_lm_metrics": [
	0: "sacrebleu",
	0: "comet",
	0: "ter",
	0: "chrf"
	0: ],
	0: "eval_max_new_tokens": 128,
	0: "eval_sample_packing": true,
	0: "eval_table_size": 0,
	0: "evals_per_epoch": 0,
	0: "flash_attention": true,
	0: "fp16": false,
	0: "gradient_accumulation_steps": 1,
	0: "gradient_checkpointing": true,
	0: "gradient_checkpointing_kwargs": {
	0: "use_reentrant": true
	0: },
	0: "is_multimodal": true,
	0: "learning_rate": 5e-06,
	0: "lisa_layers_attribute": "model.layers",
	0: "load_best_model_at_end": false,
	0: "load_in_4bit": false,
	0: "load_in_8bit": false,
	0: "local_rank": 0,
	0: "logging_steps": 10,
	0: "lora_dropout": 0.0,
	0: "loraplus_lr_embedding": 1e-06,
	0: "lr_scheduler": "warmup_stable_decay",
	0: "lr_scheduler_kwargs": {
	0: "min_lr_ratio": 0.1,
	0: "num_decay_steps": 200
	0: },
	0: "max_prompt_len": 512,
	0: "mean_resizing_embeddings": false,
	0: "micro_batch_size": 1,
	0: "model_config_type": "gemma3",
	0: "num_epochs": 0.6,
	0: "optimizer": "adamw_torch_fused",
	0: "output_dir": "/lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0",
	0: "pad_to_sequence_len": true,
	0: "pretrain_multipack_attn": true,
	0: "pretrain_multipack_buffer_size": 10000,
	0: "processor_config": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-4b",
	0: "profiler_steps_start": 0,
	0: "qlora_sharded_model_loading": false,
	0: "ray_num_workers": 1,
	0: "resources_per_worker": {
	0: "GPU": 1
	0: },
	0: "sample_packing": true,
	0: "sample_packing_bin_size": 200,
	0: "sample_packing_group_size": 100000,
	0: "save_only_model": true,
	0: "save_safetensors": true,
	0: "save_total_limit": 20,
	0: "saves_per_epoch": 1,
	0: "sequence_len": 16384,
	0: "shuffle_before_merging_datasets": false,
	0: "shuffle_merged_datasets": true,
	0: "skip_prepare_dataset": false,
	0: "strict": false,
	0: "tensor_parallel_size": 1,
	0: "tf32": false,
	0: "tiled_mlp_use_original_mlp": true,
	0: "tokenizer_config": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-27b",
	0: "torch_dtype": "torch.bfloat16",
	0: "train_on_inputs": false,
	0: "trl": {
	0: "log_completions": false,
	0: "mask_truncated_completions": false,
	0: "ref_model_mixup_alpha": 0.9,
	0: "ref_model_sync_steps": 64,
	0: "scale_rewards": true,
	0: "sync_ref_model": false,
	0: "use_vllm": false,
	0: "vllm_server_host": "0.0.0.0",
	0: "vllm_server_port": 8000
	0: },
	0: "use_ray": false,
	0: "use_tensorboard": true,
	0: "val_set_size": 0.0,
	0: "vllm": {
	0: "device": "auto",
	0: "dtype": "auto",
	0: "gpu_memory_utilization": 0.9,
	0: "host": "0.0.0.0",
	0: "port": 8000
	0: },
	0: "warmup_steps": 100,
	0: "weight_decay": 0.0,
	0: "world_size": 16
	0: }[39m
	0: [2025-11-24 00:08:49,876] [INFO] [axolotl.cli.checks.check_user_token:35] [PID:1912876] [RANK:0] Skipping HuggingFace token verification because HF_HUB_OFFLINE is set to True. Only local files will be used.[39m
	0: [2025-11-24 00:08:51,148] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:472] [PID:1912876] [RANK:0] Loading prepared dataset from disk at /lustre/fswork/projects/rech/dgo/udv55np/dataset_gemma/Nemotron-Super-49B-v1_5/split_0/06698e902d3dba325ca34849b1dea5ea...[39m
	0: [2025-11-24 00:09:24,738] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:436] [PID:1912876] [RANK:0] gather_len_batches: [18976, 18976, 18975, 18976, 18976, 18976, 18976, 18976, 18976, 18975, 18976, 18976, 18976, 18976, 18976, 18976][39m
	0: [2025-11-24 00:09:24,803] [INFO] [axolotl.utils.trainer.calc_sample_packing_eff_est:495] [PID:1912876] [RANK:0] sample_packing_eff_est across ranks: [0.9988827705383301, 0.9989354014396667, 0.9989354014396667, 0.9988827705383301, 0.9989354014396667, 0.9989354014396667, 0.9989354014396667, 0.9989354014396667, 0.9988827705383301, 0.9988827705383301, 0.9988827705383301, 0.9989880323410034, 0.9988301396369934, 0.9989354014396667, 0.9989354014396667, 0.9989354014396667][39m
	0: [2025-11-24 00:09:24,810] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:127] [PID:1912876] [RANK:0] Maximum number of steps set at 711[39m
	1: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	1: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	1: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	1: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	2: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	3: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	0: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	0: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	2: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	0: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	3: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	2: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	3: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	3: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	2: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	0: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
	0: [2025-11-24 00:09:31,797] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:110] [PID:1912876] [RANK:0] Patched Trainer.evaluation_loop with nanmean loss calculation[39m
	0: [2025-11-24 00:09:31,798] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:164] [PID:1912876] [RANK:0] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation[39m
	0: Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.45s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.45s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.45s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:20<00:20, 20.54s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.10s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.10s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.35s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.35s/it]
	0:
	1: Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.42s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.43s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.43s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.43s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.09s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.09s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.09s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.34s/it]
	3: Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.40s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.40s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.40s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.40s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.08s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.08s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.08s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.33s/it]
	2: Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.42s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.42s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.42s/it] Loading checkpoint shards: 50%\|█████ \| 1/2 [00:19<00:19, 19.42s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.09s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.09s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.09s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.34s/it] Loading checkpoint shards: 100%\|█████████
	3: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.33s/it]
	3: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.33s/it]
	1: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.34s/it]
	1: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.34s/it]
	0: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.10s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.35s/it]
	1: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.09s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.34s/it]
	2: █\| 2/2 [00:24<00:00, 12.34s/it]
	2:
	2: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.34s/it]
	2: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.09s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.34s/it]
	3: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 11.08s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:24<00:00, 12.33s/it]
	0: Loading checkpoint shards: 100%\|██████████\| 2/2 [00:25<00:00, 11.41s/it] Loading checkpoint shards: 100%\|██████████\| 2/2 [00:25<00:00, 12.78s/it]
	0: [2025-11-24 00:09:59,406] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:345] [PID:1912876] [RANK:0] Converting modules to torch.bfloat16[39m
	0: [2025-11-24 00:10:03,242] [INFO] [axolotl.train.save_initial_configs:416] [PID:1912876] [RANK:0] Pre-saving tokenizer to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0...[39m
	0: [2025-11-24 00:10:03,660] [INFO] [axolotl.train.save_initial_configs:419] [PID:1912876] [RANK:0] Pre-saving model config to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0...[39m
	0: [2025-11-24 00:10:03,690] [INFO] [axolotl.train.save_initial_configs:423] [PID:1912876] [RANK:0] Pre-saving processor to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0...[39m
	0: [2025-11-24 00:10:06,488] [INFO] [axolotl.train.execute_training:203] [PID:1912876] [RANK:0] Starting trainer...[39m
	0: [2025-11-24 00:11:39,109] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:436] [PID:1912876] [RANK:0] gather_len_batches: [18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976, 18976][39m
	0: Parameter Offload - Persistent parameters statistics: param_count = 479, numel = 768880
	0: {'loss': 0.7282, 'grad_norm': 2.3999579863224176, 'learning_rate': 9.05e-07, 'memory/max_mem_active(gib)': 57.15, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.2, 'epoch': 0.01}
	0: 0%\| \| 0/711 [00:00<?, ?it/s] 0%\| \| 1/711 [03:10<37:32:11, 190.33s/it] 0%\| \| 2/711 [03:14<15:52:57, 80.65s/it] 0%\| \| 3/711 [03:16<8:49:26, 44.87s/it] 1%\| \| 4/711 [03:18<5:30:58, 28.09s/it] 1%\| \| 5/711 [03:21<3:41:20, 18.81s/it] 1%\| \| 6/711 [03:23<2:35:25, 13.23s/it] 1%\| \| 7/711 [03:25<1:53:16, 9.65s/it] 1%\| \| 8/711 [03:28<1:25:41, 7.31s/it] 1%\|▏ \| 9/711 [03:30<1:07:15, 5.75s/it] 1%\|▏ \| 10/711 [03:32<54:44, 4.68s/it] 1%\|▏ \| 10/711 [03:32<54:44, 4.68s/it] 2%\|▏ \| 11/711 [03:35<46:17, 3.97s/it] 2%\|▏ \| 12/711 [03:37<40:18, 3.46s/it] 2%\|▏ \| 13/711 [03:39<36:10, 3.11s/it] 2%\|▏ \| 14/711 [03:42<34:11, 2.94s/it] 2%\|▏ \| 15/711 [03:44<32:06, 2.77s/it] 2%\|▏ \| 16/711 [03:46<30:28, 2.63s/it] 2%\|▏ \| 17/711 [03:49<29:24, 2.54s/it] 3%\|▎
	0: {'loss': 0.6672, 'grad_norm': 1.3408937334456381, 'learning_rate': 1.3550000000000002e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.02}
	0: {'loss': 0.6271, 'grad_norm': 0.8591296514459729, 'learning_rate': 1.805e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.03}
	0: \| 18/711 [03:51<28:42, 2.49s/it] 3%\|▎ \| 19/711 [03:53<28:02, 2.43s/it] 3%\|▎ \| 20/711 [03:56<27:33, 2.39s/it] 3%\|▎ \| 20/711 [03:56<27:33, 2.39s/it] 3%\|▎ \| 21/711 [03:58<27:12, 2.37s/it] 3%\|▎ \| 22/711 [04:00<26:57, 2.35s/it] 3%\|▎ \| 23/711 [04:03<26:45, 2.33s/it] 3%\|▎ \| 24/711 [04:05<26:37, 2.33s/it] 4%\|▎ \| 25/711 [04:07<26:33, 2.32s/it] 4%\|▎ \| 26/711 [04:10<26:26, 2.32s/it] 4%\|▍ \| 27/711 [04:12<26:21, 2.31s/it] 4%\|▍ \| 28/711 [04:14<26:17, 2.31s/it] 4%\|▍ \| 29/711 [04:17<26:14, 2.31s/it] 4%\|▍ \| 30/711 [04:19<26:39, 2.35s/it] 4%\|▍ \| 30/711 [04:19<26:39, 2.35s/it] 4%\|▍ \| 31/711 [04:21<26:38, 2.35s/it] 5%\|▍ \| 32/711 [04:24<26:30, 2.34s/it] 5%\|▍ \| 33/711 [04:26<26:21, 2.33s/it] 5%\|▍ \| 34/
	0: {'loss': 0.6047, 'grad_norm': 0.8292303871371115, 'learning_rate': 2.2550000000000004e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.03}
	0: {'loss': 0.5823, 'grad_norm': 0.7246674717655568, 'learning_rate': 2.7050000000000004e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.04}
	0: 711 [04:28<26:13, 2.32s/it] 5%\|▍ \| 35/711 [04:31<26:08, 2.32s/it] 5%\|▌ \| 36/711 [04:33<26:03, 2.32s/it] 5%\|▌ \| 37/711 [04:35<26:01, 2.32s/it] 5%\|▌ \| 38/711 [04:38<25:57, 2.31s/it] 5%\|▌ \| 39/711 [04:40<25:55, 2.31s/it] 6%\|▌ \| 40/711 [04:42<25:53, 2.31s/it] 6%\|▌ \| 40/711 [04:42<25:53, 2.31s/it] 6%\|▌ \| 41/711 [04:44<25:50, 2.31s/it] 6%\|▌ \| 42/711 [04:47<25:47, 2.31s/it] 6%\|▌ \| 43/711 [04:49<25:50, 2.32s/it] 6%\|▌ \| 44/711 [04:51<25:45, 2.32s/it] 6%\|▋ \| 45/711 [04:54<25:40, 2.31s/it] 6%\|▋ \| 46/711 [04:56<25:51, 2.33s/it] 7%\|▋ \| 47/711 [04:58<25:53, 2.34s/it] 7%\|▋ \| 48/711 [05:01<25:46, 2.33s/it] 7%\|▋ \| 49/711 [05:03<25:41, 2.33s/it] 7%\|▋ \| 50/711 [05:05<25:40, 2.33s/it] 7%\|▋ \| 50/711 [0
	0: {'loss': 0.5601, 'grad_norm': 0.7685808720049759, 'learning_rate': 3.1550000000000003e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.05}
	0: 5:05<25:40, 2.33s/it] 7%\|▋ \| 51/711 [05:08<25:33, 2.32s/it] 7%\|▋ \| 52/711 [05:10<25:29, 2.32s/it] 7%\|▋ \| 53/711 [05:12<25:34, 2.33s/it] 8%\|▊ \| 54/711 [05:15<25:28, 2.33s/it] 8%\|▊ \| 55/711 [05:17<25:23, 2.32s/it] 8%\|▊ \| 56/711 [05:19<25:34, 2.34s/it] 8%\|▊ \| 57/711 [05:22<25:26, 2.33s/it] 8%\|▊ \| 58/711 [05:24<25:37, 2.35s/it] 8%\|▊ \| 59/711 [05:26<25:28, 2.34s/it] 8%\|▊ \| 60/711 [05:29<25:19, 2.33s/it] 8%\|▊ \| 60/711 [05:29<25:19, 2.33s/it] 9%\|▊ \| 61/711 [05:31<25:11, 2.33s/it] 9%\|▊ \| 62/711 [05:33<25:19, 2.34s/it] 9%\|▉ \| 63/711 [05:36<25:30, 2.36s/it] 9%\|▉ \| 64/711 [05:38<25:18, 2.35s/it] 9%\|▉ \| 65/711 [05:40<25:08, 2.33s/it] 9%\|▉ \| 66/711 [05:43<25:00, 2.33s/it] 9%\|▉ \| 67/711 [05:45<24:55, 2.32s/it] 10%\|▉ \| 68/711 [05:47<
	0: {'loss': 0.5653, 'grad_norm': 0.7575662741162992, 'learning_rate': 3.6050000000000002e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.06}
	0: {'loss': 0.5571, 'grad_norm': 0.7808588522979137, 'learning_rate': 4.055000000000001e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.07}
	0: 24:49, 2.32s/it] 10%\|▉ \| 69/711 [05:50<24:46, 2.31s/it] 10%\|▉ \| 70/711 [05:52<24:42, 2.31s/it] 10%\|▉ \| 70/711 [05:52<24:42, 2.31s/it] 10%\|▉ \| 71/711 [05:54<24:39, 2.31s/it] 10%\|█ \| 72/711 [05:57<24:35, 2.31s/it] 10%\|█ \| 73/711 [05:59<24:32, 2.31s/it] 10%\|█ \| 74/711 [06:01<24:29, 2.31s/it] 11%\|█ \| 75/711 [06:04<24:26, 2.31s/it] 11%\|█ \| 76/711 [06:06<24:23, 2.30s/it] 11%\|█ \| 77/711 [06:08<24:20, 2.30s/it] 11%\|█ \| 78/711 [06:11<24:30, 2.32s/it] 11%\|█ \| 79/711 [06:13<24:35, 2.33s/it] 11%\|█▏ \| 80/711 [06:15<24:32, 2.33s/it] 11%\|█▏ \| 80/711 [06:15<24:32, 2.33s/it] 11%\|█▏ \| 81/711 [06:18<24:42, 2.35s/it] 12%\|█▏ \| 82/711 [06:20<24:32, 2.34s/it] 12%\|█▏ \| 83/711 [06:22<24:38, 2.35s/it] 12%\|█▏ \| 84/711 [
	0: {'loss': 0.5341, 'grad_norm': 0.8642141374162505, 'learning_rate': 4.505e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.08}
	0: 06:25<24:27, 2.34s/it] 12%\|█▏ \| 85/711 [06:27<24:18, 2.33s/it] 12%\|█▏ \| 86/711 [06:29<24:24, 2.34s/it] 12%\|█▏ \| 87/711 [06:32<24:14, 2.33s/it] 12%\|█▏ \| 88/711 [06:34<24:07, 2.32s/it] 13%\|█▎ \| 89/711 [06:36<24:02, 2.32s/it] 13%\|█▎ \| 90/711 [06:39<23:59, 2.32s/it] 13%\|█▎ \| 90/711 [06:39<23:59, 2.32s/it] 13%\|█▎ \| 91/711 [06:41<23:57, 2.32s/it] 13%\|█▎ \| 92/711 [06:43<23:58, 2.32s/it] 13%\|█▎ \| 93/711 [06:46<24:09, 2.35s/it] 13%\|█▎ \| 94/711 [06:48<24:28, 2.38s/it] 13%\|█▎ \| 95/711 [06:50<24:37, 2.40s/it] 14%\|█▎ \| 96/711 [06:53<24:19, 2.37s/it] 14%\|█▎ \| 97/711 [06:55<24:06, 2.36s/it] 14%\|█▍ \| 98/711 [06:58<24:10, 2.37s/it] 14%\|█▍ \| 99/711 [07:00<23:57, 2.35s/it] 14%\|█▍ \| 100/711 [07:02<23:47, 2.34s/it]
	0: {'loss': 0.5192, 'grad_norm': 0.8305549171618009, 'learning_rate': 4.955e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.08}
	0: {'loss': 0.5459, 'grad_norm': 0.8622685683478952, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.09}
	0: 14%\|█▍ \| 100/711 [07:02<23:47, 2.34s/it] 14%\|█▍ \| 101/711 [07:04<23:40, 2.33s/it] 14%\|█▍ \| 102/711 [07:07<23:35, 2.32s/it] 14%\|█▍ \| 103/711 [07:09<23:30, 2.32s/it] 15%\|█▍ \| 104/711 [07:11<23:26, 2.32s/it] 15%\|█▍ \| 105/711 [07:14<23:23, 2.32s/it] 15%\|█▍ \| 106/711 [07:16<23:21, 2.32s/it] 15%\|█▌ \| 107/711 [07:18<23:18, 2.32s/it] 15%\|█▌ \| 108/711 [07:21<23:13, 2.31s/it] 15%\|█▌ \| 109/711 [07:23<23:10, 2.31s/it] 15%\|█▌ \| 110/711 [07:25<23:20, 2.33s/it] 15%\|█▌ \| 110/711 [07:25<23:20, 2.33s/it] 16%\|█▌ \| 111/711 [07:28<23:30, 2.35s/it] 16%\|█▌ \| 112/711 [07:30<23:23, 2.34s/it] 16%\|█▌ \| 113/711 [07:32<23:27, 2.35s/it] 16%\|█▌ \| 114/711 [07:35<23:16, 2.34s/it] 16%\|█▌ \| 115/711 [07:37<23:07, 2.33s/it] 16%\|█▋ \| 116/711 [07:39<23:01, 2.32s/it
	0: {'loss': 0.5358, 'grad_norm': 0.8178819353819496, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.1}
	0: {'loss': 0.5179, 'grad_norm': 1.6879902769065394, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.11}
	0: ] 16%\|█▋ \| 117/711 [07:42<22:57, 2.32s/it] 17%\|█▋ \| 118/711 [07:44<22:52, 2.31s/it] 17%\|█▋ \| 119/711 [07:46<22:48, 2.31s/it] 17%\|█▋ \| 120/711 [07:49<22:46, 2.31s/it] 17%\|█▋ \| 120/711 [07:49<22:46, 2.31s/it] 17%\|█▋ \| 121/711 [07:51<22:45, 2.31s/it] 17%\|█▋ \| 122/711 [07:53<22:45, 2.32s/it] 17%\|█▋ \| 123/711 [07:56<22:42, 2.32s/it] 17%\|█▋ \| 124/711 [07:58<22:42, 2.32s/it] 18%\|█▊ \| 125/711 [08:00<22:52, 2.34s/it] 18%\|█▊ \| 126/711 [08:03<23:14, 2.38s/it] 18%\|█▊ \| 127/711 [08:05<23:21, 2.40s/it] 18%\|█▊ \| 128/711 [08:07<23:04, 2.38s/it] 18%\|█▊ \| 129/711 [08:10<22:51, 2.36s/it] 18%\|█▊ \| 130/711 [08:12<22:42, 2.35s/it] 18%\|█▊ \| 130/711 [08:12<22:42, 2.35s/it] 18%\|█▊ \| 131/711 [08:14<22:36, 2.34s/it] 1
	0: {'loss': 0.5223, 'grad_norm': 0.8227895864412552, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.12}
	0: 9%\|█▊ \| 132/711 [08:17<22:30, 2.33s/it] 19%\|█▊ \| 133/711 [08:19<22:31, 2.34s/it] 19%\|█▉ \| 134/711 [08:21<22:25, 2.33s/it] 19%\|█▉ \| 135/711 [08:24<22:19, 2.33s/it] 19%\|█▉ \| 136/711 [08:26<22:15, 2.32s/it] 19%\|█▉ \| 137/711 [08:28<22:10, 2.32s/it] 19%\|█▉ \| 138/711 [08:31<22:05, 2.31s/it] 20%\|█▉ \| 139/711 [08:33<22:00, 2.31s/it] 20%\|█▉ \| 140/711 [08:35<21:57, 2.31s/it] 20%\|█▉ \| 140/711 [08:35<21:57, 2.31s/it] 20%\|█▉ \| 141/711 [08:38<21:56, 2.31s/it] 20%\|█▉ \| 142/711 [08:40<22:05, 2.33s/it] 20%\|██ \| 143/711 [08:42<22:07, 2.34s/it] 20%\|██ \| 144/711 [08:45<22:03, 2.34s/it] 20%\|██ \| 145/711 [08:47<21:58, 2.33s/it] 21%\|██ \| 146/711 [08:49<21:53, 2.32s/it] 21%\|██ \| 147/711 [08:52<21:49, 2.32s/it] 21%\|██ \| 148/711 [08:54<21:45, 2.32s/it] 21
	0: {'loss': 0.523, 'grad_norm': 0.7452356447124456, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.13}
	0: {'loss': 0.5237, 'grad_norm': 0.8791556578937845, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.13}
	0: %\|██ \| 149/711 [08:56<21:41, 2.32s/it] 21%\|██ \| 150/711 [08:58<21:38, 2.32s/it] 21%\|██ \| 150/711 [08:59<21:38, 2.32s/it] 21%\|██ \| 151/711 [09:01<21:35, 2.31s/it] 21%\|██▏ \| 152/711 [09:03<21:40, 2.33s/it] 22%\|██▏ \| 153/711 [09:05<21:36, 2.32s/it] 22%\|██▏ \| 154/711 [09:08<21:31, 2.32s/it] 22%\|██▏ \| 155/711 [09:10<21:26, 2.31s/it] 22%\|██▏ \| 156/711 [09:12<21:22, 2.31s/it] 22%\|██▏ \| 157/711 [09:15<21:20, 2.31s/it] 22%\|██▏ \| 158/711 [09:17<21:20, 2.31s/it] 22%\|██▏ \| 159/711 [09:19<21:41, 2.36s/it] 23%\|██▎ \| 160/711 [09:22<21:36, 2.35s/it] 23%\|██▎ \| 160/711 [09:22<21:36, 2.35s/it] 23%\|██▎ \| 161/711 [09:24<21:28, 2.34s/it] 23%\|██▎ \| 162/711 [09:26<21:23, 2.34s/it] 23%\|██▎ \| 163/711 [09:29
	0: {'loss': 0.5143, 'grad_norm': 0.7496360454577663, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.14}
	0: <21:15, 2.33s/it] 23%\|██▎ \| 164/711 [09:31<21:12, 2.33s/it] 23%\|██▎ \| 165/711 [09:33<21:08, 2.32s/it] 23%\|██▎ \| 166/711 [09:36<21:17, 2.34s/it] 23%\|██▎ \| 167/711 [09:38<21:09, 2.33s/it] 24%\|██▎ \| 168/711 [09:40<21:04, 2.33s/it] 24%\|██▍ \| 169/711 [09:43<20:59, 2.32s/it] 24%\|██▍ \| 170/711 [09:45<20:54, 2.32s/it] 24%\|██▍ \| 170/711 [09:45<20:54, 2.32s/it] 24%\|██▍ \| 171/711 [09:47<20:52, 2.32s/it] 24%\|██▍ \| 172/711 [09:50<20:49, 2.32s/it] 24%\|██▍ \| 173/711 [09:52<20:46, 2.32s/it] 24%\|██▍ \| 174/711 [09:54<20:44, 2.32s/it] 25%\|██▍ \| 175/711 [09:57<20:48, 2.33s/it] 25%\|██▍ \| 176/711 [09:59<20:53, 2.34s/it] 25%\|██▍ \| 177/711 [10:01<20:46, 2.33s/it] 25%\|██▌ \| 178/711 [10:04<20:53, 2.35s/it] 25%\|██▌ \| 179/711 [10:06<20:45, 2.34s/it] 2
	0: {'loss': 0.5072, 'grad_norm': 0.7656965770735714, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 67.45, 'epoch': 0.15}
	0: {'loss': 0.5029, 'grad_norm': 0.7795187884752995, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.16}
	0: 5%\|██▌ \| 180/711 [10:08<20:39, 2.33s/it] 25%\|██▌ \| 180/711 [10:08<20:39, 2.33s/it] 25%\|██▌ \| 181/711 [10:11<20:33, 2.33s/it] 26%\|██▌ \| 182/711 [10:13<20:27, 2.32s/it] 26%\|██▌ \| 183/711 [10:15<20:23, 2.32s/it] 26%\|██▌ \| 184/711 [10:18<20:19, 2.31s/it] 26%\|██▌ \| 185/711 [10:20<20:15, 2.31s/it] 26%\|██▌ \| 186/711 [10:22<20:13, 2.31s/it] 26%\|██▋ \| 187/711 [10:25<20:09, 2.31s/it] 26%\|██▋ \| 188/711 [10:27<20:20, 2.33s/it] 27%\|██▋ \| 189/711 [10:29<20:14, 2.33s/it] 27%\|██▋ \| 190/711 [10:32<20:44, 2.39s/it] 27%\|██▋ \| 190/711 [10:32<20:44, 2.39s/it] 27%\|██▋ \| 191/711 [10:34<20:38, 2.38s/it] 27%\|██▋ \| 192/711 [10:37<20:32, 2.37s/it] 27%\|██▋ \| 193/711 [10:39<20:19, 2.35s/it] 27%\|██▋ \| 194/7
	0: {'loss': 0.5088, 'grad_norm': 0.8707955733484418, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.17}
	0: 11 [10:41<20:10, 2.34s/it] 27%\|██▋ \| 195/711 [10:43<20:06, 2.34s/it] 28%\|██▊ \| 196/711 [10:46<19:58, 2.33s/it] 28%\|██▊ \| 197/711 [10:48<20:00, 2.34s/it] 28%\|██▊ \| 198/711 [10:51<20:08, 2.36s/it] 28%\|██▊ \| 199/711 [10:53<19:59, 2.34s/it] 28%\|██▊ \| 200/711 [10:55<19:52, 2.33s/it] 28%\|██▊ \| 200/711 [10:55<19:52, 2.33s/it] 28%\|██▊ \| 201/711 [10:57<19:47, 2.33s/it] 28%\|██▊ \| 202/711 [11:00<19:40, 2.32s/it] 29%\|██▊ \| 203/711 [11:02<19:37, 2.32s/it] 29%\|██▊ \| 204/711 [11:04<19:45, 2.34s/it] 29%\|██▉ \| 205/711 [11:07<19:39, 2.33s/it] 29%\|██▉ \| 206/711 [11:09<19:37, 2.33s/it] 29%\|██▉ \| 207/711 [11:12<20:12, 2.41s/it] 29%\|██▉ \| 208/711 [11:14<20:01, 2.39s/it] 29%\|██▉ \| 209/711 [11:16<19:46, 2.36s/it] 30%\|██▉ \| 210/711 [11:19<19:37, 2.3
	0: {'loss': 0.5012, 'grad_norm': 0.8320228094655582, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.18}
	0: {'loss': 0.5124, 'grad_norm': 1.083294587127778, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.19}
	0: 5s/it] 30%\|██▉ \| 210/711 [11:19<19:37, 2.35s/it] 30%\|██▉ \| 211/711 [11:21<19:30, 2.34s/it] 30%\|██▉ \| 212/711 [11:23<19:23, 2.33s/it] 30%\|██▉ \| 213/711 [11:26<19:16, 2.32s/it] 30%\|███ \| 214/711 [11:28<19:12, 2.32s/it] 30%\|███ \| 215/711 [11:30<19:09, 2.32s/it] 30%\|███ \| 216/711 [11:33<19:06, 2.32s/it] 31%\|███ \| 217/711 [11:35<19:03, 2.31s/it] 31%\|███ \| 218/711 [11:37<19:03, 2.32s/it] 31%\|███ \| 219/711 [11:39<19:01, 2.32s/it] 31%\|███ \| 220/711 [11:42<18:59, 2.32s/it] 31%\|███ \| 220/711 [11:42<18:59, 2.32s/it] 31%\|███ \| 221/711 [11:44<18:58, 2.32s/it] 31%\|███ \| 222/711 [11:47<19:09, 2.35s/it] 31%\|███▏ \| 223/711 [11:49<19:11, 2.36s/it] 32%\|███▏ \| 224/711 [11:51<19:07, 2.36s/it] 32%\|███�
	0: {'loss': 0.4994, 'grad_norm': 0.7355209967169852, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.19}
	0: �� \| 225/711 [11:54<18:57, 2.34s/it] 32%\|███▏ \| 226/711 [11:56<19:01, 2.35s/it] 32%\|███▏ \| 227/711 [11:58<18:52, 2.34s/it] 32%\|███▏ \| 228/711 [12:01<18:45, 2.33s/it] 32%\|███▏ \| 229/711 [12:03<18:40, 2.33s/it] 32%\|███▏ \| 230/711 [12:05<18:35, 2.32s/it] 32%\|███▏ \| 230/711 [12:05<18:35, 2.32s/it] 32%\|███▏ \| 231/711 [12:08<18:32, 2.32s/it] 33%\|███▎ \| 232/711 [12:10<18:30, 2.32s/it] 33%\|███▎ \| 233/711 [12:12<18:28, 2.32s/it] 33%\|███▎ \| 234/711 [12:14<18:24, 2.32s/it] 33%\|███▎ \| 235/711 [12:17<18:21, 2.31s/it] 33%\|███▎ \| 236/711 [12:19<18:19, 2.31s/it] 33%\|███▎ \| 237/711 [12:21<18:15, 2.31s/it] 33%\|███▎ \| 238/711 [12:24<18:13, 2.31s/it] 34%\|███▎ \| 239/711 [12:26<18:18, 2.33s/it] 34%\|███▍ \| 240/711 [12:28<18:20, 2.34s/it]
	0: {'loss': 0.5051, 'grad_norm': 0.7864641958494194, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.2}
	0: {'loss': 0.4913, 'grad_norm': 0.8505484395139187, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.21}
	0: 34%\|███▍ \| 240/711 [12:28<18:20, 2.34s/it] 34%\|███▍ \| 241/711 [12:31<18:15, 2.33s/it] 34%\|███▍ \| 242/711 [12:33<18:09, 2.32s/it] 34%\|███▍ \| 243/711 [12:35<18:11, 2.33s/it] 34%\|███▍ \| 244/711 [12:38<18:05, 2.32s/it] 34%\|███▍ \| 245/711 [12:40<18:00, 2.32s/it] 35%\|███▍ \| 246/711 [12:42<17:56, 2.32s/it] 35%\|███▍ \| 247/711 [12:45<17:53, 2.31s/it] 35%\|███▍ \| 248/711 [12:47<17:51, 2.31s/it] 35%\|███▌ \| 249/711 [12:49<17:49, 2.31s/it] 35%\|███▌ \| 250/711 [12:52<17:46, 2.31s/it] 35%\|███▌ \| 250/711 [12:52<17:46, 2.31s/it] 35%\|███▌ \| 251/711 [12:54<17:44, 2.31s/it] 35%\|███▌ \| 252/711 [12:56<17:42, 2.32s/it] 36%\|███▌ \| 253/711 [12:59<17:41, 2.32s/it] 36%\|███▌ \| 254/711 [13:01<17:38, 2.32s/
	0: {'loss': 0.4871, 'grad_norm': 0.8233442983825041, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.22}
	0: it] 36%\|███▌ \| 255/711 [13:03<17:45, 2.34s/it] 36%\|███▌ \| 256/711 [13:06<17:52, 2.36s/it] 36%\|███▌ \| 257/711 [13:08<17:45, 2.35s/it] 36%\|███▋ \| 258/711 [13:10<17:40, 2.34s/it] 36%\|███▋ \| 259/711 [13:13<17:34, 2.33s/it] 37%\|███▋ \| 260/711 [13:15<17:29, 2.33s/it] 37%\|███▋ \| 260/711 [13:15<17:29, 2.33s/it] 37%\|███▋ \| 261/711 [13:17<17:25, 2.32s/it] 37%\|███▋ \| 262/711 [13:20<17:21, 2.32s/it] 37%\|███▋ \| 263/711 [13:22<17:17, 2.32s/it] 37%\|███▋ \| 264/711 [13:24<17:15, 2.32s/it] 37%\|███▋ \| 265/711 [13:26<17:13, 2.32s/it] 37%\|███▋ \| 266/711 [13:29<17:10, 2.32s/it] 38%\|███▊ \| 267/711 [13:31<17:07, 2.31s/it] 38%\|███▊ \| 268/711 [13:34<17:16, 2.34s/it] 38%\|███▊ \| 269/711 [13:36<17:13, 2.34s/it] 38%\|███▊ \| 270/711 [13:38<1
	0: {'loss': 0.4887, 'grad_norm': 0.7977283697648062, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.23}
	0: {'loss': 0.501, 'grad_norm': 0.788114718310765, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.24}
	0: 7:08, 2.33s/it] 38%\|███▊ \| 270/711 [13:38<17:08, 2.33s/it] 38%\|███▊ \| 271/711 [13:41<17:12, 2.35s/it] 38%\|███▊ \| 272/711 [13:43<17:13, 2.35s/it] 38%\|███▊ \| 273/711 [13:45<17:07, 2.35s/it] 39%\|███▊ \| 274/711 [13:48<17:10, 2.36s/it] 39%\|███▊ \| 275/711 [13:50<17:06, 2.35s/it] 39%\|███▉ \| 276/711 [13:52<16:58, 2.34s/it] 39%\|███▉ \| 277/711 [13:55<16:53, 2.34s/it] 39%\|███▉ \| 278/711 [13:57<16:49, 2.33s/it] 39%\|███▉ \| 279/711 [13:59<16:44, 2.33s/it] 39%\|███▉ \| 280/711 [14:02<16:40, 2.32s/it] 39%\|███▉ \| 280/711 [14:02<16:40, 2.32s/it] 40%\|███▉ \| 281/711 [14:04<16:36, 2.32s/it] 40%\|███▉ \| 282/711 [14:06<16:34, 2.32s/it] 40%\|███▉ \| 283/711 [14:09<16:34, 2.32s/it] 40%\|███▉ \| 284/711 [1
	0: {'loss': 0.4765, 'grad_norm': 0.7750954499136393, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.24}
	0: 4:11<16:41, 2.35s/it] 40%\|████ \| 285/711 [14:13<16:36, 2.34s/it] 40%\|████ \| 286/711 [14:16<16:31, 2.33s/it] 40%\|████ \| 287/711 [14:18<16:43, 2.37s/it] 41%\|████ \| 288/711 [14:20<16:41, 2.37s/it] 41%\|████ \| 289/711 [14:23<16:31, 2.35s/it] 41%\|████ \| 290/711 [14:25<16:26, 2.34s/it] 41%\|████ \| 290/711 [14:25<16:26, 2.34s/it] 41%\|████ \| 291/711 [14:27<16:20, 2.33s/it] 41%\|████ \| 292/711 [14:30<16:25, 2.35s/it] 41%\|████ \| 293/711 [14:32<16:24, 2.35s/it] 41%\|████▏ \| 294/711 [14:34<16:18, 2.35s/it] 41%\|████▏ \| 295/711 [14:37<16:13, 2.34s/it] 42%\|████▏ \| 296/711 [14:39<16:08, 2.33s/it] 42%\|████▏ \| 297/711 [14:41<16:13, 2.35s/it] 42%\|████▏ \| 298/711 [14:44<16:07, 2.34s/it] 42%\|████▏ \| 299/711 [14:46<16:03, 2.34s/it] 42%\|█�
	0: {'loss': 0.488, 'grad_norm': 0.778477888845856, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.25}
	0: {'loss': 0.4871, 'grad_norm': 0.7785532844235397, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.26}
	0: �██▏ \| 300/711 [14:48<15:59, 2.33s/it] 42%\|████▏ \| 300/711 [14:48<15:59, 2.33s/it] 42%\|████▏ \| 301/711 [14:51<15:55, 2.33s/it] 42%\|████▏ \| 302/711 [14:53<15:50, 2.32s/it] 43%\|████▎ \| 303/711 [14:55<15:54, 2.34s/it] 43%\|████▎ \| 304/711 [14:58<15:55, 2.35s/it] 43%\|████▎ \| 305/711 [15:00<15:51, 2.34s/it] 43%\|████▎ \| 306/711 [15:02<15:46, 2.34s/it] 43%\|████▎ \| 307/711 [15:05<15:42, 2.33s/it] 43%\|████▎ \| 308/711 [15:07<15:37, 2.33s/it] 43%\|████▎ \| 309/711 [15:10<15:47, 2.36s/it] 44%\|████▎ \| 310/711 [15:12<15:52, 2.37s/it] 44%\|████▎ \| 310/711 [15:12<15:52, 2.37s/it] 44%\|████▎ \| 311/711 [15:14<15:42, 2.36s/it] 44%\|████▍ \| 312/711 [15:17<15:49, 2.38s/it] 44%\|████▍ \| 313/7
	0: {'loss': 0.4915, 'grad_norm': 0.8063698152361907, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.27}
	0: 11 [15:19<16:01, 2.42s/it] 44%\|████▍ \| 314/711 [15:22<15:55, 2.41s/it] 44%\|████▍ \| 315/711 [15:24<15:44, 2.38s/it] 44%\|████▍ \| 316/711 [15:26<15:33, 2.36s/it] 45%\|████▍ \| 317/711 [15:29<15:25, 2.35s/it] 45%\|████▍ \| 318/711 [15:31<15:25, 2.35s/it] 45%\|████▍ \| 319/711 [15:33<15:25, 2.36s/it] 45%\|████▌ \| 320/711 [15:36<15:22, 2.36s/it] 45%\|████▌ \| 320/711 [15:36<15:22, 2.36s/it] 45%\|████▌ \| 321/711 [15:38<15:45, 2.43s/it] 45%\|████▌ \| 322/711 [15:41<15:30, 2.39s/it] 45%\|████▌ \| 323/711 [15:43<16:06, 2.49s/it] 46%\|████▌ \| 324/711 [15:46<16:30, 2.56s/it] 46%\|████▌ \| 325/711 [15:48<15:59, 2.49s/it] 46%\|████▌ \| 326/711 [15:51<16:07, 2.51s/it] 46%\|████▌ \| 327/711 [15:53<15:44, 2.46s/it] 46%\|████▌ \| 328/711 [15:55<15:
	0: {'loss': 0.4894, 'grad_norm': 0.7798282062358487, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.28}
	0: {'loss': 0.4825, 'grad_norm': 0.750224606954942, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.29}
	0: 26, 2.42s/it] 46%\|████▋ \| 329/711 [15:58<15:13, 2.39s/it] 46%\|████▋ \| 330/711 [16:00<15:07, 2.38s/it] 46%\|████▋ \| 330/711 [16:00<15:07, 2.38s/it] 47%\|████▋ \| 331/711 [16:03<14:57, 2.36s/it] 47%\|████▋ \| 332/711 [16:05<14:49, 2.35s/it] 47%\|████▋ \| 333/711 [16:07<15:21, 2.44s/it] 47%\|████▋ \| 334/711 [16:10<15:12, 2.42s/it] 47%\|████▋ \| 335/711 [16:12<15:12, 2.43s/it] 47%\|████▋ \| 336/711 [16:15<14:57, 2.39s/it] 47%\|████▋ \| 337/711 [16:17<14:46, 2.37s/it] 48%\|████▊ \| 338/711 [16:19<14:39, 2.36s/it] 48%\|████▊ \| 339/711 [16:22<14:32, 2.35s/it] 48%\|████▊ \| 340/711 [16:24<14:27, 2.34s/it] 48%\|████▊ \| 340/711 [16:24<14:27, 2.34s/it] 48%\|████▊ \| 341/711 [16:26<14:23, 2.33s/it] 48%\|�
	0: {'loss': 0.4856, 'grad_norm': 0.736611045158727, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.3}
	0: �███▊ \| 342/711 [16:29<14:19, 2.33s/it] 48%\|████▊ \| 343/711 [16:31<14:15, 2.33s/it] 48%\|████▊ \| 344/711 [16:33<14:14, 2.33s/it] 49%\|████▊ \| 345/711 [16:35<14:08, 2.32s/it] 49%\|████▊ \| 346/711 [16:38<14:38, 2.41s/it] 49%\|████▉ \| 347/711 [16:40<14:25, 2.38s/it] 49%\|████▉ \| 348/711 [16:43<14:14, 2.35s/it] 49%\|████▉ \| 349/711 [16:45<14:06, 2.34s/it] 49%\|████▉ \| 350/711 [16:47<14:06, 2.35s/it] 49%\|████▉ \| 350/711 [16:47<14:06, 2.35s/it] 49%\|████▉ \| 351/711 [16:50<14:07, 2.36s/it] 50%\|████▉ \| 352/711 [16:52<14:00, 2.34s/it] 50%\|████▉ \| 353/711 [16:54<13:55, 2.33s/it] 50%\|████▉ \| 354/711 [16:57<13:50, 2.33s/it] 50%\|████▉ \| 355/711 [16:59<13:47, 2.32s/it] 50%\|█████ \| 356/711 [17:01<13:44, 2.32s/it] 50%\|█████
	0: {'loss': 0.4927, 'grad_norm': 0.7853737850371227, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.3}
	0: {'loss': 0.4881, 'grad_norm': 0.7490924239534897, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.31}
	0: \| 357/711 [17:04<13:41, 2.32s/it] 50%\|█████ \| 358/711 [17:06<13:38, 2.32s/it] 50%\|█████ \| 359/711 [17:08<13:55, 2.37s/it] 51%\|█████ \| 360/711 [17:11<13:46, 2.35s/it] 51%\|█████ \| 360/711 [17:11<13:46, 2.35s/it] 51%\|█████ \| 361/711 [17:13<13:40, 2.34s/it] 51%\|█████ \| 362/711 [17:15<13:35, 2.34s/it] 51%\|█████ \| 363/711 [17:18<13:30, 2.33s/it] 51%\|█████ \| 364/711 [17:20<13:26, 2.32s/it] 51%\|█████▏ \| 365/711 [17:22<13:27, 2.33s/it] 51%\|█████▏ \| 366/711 [17:25<13:28, 2.34s/it] 52%\|█████▏ \| 367/711 [17:27<13:29, 2.35s/it] 52%\|█████▏ \| 368/711 [17:29<13:23, 2.34s/it] 52%\|█████▏ \| 369/711 [17:32<13:19, 2.34s/it] 52%\|█████▏ \| 370/711 [17:34<13:14, 2.33s/it] 52%\|█████▏ \| 3
	0: {'loss': 0.4889, 'grad_norm': 0.7921991687866194, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.32}
	0: 70/711 [17:34<13:14, 2.33s/it] 52%\|█████▏ \| 371/711 [17:36<13:11, 2.33s/it] 52%\|█████▏ \| 372/711 [17:39<13:08, 2.33s/it] 52%\|█████▏ \| 373/711 [17:41<13:07, 2.33s/it] 53%\|█████▎ \| 374/711 [17:43<13:04, 2.33s/it] 53%\|█████▎ \| 375/711 [17:46<13:01, 2.33s/it] 53%\|█████▎ \| 376/711 [17:48<12:57, 2.32s/it] 53%\|█████▎ \| 377/711 [17:50<12:54, 2.32s/it] 53%\|█████▎ \| 378/711 [17:53<12:53, 2.32s/it] 53%\|█████▎ \| 379/711 [17:55<12:57, 2.34s/it] 53%\|█████▎ \| 380/711 [17:57<12:52, 2.33s/it] 53%\|█████▎ \| 380/711 [17:57<12:52, 2.33s/it] 54%\|█████▎ \| 381/711 [18:00<12:57, 2.35s/it] 54%\|█████▎ \| 382/711 [18:02<12:56, 2.36s/it] 54%\|█████▍ \| 383/711 [18:05<12:55, 2.36s/it] 54%\|█████▍ \| 384/711 [18:07<12:48, 2.35s/it] 54%\|██
	0: {'loss': 0.4822, 'grad_norm': 0.8102116642711951, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.33}
	0: ███▍ \| 385/711 [18:09<12:42, 2.34s/it] 54%\|█████▍ \| 386/711 [18:11<12:38, 2.33s/it] 54%\|█████▍ \| 387/711 [18:14<12:33, 2.33s/it] 55%\|█████▍ \| 388/711 [18:16<12:30, 2.32s/it] 55%\|█████▍ \| 389/711 [18:18<12:28, 2.32s/it] 55%\|█████▍ \| 390/711 [18:21<12:28, 2.33s/it] 55%\|█████▍ \| 390/711 [18:21<12:28, 2.33s/it] 55%\|█████▍ \| 391/711 [18:23<12:24, 2.33s/it] 55%\|█████▌ \| 392/711 [18:25<12:24, 2.34s/it] 55%\|█████▌ \| 393/711 [18:28<12:20, 2.33s/it] 55%\|█████▌ \| 394/711 [18:30<12:16, 2.32s/it] 56%\|█████▌ \| 395/711 [18:32<12:22, 2.35s/it] 56%\|█████▌ \| 396/711 [18:35<12:16, 2.34s/it] 56%\|█████▌ \| 397/711 [18:37<12:11, 2.33s/it] 56%\|█████▌ \| 398/711 [18:39<12:11, 2.34s/it] 56%\|█████▌ \| 399/711 [18:42<12:12, 2.
	0: {'loss': 0.4678, 'grad_norm': 0.7889843890610096, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.34}
	0: {'loss': 0.4732, 'grad_norm': 0.7803377614587503, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.35}
	0: 35s/it] 56%\|█████▋ \| 400/711 [18:44<12:07, 2.34s/it] 56%\|█████▋ \| 400/711 [18:44<12:07, 2.34s/it] 56%\|█████▋ \| 401/711 [18:46<12:02, 2.33s/it] 57%\|█████▋ \| 402/711 [18:49<11:59, 2.33s/it] 57%\|█████▋ \| 403/711 [18:51<11:56, 2.32s/it] 57%\|█████▋ \| 404/711 [18:53<11:52, 2.32s/it] 57%\|█████▋ \| 405/711 [18:56<12:01, 2.36s/it] 57%\|█████▋ \| 406/711 [18:58<11:54, 2.34s/it] 57%\|█████▋ \| 407/711 [19:00<11:49, 2.33s/it] 57%\|█████▋ \| 408/711 [19:03<11:44, 2.33s/it] 58%\|█████▊ \| 409/711 [19:05<11:48, 2.35s/it] 58%\|█████▊ \| 410/711 [19:07<11:42, 2.33s/it] 58%\|█████▊ \| 410/711 [19:07<11:42, 2.33s/it] 58%\|█████▊ \| 411/711 [19:10<11:45, 2.35s/it] 58%\|█████▊ \| 412/711 [19:12<11
	0: {'loss': 0.4773, 'grad_norm': 0.7786794033275539, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.35}
	0: :38, 2.34s/it] 58%\|█████▊ \| 413/711 [19:14<11:34, 2.33s/it] 58%\|█████▊ \| 414/711 [19:17<11:35, 2.34s/it] 58%\|█████▊ \| 415/711 [19:19<11:35, 2.35s/it] 59%\|█████▊ \| 416/711 [19:22<11:37, 2.36s/it] 59%\|█████▊ \| 417/711 [19:24<11:30, 2.35s/it] 59%\|█████▉ \| 418/711 [19:26<11:26, 2.34s/it] 59%\|█████▉ \| 419/711 [19:29<11:22, 2.34s/it] 59%\|█████▉ \| 420/711 [19:31<11:18, 2.33s/it] 59%\|█████▉ \| 420/711 [19:31<11:18, 2.33s/it] 59%\|█████▉ \| 421/711 [19:33<11:15, 2.33s/it] 59%\|█████▉ \| 422/711 [19:36<11:11, 2.32s/it] 59%\|█████▉ \| 423/711 [19:38<11:07, 2.32s/it] 60%\|█████▉ \| 424/711 [19:40<11:06, 2.32s/it] 60%\|█████▉ \| 425/711 [19:42<11:02, 2.32s/it] 60%\|█████▉ \| 426/711 [19:45<11:00, 2.32s/it] 60%\|██████
	0: {'loss': 0.4759, 'grad_norm': 0.7409304393739385, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.36}
	0: {'loss': 0.4784, 'grad_norm': 0.7489672735206069, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.37}
	0: \| 427/711 [19:47<10:58, 2.32s/it] 60%\|██████ \| 428/711 [19:49<11:02, 2.34s/it] 60%\|██████ \| 429/711 [19:52<10:58, 2.34s/it] 60%\|██████ \| 430/711 [19:54<10:59, 2.35s/it] 60%\|██████ \| 430/711 [19:54<10:59, 2.35s/it] 61%\|██████ \| 431/711 [19:57<10:59, 2.35s/it] 61%\|██████ \| 432/711 [19:59<10:53, 2.34s/it] 61%\|██████ \| 433/711 [20:01<10:49, 2.34s/it] 61%\|██████ \| 434/711 [20:03<10:44, 2.33s/it] 61%\|██████ \| 435/711 [20:06<10:40, 2.32s/it] 61%\|██████▏ \| 436/711 [20:08<10:42, 2.33s/it] 61%\|██████▏ \| 437/711 [20:11<10:40, 2.34s/it] 62%\|██████▏ \| 438/711 [20:13<10:36, 2.33s/it] 62%\|██████▏ \| 439/711 [20:15<10:32, 2.33s/it] 62%\|██████▏ \| 440/711 [20:17<10:29, 2.32s/it] 62%\|█�
	0: {'loss': 0.4716, 'grad_norm': 0.7729942449390255, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.38}
	0: ��████▏ \| 440/711 [20:17<10:29, 2.32s/it] 62%\|██████▏ \| 441/711 [20:20<10:27, 2.32s/it] 62%\|██████▏ \| 442/711 [20:22<10:24, 2.32s/it] 62%\|██████▏ \| 443/711 [20:24<10:21, 2.32s/it] 62%\|██████▏ \| 444/711 [20:27<10:18, 2.32s/it] 63%\|██████▎ \| 445/711 [20:29<10:36, 2.39s/it] 63%\|██████▎ \| 446/711 [20:32<10:31, 2.38s/it] 63%\|██████▎ \| 447/711 [20:34<10:41, 2.43s/it] 63%\|██████▎ \| 448/711 [20:37<10:30, 2.40s/it] 63%\|██████▎ \| 449/711 [20:39<10:21, 2.37s/it] 63%\|██████▎ \| 450/711 [20:41<10:14, 2.35s/it] 63%\|██████▎ \| 450/711 [20:41<10:14, 2.35s/it] 63%\|██████▎ \| 451/711 [20:43<10:08, 2.34s/it] 64%\|██████▎ \| 452/711 [20:46<10:17, 2.39s/it] 64%\|██████▎ \| 453/711 [20:48<10:09, 2.36s/it] 64%\|█████�
	0: {'loss': 0.467, 'grad_norm': 0.7403462320672021, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.39}
	0: �▍ \| 454/711 [20:51<10:03, 2.35s/it] 64%\|██████▍ \| 455/711 [20:53<09:58, 2.34s/it] 64%\|██████▍ \| 456/711 [20:55<09:54, 2.33s/it] 64%\|██████▍ \| 457/711 [20:57<09:50, 2.32s/it] 64%\|██████▍ \| 458/711 [21:00<09:46, 2.32s/it] 65%\|██████▍ \| 459/711 [21:02<09:44, 2.32s/it] 65%\|██████▍ \| 460/711 [21:04<09:42, 2.32s/it] 65%\|██████▍ \| 460/711 [21:04<09:42, 2.32s/it] 65%\|██████▍ \| 461/711 [21:07<09:40, 2.32s/it] 65%\|██████▍ \| 462/711 [21:09<09:41, 2.34s/it] 65%\|██████▌ \| 463/711 [21:12<09:42, 2.35s/it] 65%\|██████▌ \| 464/711 [21:14<09:37, 2.34s/it] 65%\|██████▌ \| 465/711 [21:16<09:33, 2.33s/it] 66%\|██████▌ \| 466/711 [21:18<09:29, 2.32s/it] 66%\|██████▌ \| 467/711 [21:21<09:26, 2.32s/it] 66%\|██████▌ \| 468/
	0: {'loss': 0.4727, 'grad_norm': 0.7765476983805598, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.4}
	0: {'loss': 0.4761, 'grad_norm': 0.7166795921281778, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.4}
	0: 711 [21:23<09:23, 2.32s/it] 66%\|██████▌ \| 469/711 [21:25<09:26, 2.34s/it] 66%\|██████▌ \| 470/711 [21:28<09:22, 2.33s/it] 66%\|██████▌ \| 470/711 [21:28<09:22, 2.33s/it] 66%\|██████▌ \| 471/711 [21:30<09:19, 2.33s/it] 66%\|██████▋ \| 472/711 [21:32<09:16, 2.33s/it] 67%\|██████▋ \| 473/711 [21:35<09:13, 2.33s/it] 67%\|██████▋ \| 474/711 [21:37<09:11, 2.33s/it] 67%\|██████▋ \| 475/711 [21:39<09:08, 2.32s/it] 67%\|██████▋ \| 476/711 [21:42<09:10, 2.34s/it] 67%\|██████▋ \| 477/711 [21:44<09:06, 2.34s/it] 67%\|██████▋ \| 478/711 [21:46<09:07, 2.35s/it] 67%\|██████▋ \| 479/711 [21:49<09:06, 2.36s/it] 68%\|██████▊ \| 480/711 [21:51<09:02, 2.35s/it] 68%\|██████▊ \| 480/711 [21:51<09:02, 2.35s/i
	0: {'loss': 0.4545, 'grad_norm': 0.7592461340919713, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.41}
	0: t] 68%\|██████▊ \| 481/711 [21:54<09:03, 2.36s/it] 68%\|██████▊ \| 482/711 [21:56<08:58, 2.35s/it] 68%\|██████▊ \| 483/711 [21:58<08:53, 2.34s/it] 68%\|██████▊ \| 484/711 [22:01<08:49, 2.33s/it] 68%\|██████▊ \| 485/711 [22:03<08:58, 2.38s/it] 68%\|██████▊ \| 486/711 [22:05<08:50, 2.36s/it] 68%\|██████▊ \| 487/711 [22:08<08:45, 2.34s/it] 69%\|██████▊ \| 488/711 [22:10<08:40, 2.33s/it] 69%\|██████▉ \| 489/711 [22:12<08:41, 2.35s/it] 69%\|██████▉ \| 490/711 [22:15<08:37, 2.34s/it] 69%\|██████▉ \| 490/711 [22:15<08:37, 2.34s/it] 69%\|██████▉ \| 491/711 [22:17<08:37, 2.35s/it] 69%\|██████▉ \| 492/711 [22:19<08:33, 2.34s/it] 69%\|██████▉ \| 493/711 [22:22<08:29, 2.34s/it] 69%\|██████▉ \| 494/711 [22:24<08:35, 2.37s/it] 70%\|█�
	0: {'loss': 0.4621, 'grad_norm': 0.8060919908075219, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.42}
	0: �████▉ \| 495/711 [22:27<08:33, 2.38s/it] 70%\|██████▉ \| 496/711 [22:29<08:27, 2.36s/it] 70%\|██████▉ \| 497/711 [22:31<08:25, 2.36s/it] 70%\|███████ \| 498/711 [22:34<08:20, 2.35s/it] 70%\|███████ \| 499/711 [22:36<08:21, 2.37s/it] 70%\|███████ \| 500/711 [22:38<08:16, 2.35s/it] 70%\|███████ \| 500/711 [22:38<08:16, 2.35s/it] 70%\|███████ \| 501/711 [22:41<08:12, 2.34s/it] 71%\|███████ \| 502/711 [22:43<08:08, 2.34s/it] 71%\|███████ \| 503/711 [22:45<08:05, 2.33s/it] 71%\|███████ \| 504/711 [22:48<08:02, 2.33s/it] 71%\|███████ \| 505/711 [22:50<07:59, 2.33s/it] 71%\|███████ \| 506/711 [22:52<07:56, 2.32s/it] 71%\|███████▏ \| 507/711 [22:55<07:53, 2.32s/it] 71%\|███████▏ \| 508/711 [22:57<07:50, 2.32s/it] 72%\|████�
	0: {'loss': 0.4759, 'grad_norm': 0.7434049511511707, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.43}
	0: {'loss': 0.4643, 'grad_norm': 0.8519398991308196, 'learning_rate': 4.982258077957576e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.44}
	0: �██▏ \| 509/711 [22:59<07:47, 2.32s/it] 72%\|███████▏ \| 510/711 [23:02<07:49, 2.33s/it] 72%\|███████▏ \| 510/711 [23:02<07:49, 2.33s/it] 72%\|███████▏ \| 511/711 [23:04<07:52, 2.36s/it] 72%\|███████▏ \| 512/711 [23:06<07:59, 2.41s/it] 72%\|███████▏ \| 513/711 [23:09<08:02, 2.44s/it] 72%\|███████▏ \| 514/711 [23:11<07:53, 2.40s/it] 72%\|███████▏ \| 515/711 [23:14<07:45, 2.38s/it] 73%\|███████▎ \| 516/711 [23:16<07:39, 2.36s/it] 73%\|███████▎ \| 517/711 [23:18<07:35, 2.35s/it] 73%\|███████▎ \| 518/711 [23:21<07:31, 2.34s/it] 73%\|███████▎ \| 519/711 [23:23<07:27, 2.33s/it] 73%\|███████▎ \| 520/711 [23:25<07:28, 2.35s/it] 73%\|███████▎ \| 520/711 [23:25<07:28, 2.35s/it] 73%\|████
	0: {'loss': 0.4652, 'grad_norm': 0.7323364456399427, 'learning_rate': 4.910660792773122e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.45}
	0: ███▎ \| 521/711 [23:28<07:24, 2.34s/it] 73%\|███████▎ \| 522/711 [23:30<07:20, 2.33s/it] 74%\|███████▎ \| 523/711 [23:32<07:18, 2.33s/it] 74%\|███████▎ \| 524/711 [23:35<07:18, 2.34s/it] 74%\|███████▍ \| 525/711 [23:37<07:18, 2.36s/it] 74%\|███████▍ \| 526/711 [23:39<07:17, 2.37s/it] 74%\|███████▍ \| 527/711 [23:42<07:13, 2.36s/it] 74%\|███████▍ \| 528/711 [23:44<07:11, 2.36s/it] 74%\|███████▍ \| 529/711 [23:46<07:06, 2.34s/it] 75%\|███████▍ \| 530/711 [23:49<07:03, 2.34s/it] 75%\|███████▍ \| 530/711 [23:49<07:03, 2.34s/it] 75%\|███████▍ \| 531/711 [23:51<07:00, 2.33s/it] 75%\|███████▍ \| 532/711 [23:54<07:07, 2.39s/it] 75%\|███████▍ \| 533/711 [23:56<07:00, 2.36s/it] 75%\|███████▌ \| 534/711 [23:58<06:55, 2.35s/it]
	0: {'loss': 0.4762, 'grad_norm': 0.7467850320594309, 'learning_rate': 4.7858608680485444e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.46}
	0: 75%\|███████▌ \| 535/711 [24:00<06:50, 2.33s/it] 75%\|███████▌ \| 536/711 [24:03<06:49, 2.34s/it] 76%\|███████▌ \| 537/711 [24:05<06:45, 2.33s/it] 76%\|███████▌ \| 538/711 [24:07<06:41, 2.32s/it] 76%\|███████▌ \| 539/711 [24:10<06:38, 2.32s/it] 76%\|███████▌ \| 540/711 [24:12<06:35, 2.31s/it] 76%\|███████▌ \| 540/711 [24:12<06:35, 2.31s/it] 76%\|███████▌ \| 541/711 [24:14<06:36, 2.33s/it] 76%\|███████▌ \| 542/711 [24:17<06:35, 2.34s/it] 76%\|███████▋ \| 543/711 [24:19<06:33, 2.34s/it] 77%\|███████▋ \| 544/711 [24:21<06:29, 2.33s/it] 77%\|███████▋ \| 545/711 [24:24<06:26, 2.33s/it] 77%\|███████▋ \| 546/711 [24:26<06:23, 2.32s/it] 77%\|███████▋ \| 547/711 [24:28<06:20, 2.32s/it] 77%\|███████▋ \| 548/711 [24:31<0
	0: {'loss': 0.474, 'grad_norm': 0.8194845306669991, 'learning_rate': 4.610931292117764e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.46}
	0: {'loss': 0.4615, 'grad_norm': 1.4968465915839815, 'learning_rate': 4.390179411698176e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.47}
	0: 6:18, 2.32s/it] 77%\|███████▋ \| 549/711 [24:33<06:15, 2.32s/it] 77%\|███████▋ \| 550/711 [24:35<06:13, 2.32s/it] 77%\|███████▋ \| 550/711 [24:35<06:13, 2.32s/it] 77%\|███████▋ \| 551/711 [24:38<06:10, 2.32s/it] 78%\|███████▊ \| 552/711 [24:40<06:08, 2.32s/it] 78%\|███████▊ \| 553/711 [24:42<06:05, 2.32s/it] 78%\|███████▊ \| 554/711 [24:45<06:03, 2.31s/it] 78%\|███████▊ \| 555/711 [24:47<06:01, 2.32s/it] 78%\|███████▊ \| 556/711 [24:49<05:58, 2.32s/it] 78%\|███████▊ \| 557/711 [24:52<06:01, 2.35s/it] 78%\|███████▊ \| 558/711 [24:54<06:03, 2.38s/it] 79%\|███████▊ \| 559/711 [24:56<05:59, 2.37s/it] 79%\|███████▉ \| 560/711 [24:59<05:55, 2.35s/it] 79%\|███████▉ \| 560/711 [24:59
	0: {'loss': 0.4737, 'grad_norm': 0.7375795650774891, 'learning_rate': 4.129040870719198e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.48}
	0: <05:55, 2.35s/it] 79%\|███████▉ \| 561/711 [25:01<05:51, 2.34s/it] 79%\|███████▉ \| 562/711 [25:03<05:47, 2.33s/it] 79%\|███████▉ \| 563/711 [25:06<05:47, 2.35s/it] 79%\|███████▉ \| 564/711 [25:08<05:43, 2.34s/it] 79%\|███████▉ \| 565/711 [25:11<05:48, 2.39s/it] 80%\|███████▉ \| 566/711 [25:13<05:43, 2.37s/it] 80%\|███████▉ \| 567/711 [25:15<05:41, 2.37s/it] 80%\|███████▉ \| 568/711 [25:18<05:36, 2.35s/it] 80%\|████████ \| 569/711 [25:20<05:32, 2.34s/it] 80%\|████████ \| 570/711 [25:22<05:28, 2.33s/it] 80%\|████████ \| 570/711 [25:22<05:28, 2.33s/it] 80%\|████████ \| 571/711 [25:24<05:25, 2.33s/it] 80%\|████████ \| 572/711 [25:27<05:22, 2.32s/it] 81%\|████████ \| 573/711 [25:29<05:22, 2.34s/it] 81%\|████████
	0: {'loss': 0.4643, 'grad_norm': 0.7331473812508006, 'learning_rate': 3.833945766728859e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.49}
	0: \| 574/711 [25:32<05:21, 2.35s/it] 81%\|████████ \| 575/711 [25:34<05:18, 2.34s/it] 81%\|████████ \| 576/711 [25:36<05:15, 2.33s/it] 81%\|████████ \| 577/711 [25:39<05:15, 2.35s/it] 81%\|████████▏ \| 578/711 [25:41<05:13, 2.36s/it] 81%\|████████▏ \| 579/711 [25:43<05:09, 2.35s/it] 82%\|████████▏ \| 580/711 [25:46<05:13, 2.40s/it] 82%\|████████▏ \| 580/711 [25:46<05:13, 2.40s/it] 82%\|████████▏ \| 581/711 [25:48<05:08, 2.37s/it] 82%\|████████▏ \| 582/711 [25:50<05:03, 2.36s/it] 82%\|████████▏ \| 583/711 [25:53<05:00, 2.35s/it] 82%\|████████▏ \| 584/711 [25:55<04:56, 2.34s/it] 82%\|████████▏ \| 585/711 [25:57<04:53, 2.33s/it] 82%\|████████▏ \| 586/711 [26:00<04:50, 2.32s/it] 83%\|████████▎ \| 587/711 [26:02<04:50, 2
	0: {'loss': 0.4559, 'grad_norm': 0.7238676692923376, 'learning_rate': 3.512160320551906e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.5}
	0: .34s/it] 83%\|████████▎ \| 588/711 [26:04<04:48, 2.35s/it] 83%\|████████▎ \| 589/711 [26:07<04:47, 2.36s/it] 83%\|████████▎ \| 590/711 [26:09<04:47, 2.38s/it] 83%\|████████▎ \| 590/711 [26:09<04:47, 2.38s/it] 83%\|████████▎ \| 591/711 [26:12<04:43, 2.36s/it] 83%\|████████▎ \| 592/711 [26:14<04:39, 2.35s/it] 83%\|████████▎ \| 593/711 [26:16<04:36, 2.34s/it] 84%\|████████▎ \| 594/711 [26:19<04:33, 2.33s/it] 84%\|████████▎ \| 595/711 [26:21<04:30, 2.33s/it] 84%\|████████▍ \| 596/711 [26:23<04:27, 2.33s/it] 84%\|████████▍ \| 597/711 [26:25<04:25, 2.32s/it] 84%\|████████▍ \| 598/711 [26:28<04:22, 2.32s/it] 84%\|████████▍ \| 599/711 [26:30<04:20, 2.32s/it] 84%\|████████▍ \| 600/711 [26:32<04:17, 2.32s/it]
	0: {'loss': 0.4728, 'grad_norm': 0.7685550630490101, 'learning_rate': 3.171607957817881e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.51}
	0: {'loss': 0.452, 'grad_norm': 0.7387438977228977, 'learning_rate': 2.820674207925789e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.51}
	0: 84%\|████████▍ \| 600/711 [26:32<04:17, 2.32s/it] 85%\|████████▍ \| 601/711 [26:35<04:15, 2.32s/it] 85%\|████████▍ \| 602/711 [26:37<04:13, 2.32s/it] 85%\|████████▍ \| 603/711 [26:39<04:10, 2.32s/it] 85%\|████████▍ \| 604/711 [26:42<04:08, 2.32s/it] 85%\|████████▌ \| 605/711 [26:44<04:08, 2.34s/it] 85%\|████████▌ \| 606/711 [26:46<04:06, 2.35s/it] 85%\|████████▌ \| 607/711 [26:49<04:03, 2.34s/it] 86%\|████████▌ \| 608/711 [26:51<04:00, 2.33s/it] 86%\|████████▌ \| 609/711 [26:53<03:57, 2.33s/it] 86%\|████████▌ \| 610/711 [26:56<03:55, 2.33s/it] 86%\|████████▌ \| 610/711 [26:56<03:55, 2.33s/it] 86%\|████████▌ \| 611/711 [26:58<03:52, 2.32s/it] 86%\|████████▌ \| 612/711 [27:00<
	0: {'loss': 0.4531, 'grad_norm': 0.7268143927347638, 'learning_rate': 2.4680002244803154e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.52}
	0: 03:49, 2.32s/it] 86%\|████████▌ \| 613/711 [27:03<03:47, 2.32s/it] 86%\|████████▋ \| 614/711 [27:05<03:44, 2.32s/it] 86%\|████████▋ \| 615/711 [27:07<03:42, 2.32s/it] 87%\|████████▋ \| 616/711 [27:10<03:40, 2.32s/it] 87%\|████████▋ \| 617/711 [27:12<03:37, 2.32s/it] 87%\|████████▋ \| 618/711 [27:14<03:38, 2.35s/it] 87%\|████████▋ \| 619/711 [27:17<03:34, 2.34s/it] 87%\|████████▋ \| 620/711 [27:19<03:32, 2.33s/it] 87%\|████████▋ \| 620/711 [27:19<03:32, 2.33s/it] 87%\|████████▋ \| 621/711 [27:21<03:31, 2.35s/it] 87%\|████████▋ \| 622/711 [27:24<03:29, 2.36s/it] 88%\|████████▊ \| 623/711 [27:26<03:28, 2.36s/it] 88%\|████████▊ \| 624/711 [27:28<03:24, 2.35s/it] 88%\|████████▊ \| 625/711 [27:31<03:21, 2.34s/it] 88
	0: {'loss': 0.4502, 'grad_norm': 0.6625178435258192, 'learning_rate': 2.1222700114117344e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.53}
	0: %\|████████▊ \| 626/711 [27:33<03:18, 2.33s/it] 88%\|████████▊ \| 627/711 [27:35<03:15, 2.33s/it] 88%\|████████▊ \| 628/711 [27:38<03:12, 2.32s/it] 88%\|████████▊ \| 629/711 [27:40<03:10, 2.32s/it] 89%\|████████▊ \| 630/711 [27:42<03:10, 2.35s/it] 89%\|████████▊ \| 630/711 [27:42<03:10, 2.35s/it] 89%\|████████▊ \| 631/711 [27:45<03:07, 2.35s/it] 89%\|████████▉ \| 632/711 [27:47<03:09, 2.39s/it] 89%\|████████▉ \| 633/711 [27:50<03:04, 2.37s/it] 89%\|████████▉ \| 634/711 [27:52<03:00, 2.35s/it] 89%\|████████▉ \| 635/711 [27:54<02:57, 2.34s/it] 89%\|████████▉ \| 636/711 [27:57<02:56, 2.35s/it] 90%\|████████▉ \| 637/711 [27:59<02:55, 2.37s/it] 90%\|████████▉ \| 638/711 [28:01<02:52, 2.36s/it] 90%\|██████�
	0: {'loss': 0.455, 'grad_norm': 0.7830830598414732, 'learning_rate': 1.7919965939785867e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.54}
	0: {'loss': 0.4578, 'grad_norm': 0.6885412177331103, 'learning_rate': 1.4853123998327068e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.55}
	0: ��█▉ \| 639/711 [28:04<02:49, 2.35s/it] 90%\|█████████ \| 640/711 [28:06<02:46, 2.34s/it] 90%\|█████████ \| 640/711 [28:06<02:46, 2.34s/it] 90%\|█████████ \| 641/711 [28:08<02:43, 2.33s/it] 90%\|█████████ \| 642/711 [28:11<02:40, 2.33s/it] 90%\|█████████ \| 643/711 [28:13<02:38, 2.33s/it] 91%\|█████████ \| 644/711 [28:15<02:35, 2.33s/it] 91%\|█████████ \| 645/711 [28:18<02:35, 2.35s/it] 91%\|█████████ \| 646/711 [28:20<02:32, 2.34s/it] 91%\|█████████ \| 647/711 [28:22<02:29, 2.34s/it] 91%\|█████████ \| 648/711 [28:25<02:27, 2.33s/it] 91%\|█████████▏\| 649/711 [28:27<02:24, 2.33s/it] 91%\|█████████▏\| 650/711 [28:29<02:23, 2.35s/it] 91%\|█████████▏\| 650/711 [28:29<02:23,
	0: {'loss': 0.4629, 'grad_norm': 0.7008874020320417, 'learning_rate': 1.2097690116604504e-06, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.56}
	0: 2.35s/it] 92%\|█████████▏\| 651/711 [28:32<02:20, 2.34s/it] 92%\|█████████▏\| 652/711 [28:34<02:17, 2.33s/it] 92%\|█████████▏\| 653/711 [28:36<02:17, 2.37s/it] 92%\|█████████▏\| 654/711 [28:39<02:14, 2.36s/it] 92%\|█████████▏\| 655/711 [28:41<02:11, 2.35s/it] 92%\|█████████▏\| 656/711 [28:44<02:10, 2.37s/it] 92%\|█████████▏\| 657/711 [28:46<02:06, 2.35s/it] 93%\|█████████▎\| 658/711 [28:48<02:04, 2.34s/it] 93%\|█████████▎\| 659/711 [28:51<02:01, 2.33s/it] 93%\|█████████▎\| 660/711 [28:53<01:58, 2.33s/it] 93%\|█████████▎\| 660/711 [28:53<01:58, 2.33s/it] 93%\|█████████▎\| 661/711 [28:55<01:57, 2.35s/it] 93%\|█████████▎\| 662/711 [28:58<01:54, 2.34s/it] 93%\|█████████▎\| 663/711 [29:00
	0: {'loss': 0.4435, 'grad_norm': 0.698029654257462, 'learning_rate': 9.721512221546967e-07, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.56}
	0: <01:51, 2.33s/it] 93%\|█████████▎\| 664/711 [29:02<01:49, 2.33s/it] 94%\|█████████▎\| 665/711 [29:05<01:46, 2.32s/it] 94%\|█████████▎\| 666/711 [29:07<01:44, 2.32s/it] 94%\|█████████▍\| 667/711 [29:09<01:42, 2.32s/it] 94%\|█████████▍\| 668/711 [29:11<01:39, 2.32s/it] 94%\|█████████▍\| 669/711 [29:14<01:38, 2.34s/it] 94%\|█████████▍\| 670/711 [29:16<01:36, 2.36s/it] 94%\|█████████▍\| 670/711 [29:16<01:36, 2.36s/it] 94%\|█████████▍\| 671/711 [29:19<01:34, 2.35s/it] 95%\|█████████▍\| 672/711 [29:21<01:31, 2.34s/it] 95%\|█████████▍\| 673/711 [29:23<01:28, 2.34s/it] 95%\|█████████▍\| 674/711 [29:26<01:26, 2.33s/it] 95%\|█████████▍\| 675/711 [29:28<01:24, 2.33s/it] 95%\|█████████▌\| 676/711
	0: {'loss': 0.4649, 'grad_norm': 0.7032914493453137, 'learning_rate': 7.783099699013075e-07, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.57}
	0: [29:30<01:21, 2.33s/it] 95%\|█████████▌\| 677/711 [29:33<01:19, 2.33s/it] 95%\|█████████▌\| 678/711 [29:35<01:16, 2.32s/it] 95%\|█████████▌\| 679/711 [29:37<01:14, 2.32s/it] 96%\|█████████▌\| 680/711 [29:39<01:11, 2.32s/it] 96%\|█████████▌\| 680/711 [29:39<01:11, 2.32s/it] 96%\|█████████▌\| 681/711 [29:42<01:09, 2.32s/it] 96%\|█████████▌\| 682/711 [29:44<01:08, 2.36s/it] 96%\|█████████▌\| 683/711 [29:47<01:05, 2.35s/it] 96%\|█████████▌\| 684/711 [29:49<01:03, 2.34s/it] 96%\|█████████▋\| 685/711 [29:51<01:01, 2.36s/it] 96%\|█████████▋\| 686/711 [29:54<00:58, 2.36s/it] 97%\|█████████▋\| 687/711 [29:56<00:56, 2.35s/it] 97%\|█████████▋\| 688/711 [29:58<00:53, 2.34s/it] 97%\|█████████▋\|
	0: {'loss': 0.4638, 'grad_norm': 0.7522821052393728, 'learning_rate': 6.330182698529928e-07, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.58}
	0: {'loss': 0.456, 'grad_norm': 0.6485448600183656, 'learning_rate': 5.398536858604507e-07, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.59}
	0: 689/711 [30:01<00:51, 2.33s/it] 97%\|█████████▋\| 690/711 [30:03<00:48, 2.32s/it] 97%\|█████████▋\| 690/711 [30:03<00:48, 2.32s/it] 97%\|█████████▋\| 691/711 [30:05<00:46, 2.34s/it] 97%\|█████████▋\| 692/711 [30:08<00:44, 2.33s/it] 97%\|█████████▋\| 693/711 [30:10<00:41, 2.33s/it] 98%\|█████████▊\| 694/711 [30:12<00:39, 2.33s/it] 98%\|█████████▊\| 695/711 [30:15<00:37, 2.32s/it] 98%\|█████████▊\| 696/711 [30:17<00:34, 2.32s/it] 98%\|█████████▊\| 697/711 [30:19<00:32, 2.32s/it] 98%\|█████████▊\| 698/711 [30:22<00:30, 2.32s/it] 98%\|█████████▊\| 699/711 [30:24<00:27, 2.32s/it] 98%\|█████████▊\| 700/711 [30:26<00:25, 2.34s/it] 98%\|█████████▊\| 700/711 [30:
	0: {'loss': 0.4527, 'grad_norm': 0.6894880207361598, 'learning_rate': 5.011102391771039e-07, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.6}
	0: [2025-11-24 00:42:38,710] [INFO] [axolotl.core.trainers.base._save:613] [PID:1912876] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0/checkpoint-711[39m
	0: [2025-11-24 00:42:45,711] [INFO] [axolotl.core.trainers.base._save:662] [PID:1912876] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`[39m
	0: {'train_runtime': 1863.7709, 'train_samples_per_second': 6.104, 'train_steps_per_second': 0.381, 'train_loss': 0.49791947950290727, 'memory/max_mem_active(gib)': 58.47, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 68.71, 'epoch': 0.6}
	0: 26<00:25, 2.34s/it] 99%\|█████████▊\| 701/711 [30:29<00:23, 2.37s/it] 99%\|█████████▊\| 702/711 [30:31<00:21, 2.37s/it] 99%\|█████████▉\| 703/711 [30:33<00:18, 2.35s/it] 99%\|█████████▉\| 704/711 [30:36<00:16, 2.35s/it] 99%\|█████████▉\| 705/711 [30:38<00:14, 2.36s/it] 99%\|█████████▉\| 706/711 [30:40<00:11, 2.34s/it] 99%\|█████████▉\| 707/711 [30:43<00:09, 2.42s/it] 100%\|█████████▉\| 708/711 [30:45<00:07, 2.39s/it] 100%\|█████████▉\| 709/711 [30:48<00:04, 2.47s/it] 100%\|█████████▉\| 710/711 [30:51<00:02, 2.57s/it] 100%\|█████████▉\| 710/711 [30:51<00:02, 2.57s/it] 100%\|██████████\| 711/711 [30:53<00:00, 2.49s/it] 100%\|██████████\| 711/711 [31:03<00:00, 2
	0: .49s/it] 100%\|██████████\| 711/711 [31:03<00:00, 2.62s/it]
	0: [2025-11-24 00:42:49,004] [INFO] [axolotl.train.save_trained_model:228] [PID:1912876] [RANK:0] Training completed! Saving trained model to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.[39m
	0: [2025-11-24 00:42:51,675] [INFO] [axolotl.core.trainers.base._save:613] [PID:1912876] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0[39m
	0: [2025-11-24 00:42:58,432] [INFO] [axolotl.core.trainers.base._save:662] [PID:1912876] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`[39m
	0: [2025-11-24 00:42:59,125] [INFO] [axolotl.train.save_trained_model:350] [PID:1912876] [RANK:0] Model successfully saved to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0[39m