3: W1122 23:27:22.837000 3329954 torch/distributed/run.py:792] 3: W1122 23:27:22.837000 3329954 torch/distributed/run.py:792] ***************************************** 3: W1122 23:27:22.837000 3329954 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 3: W1122 23:27:22.837000 3329954 torch/distributed/run.py:792] ***************************************** 0: W1122 23:27:22.840000 2438387 torch/distributed/run.py:792] 0: W1122 23:27:22.840000 2438387 torch/distributed/run.py:792] ***************************************** 0: W1122 23:27:22.840000 2438387 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 0: W1122 23:27:22.840000 2438387 torch/distributed/run.py:792] ***************************************** 1: W1122 23:27:22.870000 1819711 torch/distributed/run.py:792] 1: W1122 23:27:22.870000 1819711 torch/distributed/run.py:792] ***************************************** 1: W1122 23:27:22.870000 1819711 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 1: W1122 23:27:22.870000 1819711 torch/distributed/run.py:792] ***************************************** 2: W1122 23:27:22.894000 347335 torch/distributed/run.py:792] 2: W1122 23:27:22.894000 347335 torch/distributed/run.py:792] ***************************************** 2: W1122 23:27:22.894000 347335 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 2: W1122 23:27:22.894000 347335 torch/distributed/run.py:792] ***************************************** 3: [2025-11-22 23:27:42,451] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:3330029] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing` 3: [2025-11-22 23:27:42,451] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:3330029] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing 2: [2025-11-22 23:27:42,575] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:347410] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing` 2: [2025-11-22 23:27:42,575] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:347410] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing 0: [2025-11-22 23:27:42,589] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:2438464] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing` 0: [2025-11-22 23:27:42,590] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:2438464] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing 1: [2025-11-22 23:27:42,748] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:1819786] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing` 1: [2025-11-22 23:27:42,748] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:1819786] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing 0: [2025-11-22 23:27:45,918] [WARNING] [axolotl.utils.config.normalize_config:139] [PID:2438464] [RANK:0] Invalid value for save_steps (1.6666666666666667) from saves_per_epoch and/or num_epochs. Saving at training end only. 0: [2025-11-22 23:27:45,928] [INFO] [axolotl.cli.config.load_cfg:245] [PID:2438464] [RANK:0] config: 0: { 0: "activation_offloading": false, 0: "auto_resume_from_checkpoints": true, 0: "axolotl_config_path": "/lustre/fswork/projects/rech/dgo/udv55np/train/tmp/1763850438048925933.yaml", 0: "base_model": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-1b", 0: "base_model_config": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-1b", 0: "batch_size": 16, 0: "bf16": true, 0: "capabilities": { 0: "bf16": true, 0: "compute_capability": "sm_90", 0: "fp8": false, 0: "n_gpu": 16, 0: "n_node": 1 0: }, 0: "chat_template": "gemma3", 0: "context_parallel_size": 1, 0: "curriculum_sampling": true, 0: "dataloader_num_workers": 2, 0: "dataset_prepared_path": "/lustre/fswork/projects/rech/dgo/udv55np/dataset_gemma/Nemotron-Super-49B-v1_5/split_1", 0: "dataset_processes": 32, 0: "datasets": [ 0: { 0: "chat_template": "tokenizer_default", 0: "data_files": [ 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0007.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0009.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0005.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0006.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0014.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0010.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0012.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0008.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0001.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0002.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0013.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0015.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0004.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0011.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0000.jsonl", 0: "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0003.jsonl" 0: ], 0: "ds_type": "json", 0: "field_messages": "conversations", 0: "message_property_mappings": { 0: "content": "content", 0: "role": "role" 0: }, 0: "path": "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking", 0: "trust_remote_code": false, 0: "type": "chat_template" 0: } 0: ], 0: "ddp": true, 0: "deepspeed": { 0: "bf16": { 0: "enabled": true 0: }, 0: "gradient_accumulation_steps": "auto", 0: "gradient_clipping": "auto", 0: "train_batch_size": "auto", 0: "train_micro_batch_size_per_gpu": "auto", 0: "wall_clock_breakdown": false, 0: "zero_optimization": { 0: "contiguous_gradients": true, 0: "overlap_comm": true, 0: "reduce_bucket_size": "auto", 0: "stage": 3, 0: "stage3_gather_16bit_weights_on_model_save": true, 0: "stage3_param_persistence_threshold": "auto", 0: "stage3_prefetch_bucket_size": "auto", 0: "sub_group_size": 0 0: } 0: }, 0: "device": "cuda:0", 0: "device_map": { 0: "": 0 0: }, 0: "dion_rank_fraction": 1.0, 0: "dion_rank_multiple_of": 1, 0: "env_capabilities": { 0: "torch_version": "2.6.0" 0: }, 0: "eot_tokens": [ 0: "" 0: ], 0: "eval_batch_size": 1, 0: "eval_causal_lm_metrics": [ 0: "sacrebleu", 0: "comet", 0: "ter", 0: "chrf" 0: ], 0: "eval_max_new_tokens": 128, 0: "eval_sample_packing": true, 0: "eval_table_size": 0, 0: "evals_per_epoch": 0, 0: "flash_attention": true, 0: "fp16": false, 0: "gradient_accumulation_steps": 1, 0: "gradient_checkpointing": true, 0: "gradient_checkpointing_kwargs": { 0: "use_reentrant": true 0: }, 0: "learning_rate": 2e-05, 0: "lisa_layers_attribute": "model.layers", 0: "load_best_model_at_end": false, 0: "load_in_4bit": false, 0: "load_in_8bit": false, 0: "local_rank": 0, 0: "logging_steps": 10, 0: "lora_dropout": 0.0, 0: "loraplus_lr_embedding": 1e-06, 0: "lr_scheduler": "warmup_stable_decay", 0: "lr_scheduler_kwargs": { 0: "min_lr_ratio": 0.1, 0: "num_decay_steps": 200 0: }, 0: "max_prompt_len": 512, 0: "mean_resizing_embeddings": false, 0: "micro_batch_size": 1, 0: "model_config_type": "gemma3_text", 0: "num_epochs": 0.6, 0: "optimizer": "adamw_torch_fused", 0: "output_dir": "/lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-1b/1", 0: "pad_to_sequence_len": true, 0: "pretrain_multipack_attn": true, 0: "pretrain_multipack_buffer_size": 10000, 0: "profiler_steps_start": 0, 0: "qlora_sharded_model_loading": false, 0: "ray_num_workers": 1, 0: "resources_per_worker": { 0: "GPU": 1 0: }, 0: "sample_packing": true, 0: "sample_packing_bin_size": 200, 0: "sample_packing_group_size": 100000, 0: "sample_packing_sequentially": true, 0: "save_only_model": true, 0: "save_safetensors": true, 0: "save_total_limit": 20, 0: "saves_per_epoch": 1, 0: "sequence_len": 16384, 0: "shuffle_before_merging_datasets": true, 0: "shuffle_merged_datasets": false, 0: "skip_prepare_dataset": false, 0: "strict": false, 0: "tensor_parallel_size": 1, 0: "tf32": false, 0: "tiled_mlp_use_original_mlp": true, 0: "tokenizer_config": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-27b", 0: "torch_dtype": "torch.bfloat16", 0: "train_on_inputs": false, 0: "trl": { 0: "log_completions": false, 0: "mask_truncated_completions": false, 0: "ref_model_mixup_alpha": 0.9, 0: "ref_model_sync_steps": 64, 0: "scale_rewards": true, 0: "sync_ref_model": false, 0: "use_vllm": false, 0: "vllm_server_host": "0.0.0.0", 0: "vllm_server_port": 8000 0: }, 0: "use_ray": false, 0: "use_tensorboard": true, 0: "val_set_size": 0.0, 0: "vllm": { 0: "device": "auto", 0: "dtype": "auto", 0: "gpu_memory_utilization": 0.9, 0: "host": "0.0.0.0", 0: "port": 8000 0: }, 0: "warmup_steps": 100, 0: "weight_decay": 0.0, 0: "world_size": 16 0: } 0: [2025-11-22 23:27:45,929] [INFO] [axolotl.cli.checks.check_user_token:35] [PID:2438464] [RANK:0] Skipping HuggingFace token verification because HF_HUB_OFFLINE is set to True. Only local files will be used. 0: [2025-11-22 23:27:47,221] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:472] [PID:2438464] [RANK:0] Loading prepared dataset from disk at /lustre/fswork/projects/rech/dgo/udv55np/dataset_gemma/Nemotron-Super-49B-v1_5/split_1/9c9c4c535c90616e08bd1b7f6d00d00a... 0: jzxh275:2438464:2438464 [0] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.73<0> 0: jzxh275:2438464:2438464 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 0: jzxh275:2438464:2438464 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 0: jzxh275:2438464:2438464 [0] NCCL INFO NET/Plugin: Using internal network plugin. 0: jzxh275:2438464:2438464 [0] NCCL INFO cudaDriverVersion 12080 0: NCCL version 2.21.5+cuda12.4 0: jzxh275:2438464:2438464 [0] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438465:2438465 [1] NCCL INFO cudaDriverVersion 12080 0: jzxh275:2438465:2438465 [1] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.73<0> 0: jzxh275:2438465:2438465 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 0: jzxh275:2438465:2438465 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 0: jzxh275:2438465:2438465 [1] NCCL INFO NET/Plugin: Using internal network plugin. 0: jzxh275:2438465:2438465 [1] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438467:2438467 [3] NCCL INFO cudaDriverVersion 12080 0: jzxh275:2438466:2438466 [2] NCCL INFO cudaDriverVersion 12080 1: jzxh276:1819786:1819786 [0] NCCL INFO cudaDriverVersion 12080 0: jzxh275:2438467:2438467 [3] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.73<0> 0: jzxh275:2438466:2438466 [2] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.73<0> 0: jzxh275:2438467:2438467 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 0: jzxh275:2438466:2438466 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 0: jzxh275:2438467:2438467 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 0: jzxh275:2438467:2438467 [3] NCCL INFO NET/Plugin: Using internal network plugin. 0: jzxh275:2438466:2438466 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 0: jzxh275:2438466:2438466 [2] NCCL INFO NET/Plugin: Using internal network plugin. 2: jzxh277:347412:347412 [2] NCCL INFO cudaDriverVersion 12080 0: jzxh275:2438467:2438467 [3] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438466:2438466 [2] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330029:3330029 [0] NCCL INFO cudaDriverVersion 12080 1: jzxh276:1819789:1819789 [3] NCCL INFO cudaDriverVersion 12080 2: jzxh277:347413:347413 [3] NCCL INFO cudaDriverVersion 12080 2: jzxh277:347412:347412 [2] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.81<0> 2: jzxh277:347412:347412 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 2: jzxh277:347412:347412 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 2: jzxh277:347412:347412 [2] NCCL INFO NET/Plugin: Using internal network plugin. 3: jzxh278:3330029:3330029 [0] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.85<0> 2: jzxh277:347413:347413 [3] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.81<0> 3: jzxh278:3330029:3330029 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 2: jzxh277:347413:347413 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 2: jzxh277:347413:347413 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 2: jzxh277:347413:347413 [3] NCCL INFO NET/Plugin: Using internal network plugin. 3: jzxh278:3330029:3330029 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 3: jzxh278:3330029:3330029 [0] NCCL INFO NET/Plugin: Using internal network plugin. 1: jzxh276:1819786:1819786 [0] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.77<0> 1: jzxh276:1819789:1819789 [3] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.77<0> 3: jzxh278:3330030:3330030 [1] NCCL INFO cudaDriverVersion 12080 1: jzxh276:1819786:1819786 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 1: jzxh276:1819789:1819789 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 1: jzxh276:1819786:1819786 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 1: jzxh276:1819786:1819786 [0] NCCL INFO NET/Plugin: Using internal network plugin. 1: jzxh276:1819789:1819789 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 1: jzxh276:1819789:1819789 [3] NCCL INFO NET/Plugin: Using internal network plugin. 2: jzxh277:347411:347411 [1] NCCL INFO cudaDriverVersion 12080 2: jzxh277:347412:347412 [2] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330032:3330032 [3] NCCL INFO cudaDriverVersion 12080 3: jzxh278:3330029:3330029 [0] NCCL INFO Comm config Blocking set to 1 1: jzxh276:1819787:1819787 [1] NCCL INFO cudaDriverVersion 12080 2: jzxh277:347410:347410 [0] NCCL INFO cudaDriverVersion 12080 3: jzxh278:3330030:3330030 [1] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.85<0> 2: jzxh277:347413:347413 [3] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330032:3330032 [3] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.85<0> 1: jzxh276:1819787:1819787 [1] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.77<0> 3: jzxh278:3330030:3330030 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 2: jzxh277:347411:347411 [1] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.81<0> 1: jzxh276:1819788:1819788 [2] NCCL INFO cudaDriverVersion 12080 2: jzxh277:347410:347410 [0] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.81<0> 3: jzxh278:3330030:3330030 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 3: jzxh278:3330030:3330030 [1] NCCL INFO NET/Plugin: Using internal network plugin. 3: jzxh278:3330032:3330032 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 3: jzxh278:3330032:3330032 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 3: jzxh278:3330032:3330032 [3] NCCL INFO NET/Plugin: Using internal network plugin. 1: jzxh276:1819786:1819786 [0] NCCL INFO Comm config Blocking set to 1 1: jzxh276:1819787:1819787 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 1: jzxh276:1819787:1819787 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 1: jzxh276:1819787:1819787 [1] NCCL INFO NET/Plugin: Using internal network plugin. 1: jzxh276:1819789:1819789 [3] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330031:3330031 [2] NCCL INFO cudaDriverVersion 12080 2: jzxh277:347411:347411 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 2: jzxh277:347411:347411 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 2: jzxh277:347411:347411 [1] NCCL INFO NET/Plugin: Using internal network plugin. 2: jzxh277:347410:347410 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 2: jzxh277:347410:347410 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 2: jzxh277:347410:347410 [0] NCCL INFO NET/Plugin: Using internal network plugin. 1: jzxh276:1819788:1819788 [2] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.77<0> 3: jzxh278:3330031:3330031 [2] NCCL INFO Bootstrap : Using ibp24s0:10.100.8.85<0> 1: jzxh276:1819788:1819788 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 3: jzxh278:3330031:3330031 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) 1: jzxh276:1819788:1819788 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 1: jzxh276:1819788:1819788 [2] NCCL INFO NET/Plugin: Using internal network plugin. 3: jzxh278:3330031:3330031 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so 3: jzxh278:3330031:3330031 [2] NCCL INFO NET/Plugin: Using internal network plugin. 1: jzxh276:1819787:1819787 [1] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330030:3330030 [1] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330032:3330032 [3] NCCL INFO Comm config Blocking set to 1 2: jzxh277:347411:347411 [1] NCCL INFO Comm config Blocking set to 1 2: jzxh277:347410:347410 [0] NCCL INFO Comm config Blocking set to 1 1: jzxh276:1819788:1819788 [2] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330031:3330031 [2] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438464:2439061 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.73<0> 0: jzxh275:2438464:2439061 [0] NCCL INFO Using non-device net plugin version 0 0: jzxh275:2438464:2439061 [0] NCCL INFO Using network IB 0: jzxh275:2438465:2439062 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.73<0> 0: jzxh275:2438465:2439062 [1] NCCL INFO Using non-device net plugin version 0 0: jzxh275:2438465:2439062 [1] NCCL INFO Using network IB 2: jzxh277:347413:348009 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.81<0> 2: jzxh277:347413:348009 [3] NCCL INFO Using non-device net plugin version 0 2: jzxh277:347413:348009 [3] NCCL INFO Using network IB 1: jzxh276:1819787:1820384 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.77<0> 1: jzxh276:1819787:1820384 [1] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819787:1820384 [1] NCCL INFO Using network IB 0: jzxh275:2438466:2439064 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.73<0> 0: jzxh275:2438467:2439063 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.73<0> 0: jzxh275:2438466:2439064 [2] NCCL INFO Using non-device net plugin version 0 0: jzxh275:2438466:2439064 [2] NCCL INFO Using network IB 0: jzxh275:2438467:2439063 [3] NCCL INFO Using non-device net plugin version 0 0: jzxh275:2438467:2439063 [3] NCCL INFO Using network IB 2: jzxh277:347411:348010 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.81<0> 2: jzxh277:347412:348008 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.81<0> 2: jzxh277:347411:348010 [1] NCCL INFO Using non-device net plugin version 0 2: jzxh277:347411:348010 [1] NCCL INFO Using network IB 2: jzxh277:347412:348008 [2] NCCL INFO Using non-device net plugin version 0 2: jzxh277:347412:348008 [2] NCCL INFO Using network IB 2: jzxh277:347410:348011 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.81<0> 2: jzxh277:347410:348011 [0] NCCL INFO Using non-device net plugin version 0 2: jzxh277:347410:348011 [0] NCCL INFO Using network IB 3: jzxh278:3330030:3330624 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.85<0> 3: jzxh278:3330029:3330623 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.85<0> 3: jzxh278:3330030:3330624 [1] NCCL INFO Using non-device net plugin version 0 3: jzxh278:3330030:3330624 [1] NCCL INFO Using network IB 3: jzxh278:3330029:3330623 [0] NCCL INFO Using non-device net plugin version 0 3: jzxh278:3330029:3330623 [0] NCCL INFO Using network IB 3: jzxh278:3330031:3330626 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.85<0> 3: jzxh278:3330031:3330626 [2] NCCL INFO Using non-device net plugin version 0 3: jzxh278:3330031:3330626 [2] NCCL INFO Using network IB 3: jzxh278:3330032:3330625 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.85<0> 3: jzxh278:3330032:3330625 [3] NCCL INFO Using non-device net plugin version 0 3: jzxh278:3330032:3330625 [3] NCCL INFO Using network IB 1: jzxh276:1819789:1820383 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.77<0> 1: jzxh276:1819789:1820383 [3] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819789:1820383 [3] NCCL INFO Using network IB 1: jzxh276:1819786:1820382 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.77<0> 1: jzxh276:1819788:1820385 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.8.77<0> 1: jzxh276:1819786:1820382 [0] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819786:1820382 [0] NCCL INFO Using network IB 1: jzxh276:1819788:1820385 [2] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819788:1820385 [2] NCCL INFO Using network IB 2: jzxh277:347410:348011 [0] NCCL INFO ncclCommInitRank comm 0x564073e3d4b0 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x3645fd26de633f8b - Init START 2: jzxh277:347412:348008 [2] NCCL INFO ncclCommInitRank comm 0x55d761ef97e0 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x3645fd26de633f8b - Init START 2: jzxh277:347411:348010 [1] NCCL INFO ncclCommInitRank comm 0x5622e8e76730 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x3645fd26de633f8b - Init START 1: jzxh276:1819789:1820383 [3] NCCL INFO ncclCommInitRank comm 0x559de4393fb0 rank 7 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x3645fd26de633f8b - Init START 1: jzxh276:1819788:1820385 [2] NCCL INFO ncclCommInitRank comm 0x55a1a57d54c0 rank 6 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x3645fd26de633f8b - Init START 3: jzxh278:3330029:3330623 [0] NCCL INFO ncclCommInitRank comm 0x55c0e6c0e9d0 rank 12 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x3645fd26de633f8b - Init START 0: jzxh275:2438465:2439062 [1] NCCL INFO ncclCommInitRank comm 0x560d69039660 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x3645fd26de633f8b - Init START 0: jzxh275:2438464:2439061 [0] NCCL INFO ncclCommInitRank comm 0x55e4cbf1c410 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x3645fd26de633f8b - Init START 0: jzxh275:2438466:2439064 [2] NCCL INFO ncclCommInitRank comm 0x5624ea7d7d40 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x3645fd26de633f8b - Init START 0: jzxh275:2438467:2439063 [3] NCCL INFO ncclCommInitRank comm 0x559ed6e1be60 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x3645fd26de633f8b - Init START 2: jzxh277:347413:348009 [3] NCCL INFO ncclCommInitRank comm 0x56099934a0c0 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x3645fd26de633f8b - Init START 1: jzxh276:1819787:1820384 [1] NCCL INFO ncclCommInitRank comm 0x55c1cb2b1af0 rank 5 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x3645fd26de633f8b - Init START 1: jzxh276:1819786:1820382 [0] NCCL INFO ncclCommInitRank comm 0x5639df8e2620 rank 4 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x3645fd26de633f8b - Init START 3: jzxh278:3330032:3330625 [3] NCCL INFO ncclCommInitRank comm 0x564e400d5600 rank 15 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x3645fd26de633f8b - Init START 3: jzxh278:3330031:3330626 [2] NCCL INFO ncclCommInitRank comm 0x55b5c35aa0a0 rank 14 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x3645fd26de633f8b - Init START 3: jzxh278:3330030:3330624 [1] NCCL INFO ncclCommInitRank comm 0x563d7edaad10 rank 13 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x3645fd26de633f8b - Init START 2: jzxh277:347413:348009 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000 2: jzxh277:347413:348009 [3] NCCL INFO NVLS multicast support is not available on dev 3 0: jzxh275:2438466:2439064 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000 0: jzxh275:2438466:2439064 [2] NCCL INFO NVLS multicast support is not available on dev 2 0: jzxh275:2438465:2439062 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000 0: jzxh275:2438465:2439062 [1] NCCL INFO NVLS multicast support is not available on dev 1 2: jzxh277:347412:348008 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000 2: jzxh277:347412:348008 [2] NCCL INFO NVLS multicast support is not available on dev 2 0: jzxh275:2438464:2439061 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff 0: jzxh275:2438464:2439061 [0] NCCL INFO NVLS multicast support is not available on dev 0 0: jzxh275:2438467:2439063 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000 0: jzxh275:2438467:2439063 [3] NCCL INFO NVLS multicast support is not available on dev 3 3: jzxh278:3330032:3330625 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000 3: jzxh278:3330032:3330625 [3] NCCL INFO NVLS multicast support is not available on dev 3 2: jzxh277:347410:348011 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff 2: jzxh277:347410:348011 [0] NCCL INFO NVLS multicast support is not available on dev 0 2: jzxh277:347411:348010 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000 2: jzxh277:347411:348010 [1] NCCL INFO NVLS multicast support is not available on dev 1 1: jzxh276:1819789:1820383 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000 1: jzxh276:1819789:1820383 [3] NCCL INFO NVLS multicast support is not available on dev 3 1: jzxh276:1819788:1820385 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000 1: jzxh276:1819788:1820385 [2] NCCL INFO NVLS multicast support is not available on dev 2 3: jzxh278:3330030:3330624 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000 3: jzxh278:3330030:3330624 [1] NCCL INFO NVLS multicast support is not available on dev 1 1: jzxh276:1819787:1820384 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000 1: jzxh276:1819787:1820384 [1] NCCL INFO NVLS multicast support is not available on dev 1 1: jzxh276:1819786:1820382 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff 1: jzxh276:1819786:1820382 [0] NCCL INFO NVLS multicast support is not available on dev 0 3: jzxh278:3330029:3330623 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff 3: jzxh278:3330029:3330623 [0] NCCL INFO NVLS multicast support is not available on dev 0 3: jzxh278:3330031:3330626 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000 3: jzxh278:3330031:3330626 [2] NCCL INFO NVLS multicast support is not available on dev 2 0: jzxh275:2438464:2439061 [0] NCCL INFO comm 0x55e4cbf1c410 rank 0 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 2: jzxh277:347410:348011 [0] NCCL INFO comm 0x564073e3d4b0 rank 8 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 0: jzxh275:2438465:2439062 [1] NCCL INFO comm 0x560d69039660 rank 1 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 0: jzxh275:2438467:2439063 [3] NCCL INFO comm 0x559ed6e1be60 rank 3 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 00/16 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: jzxh275:2438466:2439064 [2] NCCL INFO comm 0x5624ea7d7d40 rank 2 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 01/16 : 0 3 2 5 4 7 6 9 8 11 10 13 12 15 14 1 2: jzxh277:347411:348010 [1] NCCL INFO comm 0x5622e8e76730 rank 9 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 02/16 : 0 3 6 5 4 7 10 9 8 11 14 13 12 15 2 1 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 03/16 : 0 1 2 7 4 5 6 11 8 9 10 15 12 13 14 3 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 04/16 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 05/16 : 0 3 2 5 4 7 6 9 8 11 10 13 12 15 14 1 0: jzxh275:2438465:2439062 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/9/-1->1->-1 [2] 3/-1/-1->1->0 [3] 0/-1/-1->1->3 [4] -1/-1/-1->1->2 [5] 3/9/-1->1->-1 [6] -1/-1/-1->1->3 [7] 0/-1/-1->1->2 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->5 [10] 3/-1/-1->1->0 [11] 0/-1/-1->1->3 [12] -1/-1/-1->1->2 [13] 3/-1/-1->1->5 [14] -1/-1/-1->1->3 [15] 0/-1/-1->1->2 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 06/16 : 0 3 6 5 4 7 10 9 8 11 14 13 12 15 2 1 0: jzxh275:2438465:2439062 [1] NCCL INFO P2P Chunksize set to 131072 2: jzxh277:347410:348011 [0] NCCL INFO Trees [0] 9/4/12->8->0 [1] -1/-1/-1->8->11 [2] 9/-1/-1->8->10 [3] 10/-1/-1->8->9 [4] 11/4/12->8->0 [5] 10/-1/-1->8->11 [6] 11/-1/-1->8->10 [7] -1/-1/-1->8->9 [8] 9/-1/-1->8->4 [9] -1/-1/-1->8->11 [10] 9/-1/-1->8->10 [11] 10/-1/-1->8->9 [12] 11/-1/-1->8->4 [13] 10/-1/-1->8->11 [14] 11/-1/-1->8->10 [15] -1/-1/-1->8->9 2: jzxh277:347412:348008 [2] NCCL INFO comm 0x55d761ef97e0 rank 10 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 2: jzxh277:347410:348011 [0] NCCL INFO P2P Chunksize set to 131072 2: jzxh277:347413:348009 [3] NCCL INFO comm 0x56099934a0c0 rank 11 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 1: jzxh276:1819789:1820383 [3] NCCL INFO comm 0x559de4393fb0 rank 7 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 1: jzxh276:1819788:1820385 [2] NCCL INFO comm 0x55a1a57d54c0 rank 6 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 1: jzxh276:1819787:1820384 [1] NCCL INFO comm 0x55c1cb2b1af0 rank 5 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 1: jzxh276:1819789:1820383 [3] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] 4/-1/-1->7->6 [2] -1/-1/-1->7->5 [3] 5/-1/-1->7->11 [4] 6/-1/-1->7->4 [5] 4/-1/-1->7->5 [6] 5/-1/-1->7->4 [7] 6/-1/-1->7->11 [8] -1/-1/-1->7->6 [9] 4/-1/-1->7->6 [10] -1/-1/-1->7->5 [11] 5/11/3->7->15 [12] 6/-1/-1->7->4 [13] 4/-1/-1->7->5 [14] 5/-1/-1->7->4 [15] 6/11/3->7->15 3: jzxh278:3330032:3330625 [3] NCCL INFO comm 0x564e400d5600 rank 15 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 3: jzxh278:3330031:3330626 [2] NCCL INFO comm 0x55b5c35aa0a0 rank 14 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 3: jzxh278:3330029:3330623 [0] NCCL INFO comm 0x55c0e6c0e9d0 rank 12 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 3: jzxh278:3330031:3330626 [2] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 15/-1/-1->14->13 [2] 12/-1/-1->14->10 [3] -1/-1/-1->14->12 [4] 13/-1/-1->14->15 [5] -1/-1/-1->14->12 [6] 12/-1/-1->14->10 [7] 13/-1/-1->14->15 [8] 15/-1/-1->14->13 [9] 15/-1/-1->14->13 [10] 12/6/-1->14->-1 [11] -1/-1/-1->14->12 [12] 13/-1/-1->14->15 [13] -1/-1/-1->14->12 [14] 12/6/-1->14->-1 [15] 13/-1/-1->14->15 3: jzxh278:3330031:3330626 [2] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330030:3330624 [1] NCCL INFO comm 0x563d7edaad10 rank 13 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 0: jzxh275:2438467:2439063 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 0/-1/-1->3->2 [2] -1/-1/-1->3->1 [3] 1/11/-1->3->-1 [4] 2/-1/-1->3->0 [5] 0/-1/-1->3->1 [6] 1/-1/-1->3->0 [7] 2/11/-1->3->-1 [8] -1/-1/-1->3->2 [9] 0/-1/-1->3->2 [10] -1/-1/-1->3->1 [11] 1/-1/-1->3->7 [12] 2/-1/-1->3->0 [13] 0/-1/-1->3->1 [14] 1/-1/-1->3->0 [15] 2/-1/-1->3->7 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 07/16 : 0 1 2 7 4 5 6 11 8 9 10 15 12 13 14 3 0: jzxh275:2438466:2439064 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 0/10/-1->2->-1 [3] -1/-1/-1->2->0 [4] 1/-1/-1->2->3 [5] -1/-1/-1->2->0 [6] 0/10/-1->2->-1 [7] 1/-1/-1->2->3 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 0/-1/-1->2->6 [11] -1/-1/-1->2->0 [12] 1/-1/-1->2->3 [13] -1/-1/-1->2->0 [14] 0/-1/-1->2->6 [15] 1/-1/-1->2->3 2: jzxh277:347411:348010 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 10/5/13->9->1 [2] 11/-1/-1->9->8 [3] 8/-1/-1->9->11 [4] -1/-1/-1->9->10 [5] 11/5/13->9->1 [6] -1/-1/-1->9->11 [7] 8/-1/-1->9->10 [8] 10/-1/-1->9->8 [9] 10/-1/-1->9->5 [10] 11/-1/-1->9->8 [11] 8/-1/-1->9->11 [12] -1/-1/-1->9->10 [13] 11/-1/-1->9->5 [14] -1/-1/-1->9->11 [15] 8/-1/-1->9->10 2: jzxh277:347411:348010 [1] NCCL INFO P2P Chunksize set to 131072 2: jzxh277:347412:348008 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 11/-1/-1->10->9 [2] 8/6/14->10->2 [3] -1/-1/-1->10->8 [4] 9/-1/-1->10->11 [5] -1/-1/-1->10->8 [6] 8/6/14->10->2 [7] 9/-1/-1->10->11 [8] 11/-1/-1->10->9 [9] 11/-1/-1->10->9 [10] 8/-1/-1->10->6 [11] -1/-1/-1->10->8 [12] 9/-1/-1->10->11 [13] -1/-1/-1->10->8 [14] 8/-1/-1->10->6 [15] 9/-1/-1->10->11 2: jzxh277:347412:348008 [2] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438467:2439063 [3] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 08/16 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: jzxh275:2438466:2439064 [2] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 09/16 : 0 3 2 5 4 7 6 9 8 11 10 13 12 15 14 1 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 10/16 : 0 3 6 5 4 7 10 9 8 11 14 13 12 15 2 1 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 11/16 : 0 1 2 7 4 5 6 11 8 9 10 15 12 13 14 3 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 12/16 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 13/16 : 0 3 2 5 4 7 6 9 8 11 10 13 12 15 14 1 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 14/16 : 0 3 6 5 4 7 10 9 8 11 14 13 12 15 2 1 2: jzxh277:347413:348009 [3] NCCL INFO Trees [0] -1/-1/-1->11->10 [1] 8/-1/-1->11->10 [2] -1/-1/-1->11->9 [3] 9/7/15->11->3 [4] 10/-1/-1->11->8 [5] 8/-1/-1->11->9 [6] 9/-1/-1->11->8 [7] 10/7/15->11->3 [8] -1/-1/-1->11->10 [9] 8/-1/-1->11->10 [10] -1/-1/-1->11->9 [11] 9/-1/-1->11->7 [12] 10/-1/-1->11->8 [13] 8/-1/-1->11->9 [14] 9/-1/-1->11->8 [15] 10/-1/-1->11->7 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 15/16 : 0 1 2 7 4 5 6 11 8 9 10 15 12 13 14 3 2: jzxh277:347413:348009 [3] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438464:2439061 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] -1/-1/-1->0->3 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->1 [4] 3/8/-1->0->-1 [5] 2/-1/-1->0->3 [6] 3/-1/-1->0->2 [7] -1/-1/-1->0->1 [8] 1/-1/-1->0->4 [9] -1/-1/-1->0->3 [10] 1/-1/-1->0->2 [11] 2/-1/-1->0->1 [12] 3/-1/-1->0->4 [13] 2/-1/-1->0->3 [14] 3/-1/-1->0->2 [15] -1/-1/-1->0->1 0: jzxh275:2438464:2439061 [0] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819788:1820385 [2] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 4/-1/-1->6->10 [3] -1/-1/-1->6->4 [4] 5/-1/-1->6->7 [5] -1/-1/-1->6->4 [6] 4/-1/-1->6->10 [7] 5/-1/-1->6->7 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 4/10/2->6->14 [11] -1/-1/-1->6->4 [12] 5/-1/-1->6->7 [13] -1/-1/-1->6->4 [14] 4/10/2->6->14 [15] 5/-1/-1->6->7 1: jzxh276:1819789:1820383 [3] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819788:1820385 [2] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819787:1820384 [1] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->9 [2] 7/-1/-1->5->4 [3] 4/-1/-1->5->7 [4] -1/-1/-1->5->6 [5] 7/-1/-1->5->9 [6] -1/-1/-1->5->7 [7] 4/-1/-1->5->6 [8] 6/-1/-1->5->4 [9] 6/9/1->5->13 [10] 7/-1/-1->5->4 [11] 4/-1/-1->5->7 [12] -1/-1/-1->5->6 [13] 7/9/1->5->13 [14] -1/-1/-1->5->7 [15] 4/-1/-1->5->6 1: jzxh276:1819787:1820384 [1] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819786:1820382 [0] NCCL INFO comm 0x5639df8e2620 rank 4 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 1: jzxh276:1819786:1820382 [0] NCCL INFO Trees [0] 5/-1/-1->4->8 [1] -1/-1/-1->4->7 [2] 5/-1/-1->4->6 [3] 6/-1/-1->4->5 [4] 7/-1/-1->4->8 [5] 6/-1/-1->4->7 [6] 7/-1/-1->4->6 [7] -1/-1/-1->4->5 [8] 5/8/0->4->12 [9] -1/-1/-1->4->7 [10] 5/-1/-1->4->6 [11] 6/-1/-1->4->5 [12] 7/8/0->4->12 [13] 6/-1/-1->4->7 [14] 7/-1/-1->4->6 [15] -1/-1/-1->4->5 1: jzxh276:1819786:1820382 [0] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330029:3330623 [0] NCCL INFO Trees [0] 13/-1/-1->12->8 [1] -1/-1/-1->12->15 [2] 13/-1/-1->12->14 [3] 14/-1/-1->12->13 [4] 15/-1/-1->12->8 [5] 14/-1/-1->12->15 [6] 15/-1/-1->12->14 [7] -1/-1/-1->12->13 [8] 13/4/-1->12->-1 [9] -1/-1/-1->12->15 [10] 13/-1/-1->12->14 [11] 14/-1/-1->12->13 [12] 15/4/-1->12->-1 [13] 14/-1/-1->12->15 [14] 15/-1/-1->12->14 [15] -1/-1/-1->12->13 3: jzxh278:3330029:3330623 [0] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330032:3330625 [3] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] 12/-1/-1->15->14 [2] -1/-1/-1->15->13 [3] 13/-1/-1->15->11 [4] 14/-1/-1->15->12 [5] 12/-1/-1->15->13 [6] 13/-1/-1->15->12 [7] 14/-1/-1->15->11 [8] -1/-1/-1->15->14 [9] 12/-1/-1->15->14 [10] -1/-1/-1->15->13 [11] 13/7/-1->15->-1 [12] 14/-1/-1->15->12 [13] 12/-1/-1->15->13 [14] 13/-1/-1->15->12 [15] 14/7/-1->15->-1 3: jzxh278:3330032:3330625 [3] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330030:3330624 [1] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->9 [2] 15/-1/-1->13->12 [3] 12/-1/-1->13->15 [4] -1/-1/-1->13->14 [5] 15/-1/-1->13->9 [6] -1/-1/-1->13->15 [7] 12/-1/-1->13->14 [8] 14/-1/-1->13->12 [9] 14/5/-1->13->-1 [10] 15/-1/-1->13->12 [11] 12/-1/-1->13->15 [12] -1/-1/-1->13->14 [13] 15/5/-1->13->-1 [14] -1/-1/-1->13->15 [15] 12/-1/-1->13->14 3: jzxh278:3330030:3330624 [1] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 00/0 : 14[2] -> 15[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 00/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 04/0 : 14[2] -> 15[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 04/0 : 10[2] -> 11[3] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 04/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 08/0 : 14[2] -> 15[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 08/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 12/0 : 14[2] -> 15[3] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 08/0 : 10[2] -> 11[3] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 03/0 : 9[1] -> 10[2] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 12/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 00/0 : 13[1] -> 14[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 04/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 00/0 : 5[1] -> 6[2] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 12/0 : 6[2] -> 7[3] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 07/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 03/0 : 5[1] -> 6[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 08/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 04/0 : 5[1] -> 6[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 03/0 : 13[1] -> 14[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 07/0 : 5[1] -> 6[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 11/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 08/0 : 5[1] -> 6[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 04/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 12/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 11/0 : 5[1] -> 6[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 15/0 : 9[1] -> 10[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 07/0 : 13[1] -> 14[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 12/0 : 5[1] -> 6[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 08/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 11/0 : 13[1] -> 14[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 15/0 : 5[1] -> 6[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 12/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 15/0 : 13[1] -> 14[2] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 00/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 00/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 04/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 04/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 08/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 08/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 08/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 12/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 12/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 12/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 00/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 03/0 : 8[0] -> 9[1] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 00/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 04/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 8[0] -> 9[1] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 04/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 08/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 08/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 12/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 12/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 00/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 07/0 : 8[0] -> 9[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 08/0 : 8[0] -> 9[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 03/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 11/0 : 8[0] -> 9[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 04/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 12/0 : 8[0] -> 9[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 07/0 : 12[0] -> 13[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 15/0 : 8[0] -> 9[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 08/0 : 12[0] -> 13[1] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 00/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 00/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 04/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 04/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 11/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 01/0 : 0[0] -> 3[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 08/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 08/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 12/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 00/0 : 4[0] -> 5[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 12/0 : 12[0] -> 13[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 01/0 : 8[0] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 02/0 : 0[0] -> 3[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 15/0 : 12[0] -> 13[1] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 03/0 : 4[0] -> 5[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 05/0 : 0[0] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 02/0 : 8[0] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 06/0 : 0[0] -> 3[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 04/0 : 4[0] -> 5[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 05/0 : 8[0] -> 11[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 06/0 : 8[0] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 09/0 : 0[0] -> 3[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 07/0 : 4[0] -> 5[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 10/0 : 0[0] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 09/0 : 8[0] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 01/0 : 12[0] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 08/0 : 4[0] -> 5[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 13/0 : 0[0] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 10/0 : 8[0] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 14/0 : 0[0] -> 3[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 02/0 : 12[0] -> 15[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 13/0 : 8[0] -> 11[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 11/0 : 4[0] -> 5[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 05/0 : 12[0] -> 15[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 14/0 : 8[0] -> 11[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 4[0] -> 5[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 06/0 : 12[0] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 15/0 : 4[0] -> 5[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 09/0 : 12[0] -> 15[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 10/0 : 12[0] -> 15[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 13/0 : 12[0] -> 15[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 14/0 : 12[0] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 01/0 : 4[0] -> 7[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 02/0 : 4[0] -> 7[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 05/0 : 4[0] -> 7[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 06/0 : 4[0] -> 7[3] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 02/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 09/0 : 4[0] -> 7[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 10/0 : 4[0] -> 7[3] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 06/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 10/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 10/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 14/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 14/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 13/0 : 4[0] -> 7[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 02/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 06/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 10/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 14/0 : 4[0] -> 7[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 14/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 02/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 06/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 10/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 14/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 02/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 02/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 06/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 06/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 10/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 10/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 14/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 14/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 02/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 02/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 06/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 06/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 10/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 01/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 01/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 05/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 05/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 14/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 09/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 09/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 13/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 13/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 01/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 05/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 09/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 09/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 13/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 13/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 01/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 01/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 05/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 05/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 09/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 09/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 13/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 13/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 01/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 01/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 05/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 01/0 : 13[1] -> 12[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 05/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 09/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 02/0 : 13[1] -> 12[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 13/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 05/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 06/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 09/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 01/0 : 5[1] -> 4[0] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 10/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 13/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 9[1] -> 8[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 02/0 : 5[1] -> 4[0] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 14/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 02/0 : 9[1] -> 8[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 05/0 : 5[1] -> 4[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 9[1] -> 8[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 06/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 5[1] -> 4[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 06/0 : 9[1] -> 8[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 09/0 : 9[1] -> 8[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 10/0 : 5[1] -> 4[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 10/0 : 9[1] -> 8[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 5[1] -> 4[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 13/0 : 9[1] -> 8[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 14/0 : 5[1] -> 4[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 14/0 : 9[1] -> 8[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 03/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 03/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 03/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 03/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 07/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 07/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 07/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 07/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 11/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 11/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 11/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 15/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 15/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 15/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 03/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 07/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 11/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 11/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 03/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 15/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 15/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 11[3] -> 8[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 07/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 11[3] -> 8[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 11/0 : 11[3] -> 8[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 11/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 03/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 03/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 15/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 07/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 15/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 07/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 02/0 : 6[2] -> 5[1] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 11/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 11/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 15/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 15/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 06/0 : 6[2] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 01/0 : 15[3] -> 14[2] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 10[2] -> 9[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 03/0 : 3[3] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 05/0 : 15[3] -> 14[2] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 01/0 : 11[3] -> 10[2] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 6[2] -> 5[1] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 03/0 : 7[3] -> 4[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 6[2] -> 5[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 05/0 : 11[3] -> 10[2] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 07/0 : 3[3] -> 0[0] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 10/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 07/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 09/0 : 15[3] -> 14[2] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 11/0 : 3[3] -> 0[0] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 14/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 7[3] -> 4[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 15/0 : 3[3] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 13/0 : 15[3] -> 14[2] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 09/0 : 11[3] -> 10[2] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 7[3] -> 4[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 13/0 : 11[3] -> 10[2] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 01/0 : 7[3] -> 6[2] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 05/0 : 7[3] -> 6[2] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 09/0 : 7[3] -> 6[2] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 13/0 : 7[3] -> 6[2] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 02/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 06/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 10/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 14/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Connected all rings 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Connected all rings 0: jzxh275:2438467:2439063 [3] NCCL INFO Connected all rings 0: jzxh275:2438466:2439064 [2] NCCL INFO Connected all rings 2: jzxh277:347413:348009 [3] NCCL INFO Connected all rings 2: jzxh277:347410:348011 [0] NCCL INFO Connected all rings 3: jzxh278:3330029:3330623 [0] NCCL INFO Connected all rings 3: jzxh278:3330030:3330624 [1] NCCL INFO Connected all rings 3: jzxh278:3330031:3330626 [2] NCCL INFO Connected all rings 3: jzxh278:3330032:3330625 [3] NCCL INFO Connected all rings 2: jzxh277:347412:348008 [2] NCCL INFO Connected all rings 2: jzxh277:347411:348010 [1] NCCL INFO Connected all rings 2: jzxh277:347410:348011 [0] NCCL INFO Channel 02/0 : 8[0] -> 9[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 02/0 : 12[0] -> 13[1] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Connected all rings 1: jzxh276:1819789:1820383 [3] NCCL INFO Connected all rings 1: jzxh276:1819786:1820382 [0] NCCL INFO Connected all rings 1: jzxh276:1819787:1820384 [1] NCCL INFO Connected all rings 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 02/0 : 4[0] -> 5[1] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 10/0 : 12[0] -> 13[1] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 10/0 : 8[0] -> 9[1] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 10/0 : 4[0] -> 5[1] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 01/0 : 14[2] -> 15[3] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 01/0 : 5[1] -> 6[2] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 07/0 : 14[2] -> 15[3] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 09/0 : 9[1] -> 10[2] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 07/0 : 10[2] -> 11[3] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 01/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 01/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 07/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 09/0 : 14[2] -> 15[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 5[1] -> 6[2] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 02/0 : 0[0] -> 2[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 09/0 : 13[1] -> 14[2] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 15/0 : 14[2] -> 15[3] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 09/0 : 10[2] -> 11[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 02/0 : 8[0] -> 10[2] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 09/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 02/0 : 12[0] -> 14[2] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 15/0 : 10[2] -> 11[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 02/0 : 4[0] -> 6[2] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 15/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 02/0 : 13[1] -> 15[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 03/0 : 8[0] -> 10[2] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 05/0 : 0[0] -> 2[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 02/0 : 1[1] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 05/0 : 8[0] -> 10[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 02/0 : 9[1] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 03/0 : 12[0] -> 14[2] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 03/0 : 4[0] -> 6[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 02/0 : 5[1] -> 7[3] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 03/0 : 13[1] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 05/0 : 4[0] -> 6[2] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 05/0 : 12[0] -> 14[2] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 06/0 : 0[0] -> 2[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 06/0 : 8[0] -> 10[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 05/0 : 13[1] -> 15[3] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 03/0 : 9[1] -> 11[3] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 03/0 : 5[1] -> 7[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 10/0 : 0[0] -> 2[2] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 06/0 : 12[0] -> 14[2] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 06/0 : 4[0] -> 6[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 05/0 : 1[1] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 10/0 : 8[0] -> 10[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 06/0 : 13[1] -> 15[3] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 9[1] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 11/0 : 0[0] -> 2[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 05/0 : 5[1] -> 7[3] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 10/0 : 13[1] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 10/0 : 4[0] -> 6[2] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 11/0 : 8[0] -> 10[2] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 10/0 : 12[0] -> 14[2] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 13/0 : 0[0] -> 2[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 06/0 : 9[1] -> 11[3] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 06/0 : 5[1] -> 7[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 11/0 : 12[0] -> 14[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 11/0 : 13[1] -> 15[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 14/0 : 0[0] -> 2[2] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 13/0 : 8[0] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 11/0 : 4[0] -> 6[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 06/0 : 1[1] -> 3[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 13/0 : 12[0] -> 14[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 10/0 : 9[1] -> 11[3] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 10/0 : 5[1] -> 7[3] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 14/0 : 8[0] -> 10[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 13/0 : 13[1] -> 15[3] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 10/0 : 1[1] -> 3[3] via P2P/CUMEM 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 13/0 : 4[0] -> 6[2] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 14/0 : 12[0] -> 14[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 14/0 : 13[1] -> 15[3] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 11/0 : 1[1] -> 3[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 04/0 : 0[0] -> 3[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 10/0 : 2[2] -> 6[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 13/0 : 1[1] -> 3[3] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 14/0 : 2[2] -> 6[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 14/0 : 4[0] -> 6[2] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 14/0 : 1[1] -> 3[3] via P2P/CUMEM 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 04/0 : 12[0] -> 15[3] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 11/0 : 5[1] -> 7[3] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 11/0 : 9[1] -> 11[3] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 01/0 : 9[1] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 02/0 : 10[2] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 05/0 : 9[1] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 12/0 : 12[0] -> 15[3] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 06/0 : 10[2] -> 14[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 09/0 : 1[1] -> 5[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 13/0 : 1[1] -> 5[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 13/0 : 9[1] -> 11[3] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 5[1] -> 7[3] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 12/0 : 0[0] -> 3[3] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 03/0 : 11[3] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 14/0 : 9[1] -> 11[3] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 14/0 : 5[1] -> 7[3] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 11/0 : 3[3] -> 7[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 8[0] -> 11[3] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 07/0 : 11[3] -> 15[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 08/0 : 0[0] -> 4[0] [send] via NET/IB/0/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 04/0 : 8[0] -> 12[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 12/0 : 0[0] -> 4[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 04/0 : 4[0] -> 7[3] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 15/0 : 3[3] -> 7[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 2[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 10/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 12/0 : 8[0] -> 11[3] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 14/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 4[0] -> 7[3] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 2[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 10[2] -> 14[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 02/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 10[2] -> 14[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 06/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 1[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 02/0 : 10[2] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 06/0 : 10[2] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 02/0 : 2[2] -> 10[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 2[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 1[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 01/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 06/0 : 2[2] -> 10[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 05/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 10/0 : 6[2] -> 14[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 2[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 14/0 : 6[2] -> 14[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 10/0 : 14[2] -> 6[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 10[2] -> 2[2] [send] via NET/IB/2/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 14/0 : 14[2] -> 6[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 10/0 : 6[2] -> 2[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 09/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 10[2] -> 2[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 14[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 14/0 : 6[2] -> 2[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 01/0 : 1[1] -> 9[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 3[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 08/0 : 0[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 14[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 6[2] -> 14[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 3[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 0[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 6[2] -> 14[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 03/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 00/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 07/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 05/0 : 1[1] -> 9[1] [send] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 02/0 : 14[2] -> 10[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 13/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 04/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 06/0 : 14[2] -> 10[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 9[1] -> 13[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 09/0 : 5[1] -> 13[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 04/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 9[1] -> 13[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 14[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 08/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 00/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 14[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 02/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 13[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 06/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 03/0 : 3[3] -> 11[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 04/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 1[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 08/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 10/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 07/0 : 3[3] -> 11[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 13/0 : 5[1] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 09/0 : 13[1] -> 5[1] [send] via NET/IB/1/GDRDMA 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 13/0 : 13[1] -> 5[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 09/0 : 5[1] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 13/0 : 5[1] -> 1[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 13[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 5[1] -> 13[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 6[2] -> 2[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 5[1] -> 13[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 6[2] -> 2[2] [send] via NET/IB/2/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 02/0 : 14[2] -> 12[0] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 01/0 : 13[1] -> 9[1] [send] via NET/IB/1/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 08/0 : 4[0] -> 12[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 03/0 : 14[2] -> 12[0] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 05/0 : 13[1] -> 9[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 11/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 08/0 : 12[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 1[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 14/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 12/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 01/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 15/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 15[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 12[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 05/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 15[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 11[3] -> 15[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 08/0 : 4[0] -> 12[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 8[0] -> 12[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 7[3] -> 15[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 11[3] -> 15[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 4[0] -> 12[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 13[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 7[3] -> 15[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 13[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 12/0 : 4[0] -> 12[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 11/0 : 7[3] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 08/0 : 12[0] -> 4[0] [send] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 05/0 : 14[2] -> 12[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 15/0 : 7[3] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 12/0 : 12[0] -> 4[0] [send] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 11/0 : 15[3] -> 7[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 01/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 3[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 05/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 3[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347411:348010 [1] NCCL INFO Channel 09/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 15/0 : 15[3] -> 7[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 06/0 : 14[2] -> 12[0] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 02/0 : 10[2] -> 8[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 13/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 00/0 : 12[0] -> 8[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 00/0 : 12[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330029:3330623 [0] NCCL INFO Channel 04/0 : 12[0] -> 8[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 03/0 : 10[2] -> 8[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 03/0 : 15[3] -> 11[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 10/0 : 14[2] -> 12[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 07/0 : 15[3] -> 11[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 15[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 02/0 : 6[2] -> 4[0] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 12[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 09/0 : 5[1] -> 1[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 15[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 13/0 : 5[1] -> 1[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 05/0 : 10[2] -> 8[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 03/0 : 6[2] -> 4[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347410:348011 [0] NCCL INFO Channel 04/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 04/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 11/0 : 14[2] -> 12[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 01/0 : 15[3] -> 12[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 05/0 : 6[2] -> 4[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 08/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 07/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 08/0 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 06/0 : 6[2] -> 4[0] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 08/0 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819786:1820382 [0] NCCL INFO Channel 12/0 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 7[3] -> 3[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 7[3] -> 3[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 10/0 : 6[2] -> 4[0] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Channel 12/0 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 13/0 : 14[2] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 11/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 11/0 : 6[2] -> 4[0] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 08/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347413:348009 [3] NCCL INFO Channel 15/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 13/0 : 6[2] -> 4[0] via P2P/CUMEM 2: jzxh277:347410:348011 [0] NCCL INFO Channel 12/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348008 [2] NCCL INFO Channel 06/0 : 10[2] -> 8[0] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 10/0 : 10[2] -> 8[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 04/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 01/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 11/0 : 7[3] -> 3[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 02/0 : 2[2] -> 0[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 15/0 : 7[3] -> 3[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 05/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 01/0 : 3[3] -> 0[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 11/0 : 10[2] -> 8[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 14/0 : 6[2] -> 4[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 04/0 : 11[3] -> 8[0] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 14/0 : 14[2] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 05/0 : 11[3] -> 8[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 01/0 : 7[3] -> 4[0] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 13/0 : 10[2] -> 8[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 06/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 05/0 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 06/0 : 11[3] -> 8[0] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 14/0 : 10[2] -> 8[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 04/0 : 3[3] -> 0[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 04/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 09/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 09/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 06/0 : 2[2] -> 0[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 05/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 12/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 05/0 : 3[3] -> 0[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 10/0 : 2[2] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 13/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 06/0 : 3[3] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 14/0 : 15[3] -> 12[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 02/0 : 15[3] -> 13[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 06/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 03/0 : 15[3] -> 13[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 09/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 05/0 : 15[3] -> 13[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 12/0 : 11[3] -> 8[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 12/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 06/0 : 15[3] -> 13[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 13/0 : 11[3] -> 8[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 13/0 : 7[3] -> 4[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 14/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 10/0 : 15[3] -> 13[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 14/0 : 11[3] -> 8[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 02/0 : 7[3] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 11/0 : 15[3] -> 13[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 03/0 : 7[3] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 13/0 : 15[3] -> 13[1] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 11/0 : 2[2] -> 0[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 05/0 : 7[3] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 14/0 : 15[3] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 09/0 : 3[3] -> 0[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 06/0 : 7[3] -> 5[1] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 13/0 : 2[2] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 00/0 : 15[3] -> 14[2] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 02/0 : 11[3] -> 9[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 12/0 : 3[3] -> 0[0] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 10/0 : 7[3] -> 5[1] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 14/0 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 03/0 : 11[3] -> 9[1] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 04/0 : 15[3] -> 14[2] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 11/0 : 7[3] -> 5[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 13/0 : 3[3] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 05/0 : 11[3] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 13/0 : 7[3] -> 5[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 14/0 : 3[3] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 06/0 : 11[3] -> 9[1] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 07/0 : 15[3] -> 14[2] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 14/0 : 7[3] -> 5[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 10/0 : 11[3] -> 9[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 02/0 : 3[3] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 00/0 : 7[3] -> 6[2] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 08/0 : 15[3] -> 14[2] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 11/0 : 11[3] -> 9[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 13/0 : 11[3] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 04/0 : 7[3] -> 6[2] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 12/0 : 15[3] -> 14[2] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 05/0 : 3[3] -> 1[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 14/0 : 11[3] -> 9[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 06/0 : 3[3] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 07/0 : 7[3] -> 6[2] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 00/0 : 11[3] -> 10[2] via P2P/CUMEM 3: jzxh278:3330032:3330625 [3] NCCL INFO Channel 15/0 : 15[3] -> 14[2] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 10/0 : 3[3] -> 1[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 11/0 : 3[3] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 08/0 : 7[3] -> 6[2] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 04/0 : 11[3] -> 10[2] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 00/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 13/0 : 3[3] -> 1[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 07/0 : 11[3] -> 10[2] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 12/0 : 7[3] -> 6[2] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 01/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 14/0 : 3[3] -> 1[1] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 08/0 : 11[3] -> 10[2] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 00/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 04/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM 1: jzxh276:1819789:1820383 [3] NCCL INFO Channel 15/0 : 7[3] -> 6[2] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 07/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 03/0 : 13[1] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 12/0 : 11[3] -> 10[2] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 08/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 07/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 00/0 : 6[2] -> 5[1] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 09/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 08/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 12/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 11/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 01/0 : 6[2] -> 5[1] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 00/0 : 10[2] -> 9[1] via P2P/CUMEM 3: jzxh278:3330031:3330626 [2] NCCL INFO Channel 15/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330030:3330624 [1] NCCL INFO Channel 15/0 : 13[1] -> 12[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 04/0 : 6[2] -> 5[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM 2: jzxh277:347413:348009 [3] NCCL INFO Channel 15/0 : 11[3] -> 10[2] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 00/0 : 5[1] -> 4[0] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 01/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 07/0 : 6[2] -> 5[1] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 03/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 08/0 : 6[2] -> 5[1] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 04/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 07/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 09/0 : 6[2] -> 5[1] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 07/0 : 10[2] -> 9[1] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 08/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 08/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 12/0 : 6[2] -> 5[1] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM 0: jzxh275:2438467:2439063 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 00/0 : 9[1] -> 8[0] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 03/0 : 9[1] -> 8[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 11/0 : 5[1] -> 4[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 07/0 : 9[1] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 09/0 : 10[2] -> 9[1] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 12/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819788:1820385 [2] NCCL INFO Channel 15/0 : 6[2] -> 5[1] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 08/0 : 9[1] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM 2: jzxh277:347412:348008 [2] NCCL INFO Channel 15/0 : 10[2] -> 9[1] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 11/0 : 9[1] -> 8[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM 2: jzxh277:347411:348010 [1] NCCL INFO Channel 15/0 : 9[1] -> 8[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438466:2439064 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM 1: jzxh276:1819787:1820384 [1] NCCL INFO Channel 15/0 : 5[1] -> 4[0] via P2P/CUMEM 0: jzxh275:2438465:2439062 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438464:2439061 [0] NCCL INFO Connected all trees 0: jzxh275:2438465:2439062 [1] NCCL INFO Connected all trees 0: jzxh275:2438467:2439063 [3] NCCL INFO Connected all trees 0: jzxh275:2438465:2439062 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 0: jzxh275:2438465:2439062 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 0: jzxh275:2438467:2439063 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 0: jzxh275:2438466:2439064 [2] NCCL INFO Connected all trees 0: jzxh275:2438467:2439063 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 0: jzxh275:2438466:2439064 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 0: jzxh275:2438464:2439061 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 0: jzxh275:2438466:2439064 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 0: jzxh275:2438464:2439061 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 1: jzxh276:1819786:1820382 [0] NCCL INFO Connected all trees 1: jzxh276:1819789:1820383 [3] NCCL INFO Connected all trees 3: jzxh278:3330029:3330623 [0] NCCL INFO Connected all trees 1: jzxh276:1819786:1820382 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 1: jzxh276:1819786:1820382 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 1: jzxh276:1819789:1820383 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330029:3330623 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330029:3330623 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 3: jzxh278:3330032:3330625 [3] NCCL INFO Connected all trees 1: jzxh276:1819789:1820383 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 3: jzxh278:3330030:3330624 [1] NCCL INFO Connected all trees 1: jzxh276:1819788:1820385 [2] NCCL INFO Connected all trees 1: jzxh276:1819787:1820384 [1] NCCL INFO Connected all trees 1: jzxh276:1819788:1820385 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330031:3330626 [2] NCCL INFO Connected all trees 3: jzxh278:3330032:3330625 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 1: jzxh276:1819788:1820385 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 1: jzxh276:1819787:1820384 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 1: jzxh276:1819787:1820384 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 3: jzxh278:3330032:3330625 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 3: jzxh278:3330030:3330624 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330030:3330624 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 3: jzxh278:3330031:3330626 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330031:3330626 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347413:348009 [3] NCCL INFO Connected all trees 2: jzxh277:347413:348009 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 2: jzxh277:347413:348009 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347410:348011 [0] NCCL INFO Connected all trees 2: jzxh277:347410:348011 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 2: jzxh277:347410:348011 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347411:348010 [1] NCCL INFO Connected all trees 2: jzxh277:347411:348010 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 2: jzxh277:347411:348010 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347412:348008 [2] NCCL INFO Connected all trees 2: jzxh277:347412:348008 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 2: jzxh277:347412:348008 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 0: jzxh275:2438467:2439063 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 0: jzxh275:2438466:2439064 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 0: jzxh275:2438467:2439063 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 0: jzxh275:2438466:2439064 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 0: jzxh275:2438467:2439063 [3] NCCL INFO ncclCommInitRank comm 0x559ed6e1be60 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x3645fd26de633f8b - Init COMPLETE 0: jzxh275:2438466:2439064 [2] NCCL INFO ncclCommInitRank comm 0x5624ea7d7d40 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x3645fd26de633f8b - Init COMPLETE 0: jzxh275:2438464:2439061 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 0: jzxh275:2438465:2439062 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 0: jzxh275:2438464:2439061 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 0: jzxh275:2438465:2439062 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 0: jzxh275:2438464:2439061 [0] NCCL INFO ncclCommInitRank comm 0x55e4cbf1c410 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x3645fd26de633f8b - Init COMPLETE 0: jzxh275:2438465:2439062 [1] NCCL INFO ncclCommInitRank comm 0x560d69039660 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x3645fd26de633f8b - Init COMPLETE 2: jzxh277:347413:348009 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 2: jzxh277:347412:348008 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 2: jzxh277:347413:348009 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 2: jzxh277:347412:348008 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 2: jzxh277:347413:348009 [3] NCCL INFO ncclCommInitRank comm 0x56099934a0c0 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x3645fd26de633f8b - Init COMPLETE 2: jzxh277:347412:348008 [2] NCCL INFO ncclCommInitRank comm 0x55d761ef97e0 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x3645fd26de633f8b - Init COMPLETE 2: jzxh277:347411:348010 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 2: jzxh277:347410:348011 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 2: jzxh277:347411:348010 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 2: jzxh277:347410:348011 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 2: jzxh277:347411:348010 [1] NCCL INFO ncclCommInitRank comm 0x5622e8e76730 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x3645fd26de633f8b - Init COMPLETE 2: jzxh277:347410:348011 [0] NCCL INFO ncclCommInitRank comm 0x564073e3d4b0 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x3645fd26de633f8b - Init COMPLETE 0: jzxh275:2438467:2439093 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/CUMEM 0: jzxh275:2438465:2439094 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438467:2439093 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/CUMEM 0: jzxh275:2438465:2439094 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348040 [3] NCCL INFO Channel 04/1 : 11[3] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared 2: jzxh277:347413:348040 [3] NCCL INFO Channel 05/1 : 11[3] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared 2: jzxh277:347410:348041 [0] NCCL INFO Channel 04/1 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 08/1 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 09/1 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438466:2439096 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347410:348041 [0] NCCL INFO Channel 05/1 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared 2: jzxh277:347412:348042 [2] NCCL INFO Channel 04/1 : 10[2] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared 2: jzxh277:347412:348042 [2] NCCL INFO Channel 05/1 : 10[2] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared 2: jzxh277:347411:348043 [1] NCCL INFO Channel 04/1 : 9[1] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared 0: jzxh275:2438466:2439096 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347411:348043 [1] NCCL INFO Channel 05/1 : 9[1] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared 1: jzxh276:1819789:1820383 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 1: jzxh276:1819788:1820385 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 1: jzxh276:1819789:1820383 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 1: jzxh276:1819788:1820385 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 1: jzxh276:1819789:1820383 [3] NCCL INFO ncclCommInitRank comm 0x559de4393fb0 rank 7 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x3645fd26de633f8b - Init COMPLETE 1: jzxh276:1819788:1820385 [2] NCCL INFO ncclCommInitRank comm 0x55a1a57d54c0 rank 6 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x3645fd26de633f8b - Init COMPLETE 1: jzxh276:1819787:1820384 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 1: jzxh276:1819786:1820382 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 1: jzxh276:1819787:1820384 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 1: jzxh276:1819786:1820382 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 1: jzxh276:1819787:1820384 [1] NCCL INFO ncclCommInitRank comm 0x55c1cb2b1af0 rank 5 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x3645fd26de633f8b - Init COMPLETE 1: jzxh276:1819786:1820382 [0] NCCL INFO ncclCommInitRank comm 0x5639df8e2620 rank 4 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x3645fd26de633f8b - Init COMPLETE 3: jzxh278:3330029:3330623 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 3: jzxh278:3330030:3330624 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 3: jzxh278:3330029:3330623 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 3: jzxh278:3330030:3330624 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 3: jzxh278:3330029:3330623 [0] NCCL INFO ncclCommInitRank comm 0x55c0e6c0e9d0 rank 12 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x3645fd26de633f8b - Init COMPLETE 3: jzxh278:3330030:3330624 [1] NCCL INFO ncclCommInitRank comm 0x563d7edaad10 rank 13 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x3645fd26de633f8b - Init COMPLETE 3: jzxh278:3330031:3330626 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 3: jzxh278:3330031:3330626 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 3: jzxh278:3330032:3330625 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so 3: jzxh278:3330031:3330626 [2] NCCL INFO ncclCommInitRank comm 0x55b5c35aa0a0 rank 14 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x3645fd26de633f8b - Init COMPLETE 3: jzxh278:3330032:3330625 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin. 3: jzxh278:3330032:3330625 [3] NCCL INFO ncclCommInitRank comm 0x564e400d5600 rank 15 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x3645fd26de633f8b - Init COMPLETE 1: jzxh276:1819786:1820414 [0] NCCL INFO Channel 12/1 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared 1: jzxh276:1819786:1820414 [0] NCCL INFO Channel 13/1 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared 1: jzxh276:1819787:1820415 [1] NCCL INFO Channel 12/1 : 5[1] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared 1: jzxh276:1819787:1820415 [1] NCCL INFO Channel 13/1 : 5[1] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared 1: jzxh276:1819789:1820416 [3] NCCL INFO Channel 12/1 : 7[3] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared 1: jzxh276:1819789:1820416 [3] NCCL INFO Channel 13/1 : 7[3] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared 1: jzxh276:1819788:1820417 [2] NCCL INFO Channel 12/1 : 6[2] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared 1: jzxh276:1819788:1820417 [2] NCCL INFO Channel 13/1 : 6[2] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared 3: jzxh278:3330029:3330655 [0] NCCL INFO Channel 08/1 : 12[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared 3: jzxh278:3330029:3330655 [0] NCCL INFO Channel 09/1 : 12[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared 3: jzxh278:3330030:3330656 [1] NCCL INFO Channel 08/1 : 13[1] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared 3: jzxh278:3330031:3330657 [2] NCCL INFO Channel 08/1 : 14[2] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared 3: jzxh278:3330030:3330656 [1] NCCL INFO Channel 09/1 : 13[1] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared 3: jzxh278:3330031:3330657 [2] NCCL INFO Channel 09/1 : 14[2] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared 3: jzxh278:3330032:3330658 [3] NCCL INFO Channel 08/1 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared 3: jzxh278:3330032:3330658 [3] NCCL INFO Channel 09/1 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 08/1 : 14[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 09/1 : 14[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 08/1 : 13[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 09/1 : 13[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 08/1 : 12[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 09/1 : 12[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 04/1 : 11[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 05/1 : 11[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 04/1 : 10[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 05/1 : 10[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 04/1 : 9[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 05/1 : 9[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 04/1 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 05/1 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 12/1 : 7[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 13/1 : 7[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 12/1 : 6[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 13/1 : 6[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 12/1 : 5[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 13/1 : 5[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 12/1 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: jzxh275:2438464:2439095 [0] NCCL INFO Channel 13/1 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared 0: [2025-11-22 23:28:55,512] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:436] [PID:2438464] [RANK:0] gather_len_batches: [99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297] 0: [2025-11-22 23:28:55,537] [INFO] [axolotl.utils.trainer.calc_sample_packing_eff_est:495] [PID:2438464] [RANK:0] sample_packing_eff_est across ranks: [0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223, 0.8523049354553223] 0: [2025-11-22 23:28:55,555] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:127] [PID:2438464] [RANK:0] Maximum number of steps set at 3723 1: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 1: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 1: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 1: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 3: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 3: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 2: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 2: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 3: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 3: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 3: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 3: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 2: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 2: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 3: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 3: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 1: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 1: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 1: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 1: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: [2025-11-22 23:28:56,652] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:110] [PID:2438464] [RANK:0] Patched Trainer.evaluation_loop with nanmean loss calculation 0: [2025-11-22 23:28:56,654] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:164] [PID:2438464] [RANK:0] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation 0: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 2: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 2: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 2: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 2: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: The following generation flags are not valid and may be ignored: ['cache_implementation']. Set `TRANSFORMERS_VERBOSITY=info` for more details. 0: [2025-11-22 23:29:03,598] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:345] [PID:2438464] [RANK:0] Converting modules to torch.bfloat16 0: [2025-11-22 23:29:06,602] [INFO] [axolotl.train.save_initial_configs:416] [PID:2438464] [RANK:0] Pre-saving tokenizer to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-1b/1... 0: [2025-11-22 23:29:07,228] [INFO] [axolotl.train.save_initial_configs:419] [PID:2438464] [RANK:0] Pre-saving model config to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-1b/1... 0: [2025-11-22 23:29:07,239] [INFO] [axolotl.train.execute_training:203] [PID:2438464] [RANK:0] Starting trainer... 0: [2025-11-22 23:35:13,420] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:436] [PID:2438464] [RANK:0] gather_len_batches: [99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297, 99297] 0: Parameter Offload - Persistent parameters statistics: param_count = 157, numel = 134272 0: 0%| | 0/3723 [00:00', attn_implementation='eager')`. 0: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 0: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 0: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 1: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 1: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 1: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 3: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 2: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 2: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 3: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 3: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 2: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 2: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 3: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 1: It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `flash_attention_2`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`. 0: jzxh275:2438464:2439957 [0] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438467:2439965 [3] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438465:2439955 [1] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438464:2439967 [0] NCCL INFO Using non-device net plugin version 0 0: jzxh275:2438464:2439967 [0] NCCL INFO Using network IB 3: jzxh278:3330031:3331517 [2] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330032:3331521 [3] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438466:2439952 [2] NCCL INFO Comm config Blocking set to 1 1: jzxh276:1819788:1821272 [2] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438467:2439968 [3] NCCL INFO Using non-device net plugin version 0 0: jzxh275:2438467:2439968 [3] NCCL INFO Using network IB 0: jzxh275:2438465:2439969 [1] NCCL INFO Using non-device net plugin version 0 0: jzxh275:2438465:2439969 [1] NCCL INFO Using network IB 2: jzxh277:347411:348884 [1] NCCL INFO Comm config Blocking set to 1 2: jzxh277:347412:348895 [2] NCCL INFO Comm config Blocking set to 1 0: jzxh275:2438466:2439970 [2] NCCL INFO Using non-device net plugin version 0 0: jzxh275:2438466:2439970 [2] NCCL INFO Using network IB 3: jzxh278:3330029:3331522 [0] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330030:3331511 [1] NCCL INFO Comm config Blocking set to 1 1: jzxh276:1819787:1821276 [1] NCCL INFO Comm config Blocking set to 1 1: jzxh276:1819789:1821269 [3] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330031:3331527 [2] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819786:1821273 [0] NCCL INFO Comm config Blocking set to 1 3: jzxh278:3330031:3331527 [2] NCCL INFO Using network IB 1: jzxh276:1819788:1821282 [2] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819788:1821282 [2] NCCL INFO Using network IB 2: jzxh277:347413:348897 [3] NCCL INFO Comm config Blocking set to 1 2: jzxh277:347410:348883 [0] NCCL INFO Comm config Blocking set to 1 1: jzxh276:1819787:1821284 [1] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819787:1821284 [1] NCCL INFO Using network IB 1: jzxh276:1819789:1821283 [3] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819789:1821283 [3] NCCL INFO Using network IB 1: jzxh276:1819786:1821285 [0] NCCL INFO Using non-device net plugin version 0 1: jzxh276:1819786:1821285 [0] NCCL INFO Using network IB 3: jzxh278:3330030:3331528 [1] NCCL INFO Using non-device net plugin version 0 3: jzxh278:3330030:3331528 [1] NCCL INFO Using network IB 3: jzxh278:3330029:3331529 [0] NCCL INFO Using non-device net plugin version 0 3: jzxh278:3330032:3331526 [3] NCCL INFO Using non-device net plugin version 0 3: jzxh278:3330029:3331529 [0] NCCL INFO Using network IB 3: jzxh278:3330032:3331526 [3] NCCL INFO Using network IB 2: jzxh277:347412:348899 [2] NCCL INFO Using non-device net plugin version 0 2: jzxh277:347412:348899 [2] NCCL INFO Using network IB 2: jzxh277:347413:348900 [3] NCCL INFO Using non-device net plugin version 0 2: jzxh277:347413:348900 [3] NCCL INFO Using network IB 2: jzxh277:347410:348901 [0] NCCL INFO Using non-device net plugin version 0 2: jzxh277:347410:348901 [0] NCCL INFO Using network IB 2: jzxh277:347411:348898 [1] NCCL INFO Using non-device net plugin version 0 2: jzxh277:347411:348898 [1] NCCL INFO Using network IB 0: jzxh275:2438464:2439967 [0] NCCL INFO ncclCommInitRank comm 0x1483dc13e760 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x6e9cb86670bfb8a0 - Init START 0: jzxh275:2438466:2439970 [2] NCCL INFO ncclCommInitRank comm 0x145130129740 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x6e9cb86670bfb8a0 - Init START 0: jzxh275:2438467:2439968 [3] NCCL INFO ncclCommInitRank comm 0x144eb01303b0 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x6e9cb86670bfb8a0 - Init START 0: jzxh275:2438465:2439969 [1] NCCL INFO ncclCommInitRank comm 0x14bdac12f380 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x6e9cb86670bfb8a0 - Init START 3: jzxh278:3330032:3331526 [3] NCCL INFO ncclCommInitRank comm 0x150804130040 rank 15 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x6e9cb86670bfb8a0 - Init START 1: jzxh276:1819786:1821285 [0] NCCL INFO ncclCommInitRank comm 0x14ed18138c00 rank 4 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x6e9cb86670bfb8a0 - Init START 2: jzxh277:347410:348901 [0] NCCL INFO ncclCommInitRank comm 0x1454ec132610 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x6e9cb86670bfb8a0 - Init START 3: jzxh278:3330031:3331527 [2] NCCL INFO ncclCommInitRank comm 0x149e2412ed10 rank 14 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x6e9cb86670bfb8a0 - Init START 1: jzxh276:1819787:1821284 [1] NCCL INFO ncclCommInitRank comm 0x148920141450 rank 5 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x6e9cb86670bfb8a0 - Init START 1: jzxh276:1819788:1821282 [2] NCCL INFO ncclCommInitRank comm 0x14aad812e700 rank 6 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x6e9cb86670bfb8a0 - Init START 3: jzxh278:3330030:3331528 [1] NCCL INFO ncclCommInitRank comm 0x154b80130140 rank 13 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x6e9cb86670bfb8a0 - Init START 1: jzxh276:1819789:1821283 [3] NCCL INFO ncclCommInitRank comm 0x14fb60138940 rank 7 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x6e9cb86670bfb8a0 - Init START 3: jzxh278:3330029:3331529 [0] NCCL INFO ncclCommInitRank comm 0x151574137e00 rank 12 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x6e9cb86670bfb8a0 - Init START 2: jzxh277:347411:348898 [1] NCCL INFO ncclCommInitRank comm 0x148290127000 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x6e9cb86670bfb8a0 - Init START 2: jzxh277:347413:348900 [3] NCCL INFO ncclCommInitRank comm 0x14800014cfc0 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x6e9cb86670bfb8a0 - Init START 2: jzxh277:347412:348899 [2] NCCL INFO ncclCommInitRank comm 0x153aa01384c0 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x6e9cb86670bfb8a0 - Init START 3: jzxh278:3330030:3331528 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000 3: jzxh278:3330030:3331528 [1] NCCL INFO NVLS multicast support is not available on dev 1 2: jzxh277:347413:348900 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000 2: jzxh277:347413:348900 [3] NCCL INFO NVLS multicast support is not available on dev 3 2: jzxh277:347411:348898 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000 2: jzxh277:347411:348898 [1] NCCL INFO NVLS multicast support is not available on dev 1 0: jzxh275:2438467:2439968 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000 0: jzxh275:2438467:2439968 [3] NCCL INFO NVLS multicast support is not available on dev 3 1: jzxh276:1819789:1821283 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000 1: jzxh276:1819789:1821283 [3] NCCL INFO NVLS multicast support is not available on dev 3 0: jzxh275:2438466:2439970 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000 0: jzxh275:2438466:2439970 [2] NCCL INFO NVLS multicast support is not available on dev 2 1: jzxh276:1819787:1821284 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000 3: jzxh278:3330032:3331526 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000 3: jzxh278:3330032:3331526 [3] NCCL INFO NVLS multicast support is not available on dev 3 1: jzxh276:1819787:1821284 [1] NCCL INFO NVLS multicast support is not available on dev 1 2: jzxh277:347412:348899 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000 2: jzxh277:347412:348899 [2] NCCL INFO NVLS multicast support is not available on dev 2 1: jzxh276:1819786:1821285 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff 3: jzxh278:3330029:3331529 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff 3: jzxh278:3330029:3331529 [0] NCCL INFO NVLS multicast support is not available on dev 0 1: jzxh276:1819786:1821285 [0] NCCL INFO NVLS multicast support is not available on dev 0 2: jzxh277:347410:348901 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff 2: jzxh277:347410:348901 [0] NCCL INFO NVLS multicast support is not available on dev 0 3: jzxh278:3330031:3331527 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000 1: jzxh276:1819788:1821282 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000 3: jzxh278:3330031:3331527 [2] NCCL INFO NVLS multicast support is not available on dev 2 1: jzxh276:1819788:1821282 [2] NCCL INFO NVLS multicast support is not available on dev 2 0: jzxh275:2438465:2439969 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000 0: jzxh275:2438464:2439967 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff 0: jzxh275:2438465:2439969 [1] NCCL INFO NVLS multicast support is not available on dev 1 0: jzxh275:2438464:2439967 [0] NCCL INFO NVLS multicast support is not available on dev 0 0: jzxh275:2438464:2439967 [0] NCCL INFO comm 0x1483dc13e760 rank 0 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 0: jzxh275:2438465:2439969 [1] NCCL INFO comm 0x14bdac12f380 rank 1 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 2: jzxh277:347410:348901 [0] NCCL INFO comm 0x1454ec132610 rank 8 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 1: jzxh276:1819789:1821283 [3] NCCL INFO comm 0x14fb60138940 rank 7 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 1: jzxh276:1819788:1821282 [2] NCCL INFO comm 0x14aad812e700 rank 6 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 2: jzxh277:347411:348898 [1] NCCL INFO comm 0x148290127000 rank 9 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 2: jzxh277:347410:348901 [0] NCCL INFO Trees [0] 9/4/12->8->0 [1] -1/-1/-1->8->11 [2] 9/-1/-1->8->10 [3] 10/-1/-1->8->9 [4] 11/4/12->8->0 [5] 10/-1/-1->8->11 [6] 11/-1/-1->8->10 [7] -1/-1/-1->8->9 [8] 9/-1/-1->8->4 [9] -1/-1/-1->8->11 [10] 9/-1/-1->8->10 [11] 10/-1/-1->8->9 [12] 11/-1/-1->8->4 [13] 10/-1/-1->8->11 [14] 11/-1/-1->8->10 [15] -1/-1/-1->8->9 2: jzxh277:347410:348901 [0] NCCL INFO P2P Chunksize set to 131072 2: jzxh277:347412:348899 [2] NCCL INFO comm 0x153aa01384c0 rank 10 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 3: jzxh278:3330032:3331526 [3] NCCL INFO comm 0x150804130040 rank 15 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 3: jzxh278:3330031:3331527 [2] NCCL INFO comm 0x149e2412ed10 rank 14 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 3: jzxh278:3330030:3331528 [1] NCCL INFO comm 0x154b80130140 rank 13 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 3: jzxh278:3330029:3331529 [0] NCCL INFO comm 0x151574137e00 rank 12 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 0: jzxh275:2438466:2439970 [2] NCCL INFO comm 0x145130129740 rank 2 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 00/16 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: jzxh275:2438467:2439968 [3] NCCL INFO comm 0x144eb01303b0 rank 3 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 01/16 : 0 3 2 5 4 7 6 9 8 11 10 13 12 15 14 1 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 02/16 : 0 3 6 5 4 7 10 9 8 11 14 13 12 15 2 1 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 03/16 : 0 1 2 7 4 5 6 11 8 9 10 15 12 13 14 3 2: jzxh277:347413:348900 [3] NCCL INFO comm 0x14800014cfc0 rank 11 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 2: jzxh277:347411:348898 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 10/5/13->9->1 [2] 11/-1/-1->9->8 [3] 8/-1/-1->9->11 [4] -1/-1/-1->9->10 [5] 11/5/13->9->1 [6] -1/-1/-1->9->11 [7] 8/-1/-1->9->10 [8] 10/-1/-1->9->8 [9] 10/-1/-1->9->5 [10] 11/-1/-1->9->8 [11] 8/-1/-1->9->11 [12] -1/-1/-1->9->10 [13] 11/-1/-1->9->5 [14] -1/-1/-1->9->11 [15] 8/-1/-1->9->10 2: jzxh277:347411:348898 [1] NCCL INFO P2P Chunksize set to 131072 2: jzxh277:347412:348899 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 11/-1/-1->10->9 [2] 8/6/14->10->2 [3] -1/-1/-1->10->8 [4] 9/-1/-1->10->11 [5] -1/-1/-1->10->8 [6] 8/6/14->10->2 [7] 9/-1/-1->10->11 [8] 11/-1/-1->10->9 [9] 11/-1/-1->10->9 [10] 8/-1/-1->10->6 [11] -1/-1/-1->10->8 [12] 9/-1/-1->10->11 [13] -1/-1/-1->10->8 [14] 8/-1/-1->10->6 [15] 9/-1/-1->10->11 2: jzxh277:347412:348899 [2] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819789:1821283 [3] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] 4/-1/-1->7->6 [2] -1/-1/-1->7->5 [3] 5/-1/-1->7->11 [4] 6/-1/-1->7->4 [5] 4/-1/-1->7->5 [6] 5/-1/-1->7->4 [7] 6/-1/-1->7->11 [8] -1/-1/-1->7->6 [9] 4/-1/-1->7->6 [10] -1/-1/-1->7->5 [11] 5/11/3->7->15 [12] 6/-1/-1->7->4 [13] 4/-1/-1->7->5 [14] 5/-1/-1->7->4 [15] 6/11/3->7->15 1: jzxh276:1819789:1821283 [3] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330031:3331527 [2] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 15/-1/-1->14->13 [2] 12/-1/-1->14->10 [3] -1/-1/-1->14->12 [4] 13/-1/-1->14->15 [5] -1/-1/-1->14->12 [6] 12/-1/-1->14->10 [7] 13/-1/-1->14->15 [8] 15/-1/-1->14->13 [9] 15/-1/-1->14->13 [10] 12/6/-1->14->-1 [11] -1/-1/-1->14->12 [12] 13/-1/-1->14->15 [13] -1/-1/-1->14->12 [14] 12/6/-1->14->-1 [15] 13/-1/-1->14->15 3: jzxh278:3330032:3331526 [3] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] 12/-1/-1->15->14 [2] -1/-1/-1->15->13 [3] 13/-1/-1->15->11 [4] 14/-1/-1->15->12 [5] 12/-1/-1->15->13 [6] 13/-1/-1->15->12 [7] 14/-1/-1->15->11 [8] -1/-1/-1->15->14 [9] 12/-1/-1->15->14 [10] -1/-1/-1->15->13 [11] 13/7/-1->15->-1 [12] 14/-1/-1->15->12 [13] 12/-1/-1->15->13 [14] 13/-1/-1->15->12 [15] 14/7/-1->15->-1 0: jzxh275:2438465:2439969 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/9/-1->1->-1 [2] 3/-1/-1->1->0 [3] 0/-1/-1->1->3 [4] -1/-1/-1->1->2 [5] 3/9/-1->1->-1 [6] -1/-1/-1->1->3 [7] 0/-1/-1->1->2 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->5 [10] 3/-1/-1->1->0 [11] 0/-1/-1->1->3 [12] -1/-1/-1->1->2 [13] 3/-1/-1->1->5 [14] -1/-1/-1->1->3 [15] 0/-1/-1->1->2 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 04/16 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: jzxh275:2438465:2439969 [1] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 05/16 : 0 3 2 5 4 7 6 9 8 11 10 13 12 15 14 1 0: jzxh275:2438466:2439970 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 0/10/-1->2->-1 [3] -1/-1/-1->2->0 [4] 1/-1/-1->2->3 [5] -1/-1/-1->2->0 [6] 0/10/-1->2->-1 [7] 1/-1/-1->2->3 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 0/-1/-1->2->6 [11] -1/-1/-1->2->0 [12] 1/-1/-1->2->3 [13] -1/-1/-1->2->0 [14] 0/-1/-1->2->6 [15] 1/-1/-1->2->3 2: jzxh277:347413:348900 [3] NCCL INFO Trees [0] -1/-1/-1->11->10 [1] 8/-1/-1->11->10 [2] -1/-1/-1->11->9 [3] 9/7/15->11->3 [4] 10/-1/-1->11->8 [5] 8/-1/-1->11->9 [6] 9/-1/-1->11->8 [7] 10/7/15->11->3 [8] -1/-1/-1->11->10 [9] 8/-1/-1->11->10 [10] -1/-1/-1->11->9 [11] 9/-1/-1->11->7 [12] 10/-1/-1->11->8 [13] 8/-1/-1->11->9 [14] 9/-1/-1->11->8 [15] 10/-1/-1->11->7 2: jzxh277:347413:348900 [3] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819788:1821282 [2] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 4/-1/-1->6->10 [3] -1/-1/-1->6->4 [4] 5/-1/-1->6->7 [5] -1/-1/-1->6->4 [6] 4/-1/-1->6->10 [7] 5/-1/-1->6->7 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 4/10/2->6->14 [11] -1/-1/-1->6->4 [12] 5/-1/-1->6->7 [13] -1/-1/-1->6->4 [14] 4/10/2->6->14 [15] 5/-1/-1->6->7 1: jzxh276:1819788:1821282 [2] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438466:2439970 [2] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 06/16 : 0 3 6 5 4 7 10 9 8 11 14 13 12 15 2 1 0: jzxh275:2438467:2439968 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 0/-1/-1->3->2 [2] -1/-1/-1->3->1 [3] 1/11/-1->3->-1 [4] 2/-1/-1->3->0 [5] 0/-1/-1->3->1 [6] 1/-1/-1->3->0 [7] 2/11/-1->3->-1 [8] -1/-1/-1->3->2 [9] 0/-1/-1->3->2 [10] -1/-1/-1->3->1 [11] 1/-1/-1->3->7 [12] 2/-1/-1->3->0 [13] 0/-1/-1->3->1 [14] 1/-1/-1->3->0 [15] 2/-1/-1->3->7 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 07/16 : 0 1 2 7 4 5 6 11 8 9 10 15 12 13 14 3 0: jzxh275:2438467:2439968 [3] NCCL INFO P2P Chunksize set to 131072 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 08/16 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 09/16 : 0 3 2 5 4 7 6 9 8 11 10 13 12 15 14 1 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 10/16 : 0 3 6 5 4 7 10 9 8 11 14 13 12 15 2 1 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 11/16 : 0 1 2 7 4 5 6 11 8 9 10 15 12 13 14 3 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 12/16 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 13/16 : 0 3 2 5 4 7 6 9 8 11 10 13 12 15 14 1 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 14/16 : 0 3 6 5 4 7 10 9 8 11 14 13 12 15 2 1 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 15/16 : 0 1 2 7 4 5 6 11 8 9 10 15 12 13 14 3 0: jzxh275:2438464:2439967 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] -1/-1/-1->0->3 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->1 [4] 3/8/-1->0->-1 [5] 2/-1/-1->0->3 [6] 3/-1/-1->0->2 [7] -1/-1/-1->0->1 [8] 1/-1/-1->0->4 [9] -1/-1/-1->0->3 [10] 1/-1/-1->0->2 [11] 2/-1/-1->0->1 [12] 3/-1/-1->0->4 [13] 2/-1/-1->0->3 [14] 3/-1/-1->0->2 [15] -1/-1/-1->0->1 0: jzxh275:2438464:2439967 [0] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330030:3331528 [1] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->9 [2] 15/-1/-1->13->12 [3] 12/-1/-1->13->15 [4] -1/-1/-1->13->14 [5] 15/-1/-1->13->9 [6] -1/-1/-1->13->15 [7] 12/-1/-1->13->14 [8] 14/-1/-1->13->12 [9] 14/5/-1->13->-1 [10] 15/-1/-1->13->12 [11] 12/-1/-1->13->15 [12] -1/-1/-1->13->14 [13] 15/5/-1->13->-1 [14] -1/-1/-1->13->15 [15] 12/-1/-1->13->14 3: jzxh278:3330031:3331527 [2] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330030:3331528 [1] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330032:3331526 [3] NCCL INFO P2P Chunksize set to 131072 3: jzxh278:3330029:3331529 [0] NCCL INFO Trees [0] 13/-1/-1->12->8 [1] -1/-1/-1->12->15 [2] 13/-1/-1->12->14 [3] 14/-1/-1->12->13 [4] 15/-1/-1->12->8 [5] 14/-1/-1->12->15 [6] 15/-1/-1->12->14 [7] -1/-1/-1->12->13 [8] 13/4/-1->12->-1 [9] -1/-1/-1->12->15 [10] 13/-1/-1->12->14 [11] 14/-1/-1->12->13 [12] 15/4/-1->12->-1 [13] 14/-1/-1->12->15 [14] 15/-1/-1->12->14 [15] -1/-1/-1->12->13 3: jzxh278:3330029:3331529 [0] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819787:1821284 [1] NCCL INFO comm 0x148920141450 rank 5 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 1: jzxh276:1819787:1821284 [1] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->9 [2] 7/-1/-1->5->4 [3] 4/-1/-1->5->7 [4] -1/-1/-1->5->6 [5] 7/-1/-1->5->9 [6] -1/-1/-1->5->7 [7] 4/-1/-1->5->6 [8] 6/-1/-1->5->4 [9] 6/9/1->5->13 [10] 7/-1/-1->5->4 [11] 4/-1/-1->5->7 [12] -1/-1/-1->5->6 [13] 7/9/1->5->13 [14] -1/-1/-1->5->7 [15] 4/-1/-1->5->6 1: jzxh276:1819787:1821284 [1] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819786:1821285 [0] NCCL INFO comm 0x14ed18138c00 rank 4 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 1: jzxh276:1819786:1821285 [0] NCCL INFO Trees [0] 5/-1/-1->4->8 [1] -1/-1/-1->4->7 [2] 5/-1/-1->4->6 [3] 6/-1/-1->4->5 [4] 7/-1/-1->4->8 [5] 6/-1/-1->4->7 [6] 7/-1/-1->4->6 [7] -1/-1/-1->4->5 [8] 5/8/0->4->12 [9] -1/-1/-1->4->7 [10] 5/-1/-1->4->6 [11] 6/-1/-1->4->5 [12] 7/8/0->4->12 [13] 6/-1/-1->4->7 [14] 7/-1/-1->4->6 [15] -1/-1/-1->4->5 1: jzxh276:1819786:1821285 [0] NCCL INFO P2P Chunksize set to 131072 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 00/0 : 5[1] -> 6[2] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 00/0 : 13[1] -> 14[2] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 03/0 : 5[1] -> 6[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 00/0 : 14[2] -> 15[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 00/0 : 6[2] -> 7[3] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 04/0 : 5[1] -> 6[2] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 03/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 07/0 : 5[1] -> 6[2] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 04/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 00/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 04/0 : 14[2] -> 15[3] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 04/0 : 6[2] -> 7[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 03/0 : 9[1] -> 10[2] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 04/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 07/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 04/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 08/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 08/0 : 14[2] -> 15[3] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 08/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 12/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 08/0 : 6[2] -> 7[3] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 08/0 : 5[1] -> 6[2] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 11/0 : 5[1] -> 6[2] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 12/0 : 14[2] -> 15[3] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 12/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 04/0 : 9[1] -> 10[2] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 08/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 08/0 : 13[1] -> 14[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 12/0 : 6[2] -> 7[3] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 12/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 11/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 12/0 : 5[1] -> 6[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 00/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 03/0 : 8[0] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 00/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 04/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 12/0 : 13[1] -> 14[2] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 04/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 15/0 : 5[1] -> 6[2] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 07/0 : 9[1] -> 10[2] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 15/0 : 13[1] -> 14[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 08/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 08/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 00/0 : 4[0] -> 5[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 12/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 8[0] -> 9[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 08/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 03/0 : 4[0] -> 5[1] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 07/0 : 8[0] -> 9[1] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 04/0 : 4[0] -> 5[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 11/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 07/0 : 4[0] -> 5[1] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 08/0 : 8[0] -> 9[1] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 11/0 : 8[0] -> 9[1] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 08/0 : 4[0] -> 5[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 12/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 11/0 : 4[0] -> 5[1] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 12/0 : 8[0] -> 9[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 15/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 4[0] -> 5[1] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 00/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 15/0 : 4[0] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 00/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 04/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 04/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 15/0 : 8[0] -> 9[1] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 08/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 08/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 12/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 00/0 : 12[0] -> 13[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 12/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 01/0 : 8[0] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 03/0 : 12[0] -> 13[1] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 02/0 : 8[0] -> 11[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 05/0 : 8[0] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 04/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 00/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 06/0 : 8[0] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 04/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 07/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 08/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 09/0 : 8[0] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 08/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 12/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 10/0 : 8[0] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 11/0 : 12[0] -> 13[1] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 13/0 : 8[0] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 01/0 : 4[0] -> 7[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 14/0 : 8[0] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 12/0 : 12[0] -> 13[1] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 15/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 02/0 : 4[0] -> 7[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 05/0 : 4[0] -> 7[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 01/0 : 12[0] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 06/0 : 4[0] -> 7[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 02/0 : 12[0] -> 15[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 09/0 : 4[0] -> 7[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 05/0 : 12[0] -> 15[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 06/0 : 12[0] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 10/0 : 4[0] -> 7[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 09/0 : 12[0] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 13/0 : 4[0] -> 7[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 10/0 : 12[0] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 14/0 : 4[0] -> 7[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 13/0 : 12[0] -> 15[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 14/0 : 12[0] -> 15[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 01/0 : 0[0] -> 3[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 02/0 : 0[0] -> 3[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 05/0 : 0[0] -> 3[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 06/0 : 0[0] -> 3[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 09/0 : 0[0] -> 3[3] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 02/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 02/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 06/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 06/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 10/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 14/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 10/0 : 0[0] -> 3[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 13/0 : 0[0] -> 3[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 14/0 : 0[0] -> 3[3] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 02/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 06/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 10/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 10/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 14/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 14/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 02/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 02/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 06/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 06/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 10/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 10/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 14/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 14/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 01/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 01/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 05/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 09/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 09/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 01/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 05/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 05/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 09/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 13/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 13/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 13/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 02/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 02/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 06/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 06/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 10/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 10/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 14/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 14/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 9[1] -> 8[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 01/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 01/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 02/0 : 9[1] -> 8[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 05/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 05/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 9[1] -> 8[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 09/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 09/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 13/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 13/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 06/0 : 9[1] -> 8[0] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 09/0 : 9[1] -> 8[0] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 10/0 : 9[1] -> 8[0] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 13/0 : 9[1] -> 8[0] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 14/0 : 9[1] -> 8[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 01/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 01/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 05/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 05/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 09/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 09/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 13/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 13/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 01/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 02/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 05/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 06/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 09/0 : 13[1] -> 12[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 01/0 : 5[1] -> 4[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 02/0 : 5[1] -> 4[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 10/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 13/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 05/0 : 5[1] -> 4[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 14/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 06/0 : 5[1] -> 4[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 10/0 : 5[1] -> 4[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 14/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 03/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 03/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 07/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 07/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 11/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 15/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 03/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 03/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 07/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 07/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 11/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 11/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 15/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 15/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 03/0 : 7[3] -> 4[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 03/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 07/0 : 7[3] -> 4[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 07/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 7[3] -> 4[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 11/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 11/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 15/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 15/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 7[3] -> 4[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 01/0 : 7[3] -> 6[2] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 11/0 : 11[3] -> 8[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 15/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 05/0 : 7[3] -> 6[2] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 01/0 : 11[3] -> 10[2] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 03/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 03/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 07/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 09/0 : 7[3] -> 6[2] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 07/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 11/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 11/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 05/0 : 11[3] -> 10[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 03/0 : 3[3] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 15/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 15/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 03/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 09/0 : 11[3] -> 10[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 07/0 : 3[3] -> 0[0] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 13/0 : 7[3] -> 6[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 11/0 : 3[3] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 07/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 13/0 : 11[3] -> 10[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 02/0 : 6[2] -> 5[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 15/0 : 3[3] -> 0[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 06/0 : 6[2] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 11/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 6[2] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 15/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 6[2] -> 5[1] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 02/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 01/0 : 15[3] -> 14[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 10[2] -> 9[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 05/0 : 15[3] -> 14[2] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 10[2] -> 9[1] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 06/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 10/0 : 10[2] -> 9[1] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 10/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 09/0 : 15[3] -> 14[2] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 14/0 : 10[2] -> 9[1] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 14/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 13/0 : 15[3] -> 14[2] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Connected all rings 2: jzxh277:347412:348899 [2] NCCL INFO Connected all rings 2: jzxh277:347413:348900 [3] NCCL INFO Connected all rings 2: jzxh277:347410:348901 [0] NCCL INFO Connected all rings 2: jzxh277:347411:348898 [1] NCCL INFO Connected all rings 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 02/0 : 12[0] -> 13[1] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Connected all rings 2: jzxh277:347410:348901 [0] NCCL INFO Channel 02/0 : 8[0] -> 9[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Connected all rings 3: jzxh278:3330031:3331527 [2] NCCL INFO Connected all rings 0: jzxh275:2438467:2439968 [3] NCCL INFO Connected all rings 0: jzxh275:2438466:2439970 [2] NCCL INFO Connected all rings 0: jzxh275:2438464:2439967 [0] NCCL INFO Connected all rings 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 10/0 : 12[0] -> 13[1] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Connected all rings 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Connected all rings 1: jzxh276:1819789:1821283 [3] NCCL INFO Connected all rings 2: jzxh277:347410:348901 [0] NCCL INFO Channel 10/0 : 8[0] -> 9[1] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Connected all rings 1: jzxh276:1819787:1821284 [1] NCCL INFO Connected all rings 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 02/0 : 4[0] -> 5[1] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 01/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 01/0 : 14[2] -> 15[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 10/0 : 4[0] -> 5[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 09/0 : 13[1] -> 14[2] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 07/0 : 14[2] -> 15[3] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 07/0 : 10[2] -> 11[3] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 01/0 : 5[1] -> 6[2] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 02/0 : 12[0] -> 14[2] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 09/0 : 14[2] -> 15[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 03/0 : 12[0] -> 14[2] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 09/0 : 9[1] -> 10[2] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 5[1] -> 6[2] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 15/0 : 14[2] -> 15[3] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 09/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 05/0 : 12[0] -> 14[2] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 02/0 : 13[1] -> 15[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 06/0 : 12[0] -> 14[2] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 02/0 : 8[0] -> 10[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 01/0 : 6[2] -> 7[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 03/0 : 8[0] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 02/0 : 4[0] -> 6[2] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 15/0 : 10[2] -> 11[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 10/0 : 12[0] -> 14[2] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 05/0 : 8[0] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 03/0 : 4[0] -> 6[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 07/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 03/0 : 13[1] -> 15[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 02/0 : 9[1] -> 11[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 06/0 : 8[0] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 05/0 : 4[0] -> 6[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 09/0 : 6[2] -> 7[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 11/0 : 12[0] -> 14[2] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 10/0 : 8[0] -> 10[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 06/0 : 4[0] -> 6[2] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 15/0 : 6[2] -> 7[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 03/0 : 9[1] -> 11[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 10/0 : 4[0] -> 6[2] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 05/0 : 13[1] -> 15[3] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 02/0 : 5[1] -> 7[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 11/0 : 4[0] -> 6[2] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 13/0 : 12[0] -> 14[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 03/0 : 5[1] -> 7[3] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 14/0 : 12[0] -> 14[2] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 9[1] -> 11[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 06/0 : 9[1] -> 11[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 13/0 : 4[0] -> 6[2] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 02/0 : 0[0] -> 2[2] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 05/0 : 5[1] -> 7[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 11/0 : 8[0] -> 10[2] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 06/0 : 13[1] -> 15[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 05/0 : 0[0] -> 2[2] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 14/0 : 4[0] -> 6[2] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 04/0 : 12[0] -> 15[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 10/0 : 9[1] -> 11[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 13/0 : 8[0] -> 10[2] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 06/0 : 5[1] -> 7[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 10/0 : 13[1] -> 15[3] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 02/0 : 10[2] -> 14[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 06/0 : 0[0] -> 2[2] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 06/0 : 10[2] -> 14[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 12/0 : 12[0] -> 15[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 11/0 : 9[1] -> 11[3] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 11/0 : 13[1] -> 15[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 10/0 : 0[0] -> 2[2] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 14/0 : 8[0] -> 10[2] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 10/0 : 5[1] -> 7[3] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 13/0 : 13[1] -> 15[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 11/0 : 0[0] -> 2[2] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 14/0 : 13[1] -> 15[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 13/0 : 9[1] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 13/0 : 0[0] -> 2[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 02/0 : 1[1] -> 3[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 14/0 : 9[1] -> 11[3] via P2P/CUMEM 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 14/0 : 0[0] -> 2[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 01/0 : 9[1] -> 13[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 05/0 : 1[1] -> 3[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 05/0 : 9[1] -> 13[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 03/0 : 11[3] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 06/0 : 1[1] -> 3[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 8[0] -> 11[3] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 07/0 : 11[3] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 04/0 : 8[0] -> 12[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 11/0 : 5[1] -> 7[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 09/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 13/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 10/0 : 1[1] -> 3[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 9[1] -> 13[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 10/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 12/0 : 8[0] -> 11[3] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 9[1] -> 13[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 14/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 10[2] -> 14[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 10[2] -> 14[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 5[1] -> 7[3] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 09/0 : 5[1] -> 13[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 13/0 : 5[1] -> 13[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 00/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 09/0 : 13[1] -> 5[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 13/0 : 13[1] -> 5[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 11/0 : 1[1] -> 3[3] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 14/0 : 5[1] -> 7[3] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 10/0 : 6[2] -> 14[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 11/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 14/0 : 6[2] -> 14[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 08/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 10/0 : 14[2] -> 6[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 15/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 12/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 13/0 : 1[1] -> 3[3] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 04/0 : 4[0] -> 7[3] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 2[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 2[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 02/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 11[3] -> 15[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 14/0 : 14[2] -> 6[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 11[3] -> 15[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 8[0] -> 12[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 06/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 4[0] -> 7[3] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 11/0 : 7[3] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 08/0 : 4[0] -> 12[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 15/0 : 7[3] -> 15[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 12/0 : 4[0] -> 12[0] [receive] via NET/IB/0/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 14/0 : 1[1] -> 3[3] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 2[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 08/0 : 12[0] -> 4[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 2[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 12/0 : 12[0] -> 4[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 10[2] -> 2[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 10[2] -> 2[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 04/0 : 0[0] -> 3[3] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 1[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 10/0 : 2[2] -> 6[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 09/0 : 1[1] -> 5[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 14/0 : 2[2] -> 6[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 13/0 : 1[1] -> 5[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 1[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 01/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 02/0 : 10[2] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 12/0 : 0[0] -> 3[3] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 05/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 06/0 : 10[2] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 02/0 : 2[2] -> 10[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 06/0 : 2[2] -> 10[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 01/0 : 1[1] -> 9[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 1[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 08/0 : 0[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 14[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 3[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 05/0 : 1[1] -> 9[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 14[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 0[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 3[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 13[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 6[2] -> 14[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 03/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 00/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 10/0 : 6[2] -> 2[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 14/0 : 6[2] -> 2[2] [receive] via NET/IB/2/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 1[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 02/0 : 14[2] -> 10[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 08/0 : 0[0] -> 4[0] [send] via NET/IB/0/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 11/0 : 3[3] -> 7[3] [send] via NET/IB/3/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 12/0 : 0[0] -> 4[0] [send] via NET/IB/0/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 15/0 : 3[3] -> 7[3] [send] via NET/IB/3/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 09/0 : 5[1] -> 1[1] [receive] via NET/IB/1/GDRDMA 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 13/0 : 5[1] -> 1[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 14[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 6[2] -> 14[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 13[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 07/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 06/0 : 14[2] -> 10[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 14[2] -> 10[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 04/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 11/0 : 15[3] -> 7[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 13[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 5[1] -> 13[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 15/0 : 15[3] -> 7[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 08/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 5[1] -> 13[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 02/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 06/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 04/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 01/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 03/0 : 3[3] -> 11[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 01/0 : 13[1] -> 9[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 13[1] -> 9[1] [receive] via NET/IB/1/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 3[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 05/0 : 13[1] -> 9[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 04/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 10/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 07/0 : 3[3] -> 11[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 01/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 3[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 14/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 02/0 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 05/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347411:348898 [1] NCCL INFO Channel 09/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 05/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 6[2] -> 2[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 6[2] -> 2[2] [send] via NET/IB/2/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 08/0 : 12[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 15[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 08/0 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 09/0 : 5[1] -> 1[1] [send] via NET/IB/1/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 11/0 : 7[3] -> 3[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 12[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 15[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 02/0 : 14[2] -> 12[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 13/0 : 5[1] -> 1[1] [send] via NET/IB/1/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 03/0 : 14[2] -> 12[0] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 08/0 : 4[0] -> 12[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 7[3] -> 15[3] [send] via NET/IB/3/GDRDMA 0: jzxh275:2438464:2439967 [0] NCCL INFO Channel 12/0 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 02/0 : 6[2] -> 4[0] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 13/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 7[3] -> 15[3] [send] via NET/IB/3/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 15/0 : 7[3] -> 3[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 02/0 : 10[2] -> 8[0] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 4[0] -> 12[0] [send] via NET/IB/0/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 03/0 : 6[2] -> 4[0] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 03/0 : 15[3] -> 11[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 07/0 : 15[3] -> 11[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 05/0 : 14[2] -> 12[0] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 00/0 : 12[0] -> 8[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 03/0 : 10[2] -> 8[0] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 00/0 : 12[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 15[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 12[0] -> 8[0] [receive] via NET/IB/0/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 15[3] -> 11[3] [receive] via NET/IB/3/GDRDMA 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 05/0 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 07/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 04/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 3: jzxh278:3330029:3331529 [0] NCCL INFO Channel 04/0 : 12[0] -> 8[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 08/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 05/0 : 6[2] -> 4[0] via P2P/CUMEM 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 7[3] -> 3[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 08/0 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 01/0 : 3[3] -> 0[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 06/0 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA 2: jzxh277:347410:348901 [0] NCCL INFO Channel 04/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 05/0 : 10[2] -> 8[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 06/0 : 14[2] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 11/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 01/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 08/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 7[3] -> 3[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819786:1821285 [0] NCCL INFO Channel 12/0 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 06/0 : 6[2] -> 4[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 15/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 01/0 : 7[3] -> 4[0] via P2P/CUMEM 2: jzxh277:347410:348901 [0] NCCL INFO Channel 12/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA 2: jzxh277:347412:348899 [2] NCCL INFO Channel 06/0 : 10[2] -> 8[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 04/0 : 3[3] -> 0[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 10/0 : 6[2] -> 4[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 10/0 : 14[2] -> 12[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 10/0 : 2[2] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 01/0 : 11[3] -> 8[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 04/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 10/0 : 10[2] -> 8[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 04/0 : 11[3] -> 8[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 11/0 : 6[2] -> 4[0] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 04/0 : 7[3] -> 4[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 05/0 : 3[3] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 05/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 11/0 : 10[2] -> 8[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 11/0 : 14[2] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 05/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 11/0 : 2[2] -> 0[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 13/0 : 6[2] -> 4[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 06/0 : 3[3] -> 0[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 06/0 : 15[3] -> 12[0] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 05/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 13/0 : 14[2] -> 12[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 09/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 06/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 09/0 : 3[3] -> 0[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 13/0 : 2[2] -> 0[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 14/0 : 6[2] -> 4[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 13/0 : 10[2] -> 8[0] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 06/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 12/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 14/0 : 10[2] -> 8[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 09/0 : 11[3] -> 8[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 14/0 : 14[2] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 12/0 : 3[3] -> 0[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 14/0 : 2[2] -> 0[0] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 09/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 13/0 : 15[3] -> 12[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 12/0 : 11[3] -> 8[0] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 12/0 : 7[3] -> 4[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 14/0 : 15[3] -> 12[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 13/0 : 3[3] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 13/0 : 11[3] -> 8[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 14/0 : 3[3] -> 0[0] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 14/0 : 11[3] -> 8[0] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 02/0 : 15[3] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 02/0 : 3[3] -> 1[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 02/0 : 11[3] -> 9[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 03/0 : 15[3] -> 13[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 03/0 : 11[3] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 13/0 : 7[3] -> 4[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 05/0 : 15[3] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 05/0 : 3[3] -> 1[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 05/0 : 11[3] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 14/0 : 7[3] -> 4[0] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 06/0 : 3[3] -> 1[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 06/0 : 11[3] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 02/0 : 7[3] -> 5[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 10/0 : 3[3] -> 1[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 06/0 : 15[3] -> 13[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 10/0 : 11[3] -> 9[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 03/0 : 7[3] -> 5[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 11/0 : 11[3] -> 9[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 10/0 : 15[3] -> 13[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 11/0 : 3[3] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 05/0 : 7[3] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 11/0 : 15[3] -> 13[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 13/0 : 11[3] -> 9[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 13/0 : 3[3] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 06/0 : 7[3] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 13/0 : 15[3] -> 13[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 14/0 : 11[3] -> 9[1] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 14/0 : 3[3] -> 1[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 10/0 : 7[3] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 14/0 : 15[3] -> 13[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 11/0 : 7[3] -> 5[1] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 00/0 : 11[3] -> 10[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 00/0 : 15[3] -> 14[2] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 13/0 : 7[3] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 04/0 : 15[3] -> 14[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 14/0 : 7[3] -> 5[1] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 07/0 : 15[3] -> 14[2] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 00/0 : 7[3] -> 6[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 04/0 : 11[3] -> 10[2] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 07/0 : 11[3] -> 10[2] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 08/0 : 11[3] -> 10[2] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 08/0 : 15[3] -> 14[2] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 04/0 : 7[3] -> 6[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 12/0 : 11[3] -> 10[2] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 12/0 : 15[3] -> 14[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 07/0 : 7[3] -> 6[2] via P2P/CUMEM 2: jzxh277:347413:348900 [3] NCCL INFO Channel 15/0 : 11[3] -> 10[2] via P2P/CUMEM 3: jzxh278:3330032:3331526 [3] NCCL INFO Channel 15/0 : 15[3] -> 14[2] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 08/0 : 7[3] -> 6[2] via P2P/CUMEM 0: jzxh275:2438467:2439968 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 00/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 01/0 : 14[2] -> 13[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 12/0 : 7[3] -> 6[2] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 04/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 07/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 00/0 : 13[1] -> 12[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 00/0 : 10[2] -> 9[1] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 00/0 : 6[2] -> 5[1] via P2P/CUMEM 1: jzxh276:1819789:1821283 [3] NCCL INFO Channel 15/0 : 7[3] -> 6[2] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 03/0 : 13[1] -> 12[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 08/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 01/0 : 10[2] -> 9[1] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 09/0 : 14[2] -> 13[1] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 04/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 01/0 : 6[2] -> 5[1] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 07/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 08/0 : 13[1] -> 12[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 00/0 : 9[1] -> 8[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 12/0 : 14[2] -> 13[1] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 00/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 04/0 : 6[2] -> 5[1] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM 3: jzxh278:3330031:3331527 [2] NCCL INFO Channel 15/0 : 14[2] -> 13[1] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 11/0 : 13[1] -> 12[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 07/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 07/0 : 6[2] -> 5[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 03/0 : 9[1] -> 8[0] via P2P/CUMEM 0: jzxh275:2438466:2439970 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 07/0 : 9[1] -> 8[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 08/0 : 10[2] -> 9[1] via P2P/CUMEM 3: jzxh278:3330030:3331528 [1] NCCL INFO Channel 15/0 : 13[1] -> 12[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 03/0 : 5[1] -> 4[0] via P2P/CUMEM 0: jzxh275:2438465:2439969 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 09/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 08/0 : 6[2] -> 5[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 08/0 : 9[1] -> 8[0] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 07/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 09/0 : 6[2] -> 5[1] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 08/0 : 5[1] -> 4[0] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 12/0 : 6[2] -> 5[1] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 11/0 : 5[1] -> 4[0] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 11/0 : 9[1] -> 8[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 12/0 : 10[2] -> 9[1] via P2P/CUMEM 1: jzxh276:1819788:1821282 [2] NCCL INFO Channel 15/0 : 6[2] -> 5[1] via P2P/CUMEM 1: jzxh276:1819787:1821284 [1] NCCL INFO Channel 15/0 : 5[1] -> 4[0] via P2P/CUMEM 2: jzxh277:347412:348899 [2] NCCL INFO Channel 15/0 : 10[2] -> 9[1] via P2P/CUMEM 2: jzxh277:347411:348898 [1] NCCL INFO Channel 15/0 : 9[1] -> 8[0] via P2P/CUMEM 3: jzxh278:3330029:3331529 [0] NCCL INFO Connected all trees 3: jzxh278:3330029:3331529 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330029:3331529 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347410:348901 [0] NCCL INFO Connected all trees 0: jzxh275:2438464:2439967 [0] NCCL INFO Connected all trees 0: jzxh275:2438467:2439968 [3] NCCL INFO Connected all trees 0: jzxh275:2438465:2439969 [1] NCCL INFO Connected all trees 0: jzxh275:2438467:2439968 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 0: jzxh275:2438467:2439968 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 0: jzxh275:2438465:2439969 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 0: jzxh275:2438465:2439969 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 0: jzxh275:2438466:2439970 [2] NCCL INFO Connected all trees 0: jzxh275:2438464:2439967 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 0: jzxh275:2438464:2439967 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 0: jzxh275:2438466:2439970 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 0: jzxh275:2438466:2439970 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 1: jzxh276:1819786:1821285 [0] NCCL INFO Connected all trees 2: jzxh277:347410:348901 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 2: jzxh277:347410:348901 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347413:348900 [3] NCCL INFO Connected all trees 3: jzxh278:3330032:3331526 [3] NCCL INFO Connected all trees 2: jzxh277:347413:348900 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 2: jzxh277:347411:348898 [1] NCCL INFO Connected all trees 2: jzxh277:347413:348900 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347412:348899 [2] NCCL INFO Connected all trees 2: jzxh277:347411:348898 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330032:3331526 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330032:3331526 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 3: jzxh278:3330030:3331528 [1] NCCL INFO Connected all trees 3: jzxh278:3330031:3331527 [2] NCCL INFO Connected all trees 2: jzxh277:347411:348898 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347412:348899 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 2: jzxh277:347412:348899 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 3: jzxh278:3330030:3331528 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330030:3331528 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 3: jzxh278:3330031:3331527 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 3: jzxh278:3330031:3331527 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 1: jzxh276:1819786:1821285 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 1: jzxh276:1819786:1821285 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 1: jzxh276:1819789:1821283 [3] NCCL INFO Connected all trees 1: jzxh276:1819787:1821284 [1] NCCL INFO Connected all trees 1: jzxh276:1819789:1821283 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 1: jzxh276:1819789:1821283 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 1: jzxh276:1819788:1821282 [2] NCCL INFO Connected all trees 1: jzxh276:1819787:1821284 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 1: jzxh276:1819787:1821284 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 1: jzxh276:1819788:1821282 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 1: jzxh276:1819788:1821282 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer 2: jzxh277:347413:348900 [3] NCCL INFO ncclCommInitRank comm 0x14800014cfc0 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 2: jzxh277:347412:348899 [2] NCCL INFO ncclCommInitRank comm 0x153aa01384c0 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 2: jzxh277:347410:348901 [0] NCCL INFO ncclCommInitRank comm 0x1454ec132610 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 2: jzxh277:347411:348898 [1] NCCL INFO ncclCommInitRank comm 0x148290127000 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 1: jzxh276:1819787:1821284 [1] NCCL INFO ncclCommInitRank comm 0x148920141450 rank 5 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 1: jzxh276:1819789:1821283 [3] NCCL INFO ncclCommInitRank comm 0x14fb60138940 rank 7 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 1: jzxh276:1819786:1821285 [0] NCCL INFO ncclCommInitRank comm 0x14ed18138c00 rank 4 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 1: jzxh276:1819788:1821282 [2] NCCL INFO ncclCommInitRank comm 0x14aad812e700 rank 6 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 0: jzxh275:2438464:2439967 [0] NCCL INFO ncclCommInitRank comm 0x1483dc13e760 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 0: jzxh275:2438466:2439970 [2] NCCL INFO ncclCommInitRank comm 0x145130129740 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 0: jzxh275:2438467:2439968 [3] NCCL INFO ncclCommInitRank comm 0x144eb01303b0 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 0: jzxh275:2438465:2439969 [1] NCCL INFO ncclCommInitRank comm 0x14bdac12f380 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 3: jzxh278:3330031:3331527 [2] NCCL INFO ncclCommInitRank comm 0x149e2412ed10 rank 14 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 3: jzxh278:3330032:3331526 [3] NCCL INFO ncclCommInitRank comm 0x150804130040 rank 15 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 3: jzxh278:3330029:3331529 [0] NCCL INFO ncclCommInitRank comm 0x151574137e00 rank 12 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 3: jzxh278:3330030:3331528 [1] NCCL INFO ncclCommInitRank comm 0x154b80130140 rank 13 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x6e9cb86670bfb8a0 - Init COMPLETE 0: {'loss': 1.2128, 'grad_norm': 2.0730148245986832, 'learning_rate': 3.62e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.0} 0: 0%| | 1/3723 [00:36<37:37:15, 36.39s/it] 0%| | 2/3723 [00:45<20:49:36, 20.15s/it] 0%| | 3/3723 [00:50<13:40:47, 13.24s/it] 0%| | 4/3723 [00:55<10:19:09, 9.99s/it] 0%| | 5/3723 [01:00<8:28:20, 8.20s/it] 0%| | 6/3723 [01:05<7:21:31, 7.13s/it] 0%| | 7/3723 [01:10<6:38:43, 6.44s/it] 0%| | 8/3723 [01:15<6:09:30, 5.97s/it] 0%| | 9/3723 [01:20<5:49:58, 5.65s/it] 0%| | 10/3723 [01:25<5:35:39, 5.42s/it] 0%| | 10/3723 [01:25<5:35:39, 5.42s/it] 0%| | 11/3723 [01:30<5:25:33, 5.26s/it] 0%| | 12/3723 [01:34<5:17:51, 5.14s/it] 0%| | 13/3723 [01:39<5:12:51, 5.06s/it] 0%| | 14/3723 [01:44<5:08:45, 4.99s/it] 0%| | 15/3723 [01:49<5:06:14, 4.96s/it] 0%| | 16/3723 [01:54<5:04:07, 4.92s/it] 0%| | 17/3723 [01:59<5:03:02, 4.91s/it] 0%| | 18/3723 [02:04<5:02:1 0: {'loss': 1.1334, 'grad_norm': 1.1957056085843274, 'learning_rate': 5.420000000000001e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.0} 0: {'loss': 1.0697, 'grad_norm': 1.0779496159251154, 'learning_rate': 7.22e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.0} 0: 2, 4.89s/it] 1%| | 19/3723 [02:09<5:03:31, 4.92s/it] 1%| | 20/3723 [02:13<5:03:13, 4.91s/it] 1%| | 20/3723 [02:13<5:03:13, 4.91s/it] 1%| | 21/3723 [02:18<5:02:44, 4.91s/it] 1%| | 22/3723 [02:23<5:02:20, 4.90s/it] 1%| | 23/3723 [02:28<5:01:47, 4.89s/it] 1%| | 24/3723 [02:33<5:01:04, 4.88s/it] 1%| | 25/3723 [02:38<5:00:25, 4.87s/it] 1%| | 26/3723 [02:43<5:00:15, 4.87s/it] 1%| | 27/3723 [02:48<4:59:52, 4.87s/it] 1%| | 28/3723 [02:52<4:59:19, 4.86s/it] 1%| | 29/3723 [02:57<4:59:27, 4.86s/it] 1%| | 30/3723 [03:02<4:59:30, 4.87s/it] 1%| | 30/3723 [03:02<4:59:30, 4.87s/it] 1%| | 31/3723 [03:07<4:59:17, 4.86s/it] 1%| | 32/3723 [03:12<4:59:44, 4.87s/it] 1%| | 33/3723 [03:17<4:59:35, 4.87s/it] 1%| | 34/ 0: {'loss': 1.0208, 'grad_norm': 1.1064145019218572, 'learning_rate': 9.020000000000002e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.01} 0: 3723 [03:22<4:58:41, 4.86s/it] 1%| | 35/3723 [03:26<4:57:52, 4.85s/it] 1%| | 36/3723 [03:31<4:57:01, 4.83s/it] 1%| | 37/3723 [03:36<4:57:12, 4.84s/it] 1%| | 38/3723 [03:41<4:55:40, 4.81s/it] 1%| | 39/3723 [03:46<4:55:24, 4.81s/it] 1%| | 40/3723 [03:50<4:54:46, 4.80s/it] 1%| | 40/3723 [03:50<4:54:46, 4.80s/it] 1%| | 41/3723 [03:55<4:54:44, 4.80s/it] 1%| | 42/3723 [04:00<4:54:21, 4.80s/it] 1%| | 43/3723 [04:05<4:54:03, 4.79s/it] 1%| | 44/3723 [04:10<4:54:01, 4.80s/it] 1%| | 45/3723 [04:14<4:54:13, 4.80s/it] 1%| | 46/3723 [04:19<4:53:57, 4.80s/it] 1%|▏ | 47/3723 [04:24<4:53:45, 4.79s/it] 1%|▏ | 48/3723 [04:29<4:53:55, 4.80s/it] 1%|▏ | 49/3723 [04:34<4:54:01, 4.80s/it] 1%|▏ | 50/3723 [04:38<4:54:05, 4.80s/it] 0: {'loss': 0.9876, 'grad_norm': 1.2034771219150349, 'learning_rate': 1.0820000000000001e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.01} 0: {'loss': 0.9628, 'grad_norm': 1.3552557728483567, 'learning_rate': 1.2620000000000001e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.01} 0: 1%|▏ | 50/3723 [04:38<4:54:05, 4.80s/it] 1%|▏ | 51/3723 [04:43<4:54:03, 4.80s/it] 1%|▏ | 52/3723 [04:48<4:53:33, 4.80s/it] 1%|▏ | 53/3723 [04:53<4:53:18, 4.80s/it] 1%|▏ | 54/3723 [04:58<4:53:30, 4.80s/it] 1%|▏ | 55/3723 [05:02<4:53:26, 4.80s/it] 2%|▏ | 56/3723 [05:07<4:53:20, 4.80s/it] 2%|▏ | 57/3723 [05:12<4:53:42, 4.81s/it] 2%|▏ | 58/3723 [05:17<4:53:49, 4.81s/it] 2%|▏ | 59/3723 [05:22<4:54:03, 4.82s/it] 2%|▏ | 60/3723 [05:26<4:54:04, 4.82s/it] 2%|▏ | 60/3723 [05:26<4:54:04, 4.82s/it] 2%|▏ | 61/3723 [05:31<4:54:27, 4.82s/it] 2%|▏ | 62/3723 [05:36<4:54:08, 4.82s/it] 2%|▏ | 63/3723 [05:41<4:53:27, 4.81s/it] 2%|▏ | 64/3723 [05:46<4:52:55, 4.80s/it] 2%|▏ | 65/3723 [05:50<4:52:41, 4.80s/it] 2%|▏ | 66/3723 [05:55<4:53:10, 4.8 0: {'loss': 0.9361, 'grad_norm': 1.0725734105314813, 'learning_rate': 1.4420000000000001e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.01} 0: {'loss': 0.9358, 'grad_norm': 1.2305020234192683, 'learning_rate': 1.6220000000000004e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.01} 0: 1s/it] 2%|▏ | 67/3723 [06:00<4:53:37, 4.82s/it] 2%|▏ | 68/3723 [06:05<4:53:47, 4.82s/it] 2%|▏ | 69/3723 [06:10<4:53:34, 4.82s/it] 2%|▏ | 70/3723 [06:15<4:53:40, 4.82s/it] 2%|▏ | 70/3723 [06:15<4:53:40, 4.82s/it] 2%|▏ | 71/3723 [06:19<4:53:31, 4.82s/it] 2%|▏ | 72/3723 [06:24<4:53:09, 4.82s/it] 2%|▏ | 73/3723 [06:29<4:52:38, 4.81s/it] 2%|▏ | 74/3723 [06:34<4:52:33, 4.81s/it] 2%|▏ | 75/3723 [06:39<4:52:54, 4.82s/it] 2%|▏ | 76/3723 [06:43<4:53:10, 4.82s/it] 2%|▏ | 77/3723 [06:48<4:53:11, 4.82s/it] 2%|▏ | 78/3723 [06:53<4:52:55, 4.82s/it] 2%|▏ | 79/3723 [06:58<4:53:01, 4.82s/it] 2%|▏ | 80/3723 [07:03<4:52:37, 4.82s/it] 2%|▏ | 80/3723 [07:03<4:52:37, 4.82s/it] 2%|▏ | 81/3723 [07:08<4:54:40, 4.8 0: {'loss': 0.9092, 'grad_norm': 1.1956857491045345, 'learning_rate': 1.802e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.01} 0: 5s/it] 2%|▏ | 82/3723 [07:13<4:56:40, 4.89s/it] 2%|▏ | 83/3723 [07:18<4:58:01, 4.91s/it] 2%|▏ | 84/3723 [07:23<4:58:51, 4.93s/it] 2%|▏ | 85/3723 [07:28<4:59:17, 4.94s/it] 2%|▏ | 86/3723 [07:33<4:59:41, 4.94s/it] 2%|▏ | 87/3723 [07:37<4:57:40, 4.91s/it] 2%|▏ | 88/3723 [07:42<4:55:34, 4.88s/it] 2%|▏ | 89/3723 [07:47<4:53:42, 4.85s/it] 2%|▏ | 90/3723 [07:52<4:52:19, 4.83s/it] 2%|▏ | 90/3723 [07:52<4:52:19, 4.83s/it] 2%|▏ | 91/3723 [07:57<4:51:35, 4.82s/it] 2%|▏ | 92/3723 [08:01<4:51:02, 4.81s/it] 2%|▏ | 93/3723 [08:06<4:50:26, 4.80s/it] 3%|▎ | 94/3723 [08:11<4:50:02, 4.80s/it] 3%|▎ | 95/3723 [08:16<4:49:37, 4.79s/it] 3%|▎ | 96/3723 [08:20<4:49:34, 4.79s/it] 3%|▎ | 97/3723 [08:25<4:49:43, 4.79s/it] 3%|▎ | 98/3723 [08:30<4:49:47, 4. 0: {'loss': 0.9156, 'grad_norm': 1.1048058211484273, 'learning_rate': 1.982e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.02} 0: {'loss': 0.9386, 'grad_norm': 1.1677812520820012, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.02} 0: 80s/it] 3%|▎ | 99/3723 [08:35<4:49:56, 4.80s/it] 3%|▎ | 100/3723 [08:40<4:50:00, 4.80s/it] 3%|▎ | 100/3723 [08:40<4:50:00, 4.80s/it] 3%|▎ | 101/3723 [08:45<4:50:56, 4.82s/it] 3%|▎ | 102/3723 [08:49<4:51:52, 4.84s/it] 3%|▎ | 103/3723 [08:54<4:52:11, 4.84s/it] 3%|▎ | 104/3723 [08:59<4:52:02, 4.84s/it] 3%|▎ | 105/3723 [09:04<4:52:15, 4.85s/it] 3%|▎ | 106/3723 [09:09<4:51:55, 4.84s/it] 3%|▎ | 107/3723 [09:14<4:52:00, 4.85s/it] 3%|▎ | 108/3723 [09:18<4:51:48, 4.84s/it] 3%|▎ | 109/3723 [09:23<4:51:53, 4.85s/it] 3%|▎ | 110/3723 [09:28<4:51:44, 4.84s/it] 3%|▎ | 110/3723 [09:28<4:51:44, 4.84s/it] 3%|▎ | 111/3723 [09:33<4:52:00, 4.85s/it] 3%|▎ | 112/3723 [09:38<4:52:14, 4.86s/it] 3%|▎ | 113/3723 [ 0: {'loss': 0.9167, 'grad_norm': 1.1640855912353483, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.02} 0: 09:43<4:52:41, 4.86s/it] 3%|▎ | 114/3723 [09:48<4:52:07, 4.86s/it] 3%|▎ | 115/3723 [09:52<4:52:06, 4.86s/it] 3%|▎ | 116/3723 [09:57<4:51:30, 4.85s/it] 3%|▎ | 117/3723 [10:02<4:50:52, 4.84s/it] 3%|▎ | 118/3723 [10:07<4:49:09, 4.81s/it] 3%|▎ | 119/3723 [10:12<4:48:08, 4.80s/it] 3%|▎ | 120/3723 [10:16<4:47:26, 4.79s/it] 3%|▎ | 120/3723 [10:16<4:47:26, 4.79s/it] 3%|▎ | 121/3723 [10:21<4:47:15, 4.78s/it] 3%|▎ | 122/3723 [10:26<4:47:28, 4.79s/it] 3%|▎ | 123/3723 [10:31<4:47:16, 4.79s/it] 3%|▎ | 124/3723 [10:36<4:47:15, 4.79s/it] 3%|▎ | 125/3723 [10:40<4:46:40, 4.78s/it] 3%|▎ | 126/3723 [10:45<4:46:04, 4.77s/it] 3%|▎ | 127/3723 [10:50<4:45:37, 4.77s/it] 3%|▎ | 128/3723 [10:55<4:48:01, 4.81s/it] 3%|▎ | 129/3723 [11:00<4:49:41, 4.84s/it] 3%|▎ 0: {'loss': 0.8854, 'grad_norm': 1.3003805099264718, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.02} 0: {'loss': 0.8831, 'grad_norm': 1.1745546380126552, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.02} 0: | 130/3723 [11:05<4:50:09, 4.85s/it] 3%|▎ | 130/3723 [11:05<4:50:09, 4.85s/it] 4%|▎ | 131/3723 [11:09<4:51:35, 4.87s/it] 4%|▎ | 132/3723 [11:14<4:50:16, 4.85s/it] 4%|▎ | 133/3723 [11:19<4:48:39, 4.82s/it] 4%|▎ | 134/3723 [11:24<4:46:51, 4.80s/it] 4%|▎ | 135/3723 [11:28<4:45:20, 4.77s/it] 4%|▎ | 136/3723 [11:33<4:44:37, 4.76s/it] 4%|▎ | 137/3723 [11:38<4:44:08, 4.75s/it] 4%|▎ | 138/3723 [11:43<4:43:46, 4.75s/it] 4%|▎ | 139/3723 [11:47<4:43:15, 4.74s/it] 4%|▍ | 140/3723 [11:52<4:42:27, 4.73s/it] 4%|▍ | 140/3723 [11:52<4:42:27, 4.73s/it] 4%|▍ | 141/3723 [11:57<4:41:48, 4.72s/it] 4%|▍ | 142/3723 [12:01<4:41:00, 4.71s/it] 4%|▍ | 143/3723 [12:06<4:40:44, 4.71s/it] 4%|▍ | 144/3723 [12:11<4:40:29, 0: {'loss': 0.8642, 'grad_norm': 1.313951792939279, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.02} 0: 4.70s/it] 4%|▍ | 145/3723 [12:16<4:40:16, 4.70s/it] 4%|▍ | 146/3723 [12:20<4:39:50, 4.69s/it] 4%|▍ | 147/3723 [12:25<4:42:47, 4.74s/it] 4%|▍ | 148/3723 [12:30<4:44:58, 4.78s/it] 4%|▍ | 149/3723 [12:35<4:46:26, 4.81s/it] 4%|▍ | 150/3723 [12:40<4:47:45, 4.83s/it] 4%|▍ | 150/3723 [12:40<4:47:45, 4.83s/it] 4%|▍ | 151/3723 [12:45<4:48:36, 4.85s/it] 4%|▍ | 152/3723 [12:49<4:48:53, 4.85s/it] 4%|▍ | 153/3723 [12:54<4:49:27, 4.86s/it] 4%|▍ | 154/3723 [12:59<4:48:21, 4.85s/it] 4%|▍ | 155/3723 [13:04<4:47:58, 4.84s/it] 4%|▍ | 156/3723 [13:09<4:47:32, 4.84s/it] 4%|▍ | 157/3723 [13:14<4:47:38, 4.84s/it] 4%|▍ | 158/3723 [13:19<4:47:48, 4.84s/it] 4%|▍ | 159/3723 [13:23<4:48:17, 4.85s/it] 4%|▍ | 160/3723 [13:28<4:48:07, 4.85s/it] 0: {'loss': 0.8858, 'grad_norm': 1.1939740042734412, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.03} 0: {'loss': 0.8648, 'grad_norm': 1.0808081832833436, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.03} 0: 4%|▍ | 160/3723 [13:28<4:48:07, 4.85s/it] 4%|▍ | 161/3723 [13:33<4:47:54, 4.85s/it] 4%|▍ | 162/3723 [13:38<4:47:05, 4.84s/it] 4%|▍ | 163/3723 [13:43<4:46:23, 4.83s/it] 4%|▍ | 164/3723 [13:48<4:47:11, 4.84s/it] 4%|▍ | 165/3723 [13:52<4:47:54, 4.86s/it] 4%|▍ | 166/3723 [13:57<4:48:09, 4.86s/it] 4%|▍ | 167/3723 [14:02<4:45:23, 4.82s/it] 5%|▍ | 168/3723 [14:07<4:43:26, 4.78s/it] 5%|▍ | 169/3723 [14:11<4:41:37, 4.75s/it] 5%|▍ | 170/3723 [14:16<4:40:29, 4.74s/it] 5%|▍ | 170/3723 [14:16<4:40:29, 4.74s/it] 5%|▍ | 171/3723 [14:21<4:39:31, 4.72s/it] 5%|▍ | 172/3723 [14:26<4:39:17, 4.72s/it] 5%|▍ | 173/3723 [14:30<4:38:57, 4.71s/it] 5%|▍ | 174/3723 [14:35<4:38:15, 4.70s/it] 5%|▍ | 175/3723 [14:40<4:38:29, 4.71s/it] 5%| 0: {'loss': 0.8507, 'grad_norm': 1.0862516811419656, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.03} 0: {'loss': 0.8788, 'grad_norm': 1.1431380823872903, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.03} 0: | 176/3723 [14:44<4:38:27, 4.71s/it] 5%|▍ | 177/3723 [14:49<4:42:11, 4.77s/it] 5%|▍ | 178/3723 [14:54<4:41:58, 4.77s/it] 5%|▍ | 179/3723 [14:59<4:41:19, 4.76s/it] 5%|▍ | 180/3723 [15:04<4:41:36, 4.77s/it] 5%|▍ | 180/3723 [15:04<4:41:36, 4.77s/it] 5%|▍ | 181/3723 [15:08<4:43:11, 4.80s/it] 5%|▍ | 182/3723 [15:13<4:44:29, 4.82s/it] 5%|▍ | 183/3723 [15:18<4:45:31, 4.84s/it] 5%|▍ | 184/3723 [15:23<4:46:47, 4.86s/it] 5%|▍ | 185/3723 [15:28<4:49:29, 4.91s/it] 5%|▍ | 186/3723 [15:33<4:48:24, 4.89s/it] 5%|▌ | 187/3723 [15:38<4:47:24, 4.88s/it] 5%|▌ | 188/3723 [15:43<4:47:12, 4.87s/it] 5%|▌ | 189/3723 [15:48<4:46:23, 4.86s/it] 5%|▌ | 190/3723 [15:52<4:45:57, 4.86s/it] 5%|▌ | 190/3723 [15:52<4:45:57 0: {'loss': 0.8547, 'grad_norm': 1.301716095316999, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.03} 0: , 4.86s/it] 5%|▌ | 191/3723 [15:57<4:46:07, 4.86s/it] 5%|▌ | 192/3723 [16:02<4:45:29, 4.85s/it] 5%|▌ | 193/3723 [16:07<4:44:47, 4.84s/it] 5%|▌ | 194/3723 [16:12<4:44:35, 4.84s/it] 5%|▌ | 195/3723 [16:16<4:43:14, 4.82s/it] 5%|▌ | 196/3723 [16:21<4:43:16, 4.82s/it] 5%|▌ | 197/3723 [16:26<4:42:54, 4.81s/it] 5%|▌ | 198/3723 [16:31<4:42:29, 4.81s/it] 5%|▌ | 199/3723 [16:36<4:42:28, 4.81s/it] 5%|▌ | 200/3723 [16:40<4:41:29, 4.79s/it] 5%|▌ | 200/3723 [16:40<4:41:29, 4.79s/it] 5%|▌ | 201/3723 [16:45<4:41:17, 4.79s/it] 5%|▌ | 202/3723 [16:50<4:40:29, 4.78s/it] 5%|▌ | 203/3723 [16:55<4:40:00, 4.77s/it] 5%|▌ | 204/3723 [17:00<4:39:45, 4.77s/it] 6%|▌ | 205/3723 [17:04<4:39:26, 4.77s/it] 6%|▌ | 206/3723 [17:09<4:39:30, 4.77s/it] 6%|▌ | 207 0: {'loss': 0.8538, 'grad_norm': 1.180859562816111, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.03} 0: {'loss': 0.8445, 'grad_norm': 1.090401594629996, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.04} 0: /3723 [17:14<4:39:29, 4.77s/it] 6%|▌ | 208/3723 [17:19<4:40:11, 4.78s/it] 6%|▌ | 209/3723 [17:23<4:40:53, 4.80s/it] 6%|▌ | 210/3723 [17:28<4:40:50, 4.80s/it] 6%|▌ | 210/3723 [17:28<4:40:50, 4.80s/it] 6%|▌ | 211/3723 [17:33<4:41:41, 4.81s/it] 6%|▌ | 212/3723 [17:38<4:41:17, 4.81s/it] 6%|▌ | 213/3723 [17:43<4:40:42, 4.80s/it] 6%|▌ | 214/3723 [17:47<4:40:07, 4.79s/it] 6%|▌ | 215/3723 [17:52<4:39:59, 4.79s/it] 6%|▌ | 216/3723 [17:57<4:39:53, 4.79s/it] 6%|▌ | 217/3723 [18:02<4:38:57, 4.77s/it] 6%|▌ | 218/3723 [18:07<4:38:10, 4.76s/it] 6%|▌ | 219/3723 [18:11<4:37:13, 4.75s/it] 6%|▌ | 220/3723 [18:16<4:36:30, 4.74s/it] 6%|▌ | 220/3723 [18:16<4:36:30, 4.74s/it] 6%|▌ | 221/3723 [18:21<4:36:10, 4.73s/it] 6 0: {'loss': 0.8483, 'grad_norm': 1.2952023292352048, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.04} 0: %|▌ | 222/3723 [18:25<4:36:08, 4.73s/it] 6%|▌ | 223/3723 [18:30<4:36:02, 4.73s/it] 6%|▌ | 224/3723 [18:35<4:36:15, 4.74s/it] 6%|▌ | 225/3723 [18:40<4:36:25, 4.74s/it] 6%|▌ | 226/3723 [18:44<4:36:10, 4.74s/it] 6%|▌ | 227/3723 [18:49<4:36:00, 4.74s/it] 6%|▌ | 228/3723 [18:54<4:36:17, 4.74s/it] 6%|▌ | 229/3723 [18:59<4:36:28, 4.75s/it] 6%|▌ | 230/3723 [19:03<4:36:56, 4.76s/it] 6%|▌ | 230/3723 [19:03<4:36:56, 4.76s/it] 6%|▌ | 231/3723 [19:08<4:37:43, 4.77s/it] 6%|▌ | 232/3723 [19:13<4:37:16, 4.77s/it] 6%|▋ | 233/3723 [19:18<4:37:04, 4.76s/it] 6%|▋ | 234/3723 [19:22<4:36:38, 4.76s/it] 6%|▋ | 235/3723 [19:27<4:36:10, 4.75s/it] 6%|▋ | 236/3723 [19:32<4:36:31, 4.76s/it] 6%|▋ | 237/3723 [19:37<4:37:15, 4.77s/it] 6%|▋ | 238/3723 [19:42<4:3 0: {'loss': 0.8551, 'grad_norm': 1.216194450454156, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.04} 0: {'loss': 0.8443, 'grad_norm': 1.1503701665943242, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.04} 0: 7:52, 4.78s/it] 6%|▋ | 239/3723 [19:46<4:38:08, 4.79s/it] 6%|▋ | 240/3723 [19:51<4:38:15, 4.79s/it] 6%|▋ | 240/3723 [19:51<4:38:15, 4.79s/it] 6%|▋ | 241/3723 [19:56<4:38:46, 4.80s/it] 7%|▋ | 242/3723 [20:01<4:38:43, 4.80s/it] 7%|▋ | 243/3723 [20:06<4:39:00, 4.81s/it] 7%|▋ | 244/3723 [20:10<4:38:35, 4.80s/it] 7%|▋ | 245/3723 [20:15<4:38:14, 4.80s/it] 7%|▋ | 246/3723 [20:20<4:38:00, 4.80s/it] 7%|▋ | 247/3723 [20:25<4:37:31, 4.79s/it] 7%|▋ | 248/3723 [20:30<4:37:19, 4.79s/it] 7%|▋ | 249/3723 [20:34<4:37:28, 4.79s/it] 7%|▋ | 250/3723 [20:39<4:37:26, 4.79s/it] 7%|▋ | 250/3723 [20:39<4:37:26, 4.79s/it] 7%|▋ | 251/3723 [20:44<4:37:46, 4.80s/it] 7%|▋ | 252/3723 [20:49<4:37:22, 4.79s/it] 7%|▋ | 0: {'loss': 0.8425, 'grad_norm': 2.8042164526622972, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.04} 0: 253/3723 [20:54<4:37:00, 4.79s/it] 7%|▋ | 254/3723 [20:58<4:36:46, 4.79s/it] 7%|▋ | 255/3723 [21:03<4:36:17, 4.78s/it] 7%|▋ | 256/3723 [21:08<4:37:49, 4.81s/it] 7%|▋ | 257/3723 [21:13<4:37:57, 4.81s/it] 7%|▋ | 258/3723 [21:18<4:37:22, 4.80s/it] 7%|▋ | 259/3723 [21:22<4:37:36, 4.81s/it] 7%|▋ | 260/3723 [21:27<4:37:55, 4.82s/it] 7%|▋ | 260/3723 [21:27<4:37:55, 4.82s/it] 7%|▋ | 261/3723 [21:32<4:38:17, 4.82s/it] 7%|▋ | 262/3723 [21:37<4:40:07, 4.86s/it] 7%|▋ | 263/3723 [21:42<4:38:28, 4.83s/it] 7%|▋ | 264/3723 [21:46<4:36:36, 4.80s/it] 7%|▋ | 265/3723 [21:51<4:35:36, 4.78s/it] 7%|▋ | 266/3723 [21:56<4:35:02, 4.77s/it] 7%|▋ | 267/3723 [22:01<4:34:51, 4.77s/it] 7%|▋ | 268/3723 [22:06<4:35:16, 4.78s/it] 7%|▋ | 269/3723 [22:10<4:35:30, 4.79s/it] 0: {'loss': 0.8409, 'grad_norm': 1.4472791634783375, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.04} 0: {'loss': 0.8311, 'grad_norm': 1.089661292493689, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.05} 0: 7%|▋ | 270/3723 [22:15<4:35:39, 4.79s/it] 7%|▋ | 270/3723 [22:15<4:35:39, 4.79s/it] 7%|▋ | 271/3723 [22:20<4:36:05, 4.80s/it] 7%|▋ | 272/3723 [22:25<4:36:09, 4.80s/it] 7%|▋ | 273/3723 [22:30<4:36:11, 4.80s/it] 7%|▋ | 274/3723 [22:34<4:35:19, 4.79s/it] 7%|▋ | 275/3723 [22:39<4:36:37, 4.81s/it] 7%|▋ | 276/3723 [22:44<4:36:11, 4.81s/it] 7%|▋ | 277/3723 [22:49<4:35:42, 4.80s/it] 7%|▋ | 278/3723 [22:54<4:34:58, 4.79s/it] 7%|▋ | 279/3723 [22:58<4:34:20, 4.78s/it] 8%|▊ | 280/3723 [23:03<4:33:01, 4.76s/it] 8%|▊ | 280/3723 [23:03<4:33:01, 4.76s/it] 8%|▊ | 281/3723 [23:08<4:33:05, 4.76s/it] 8%|▊ | 282/3723 [23:13<4:32:26, 4.75s/it] 8%|▊ | 283/3723 [23:17<4:32:20, 4.75s/it] 8%|▊ | 284/3723 [23:22< 0: {'loss': 0.8349, 'grad_norm': 1.4453294535267038, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.05} 0: 4:31:46, 4.74s/it] 8%|▊ | 285/3723 [23:27<4:31:54, 4.75s/it] 8%|▊ | 286/3723 [23:31<4:31:52, 4.75s/it] 8%|▊ | 287/3723 [23:36<4:33:21, 4.77s/it] 8%|▊ | 288/3723 [23:43<5:14:02, 5.49s/it] 8%|▊ | 289/3723 [23:48<5:03:27, 5.30s/it] 8%|▊ | 290/3723 [23:53<4:53:59, 5.14s/it] 8%|▊ | 290/3723 [23:53<4:53:59, 5.14s/it] 8%|▊ | 291/3723 [23:58<4:47:49, 5.03s/it] 8%|▊ | 292/3723 [24:03<4:43:18, 4.95s/it] 8%|▊ | 293/3723 [24:07<4:39:46, 4.89s/it] 8%|▊ | 294/3723 [24:12<4:37:45, 4.86s/it] 8%|▊ | 295/3723 [24:17<4:36:51, 4.85s/it] 8%|▊ | 296/3723 [24:22<4:36:11, 4.84s/it] 8%|▊ | 297/3723 [24:27<4:35:49, 4.83s/it] 8%|▊ | 298/3723 [24:31<4:35:21, 4.82s/it] 8%|▊ | 299/3723 [24:36<4:34:38, 4.81s/it] 8%|▊ | 300/3723 [24:41<4:34:27, 4.81s/it] 0: {'loss': 0.8441, 'grad_norm': 1.2977626808496878, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.05} 0: {'loss': 0.8268, 'grad_norm': 1.0972861743433446, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.05} 0: 8%|▊ | 300/3723 [24:41<4:34:27, 4.81s/it] 8%|▊ | 301/3723 [24:46<4:34:22, 4.81s/it] 8%|▊ | 302/3723 [24:51<4:33:26, 4.80s/it] 8%|▊ | 303/3723 [24:55<4:32:32, 4.78s/it] 8%|▊ | 304/3723 [25:00<4:32:28, 4.78s/it] 8%|▊ | 305/3723 [25:05<4:33:06, 4.79s/it] 8%|▊ | 306/3723 [25:10<4:32:30, 4.78s/it] 8%|▊ | 307/3723 [25:15<4:32:28, 4.79s/it] 8%|▊ | 308/3723 [25:19<4:32:25, 4.79s/it] 8%|▊ | 309/3723 [25:24<4:32:08, 4.78s/it] 8%|▊ | 310/3723 [25:29<4:32:05, 4.78s/it] 8%|▊ | 310/3723 [25:29<4:32:05, 4.78s/it] 8%|▊ | 311/3723 [25:34<4:32:12, 4.79s/it] 8%|▊ | 312/3723 [25:38<4:31:31, 4.78s/it] 8%|▊ | 313/3723 [25:43<4:30:18, 4.76s/it] 8%|▊ | 314/3723 [25:48<4:29:56, 4.75s/it] 8%|▊ | 315/3723 [25:53<4:29:56, 4.75s/ 0: {'loss': 0.8084, 'grad_norm': 1.1916395878659545, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.05} 0: {'loss': 0.8225, 'grad_norm': 1.0729315873104106, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.05} 0: it] 8%|▊ | 316/3723 [25:57<4:29:54, 4.75s/it] 9%|▊ | 317/3723 [26:02<4:29:48, 4.75s/it] 9%|▊ | 318/3723 [26:07<4:29:53, 4.76s/it] 9%|▊ | 319/3723 [26:12<4:29:41, 4.75s/it] 9%|▊ | 320/3723 [26:16<4:29:36, 4.75s/it] 9%|▊ | 320/3723 [26:16<4:29:36, 4.75s/it] 9%|▊ | 321/3723 [26:21<4:29:52, 4.76s/it] 9%|▊ | 322/3723 [26:26<4:30:02, 4.76s/it] 9%|▊ | 323/3723 [26:31<4:29:54, 4.76s/it] 9%|▊ | 324/3723 [26:35<4:29:51, 4.76s/it] 9%|▊ | 325/3723 [26:40<4:29:37, 4.76s/it] 9%|▉ | 326/3723 [26:45<4:29:41, 4.76s/it] 9%|▉ | 327/3723 [26:50<4:29:40, 4.76s/it] 9%|▉ | 328/3723 [26:55<4:29:49, 4.77s/it] 9%|▉ | 329/3723 [26:59<4:29:55, 4.77s/it] 9%|▉ | 330/3723 [27:04<4:30:07, 4.78s/it] 9%|▉ | 330/3723 [27: 0: {'loss': 0.8042, 'grad_norm': 1.166415847808458, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.05} 0: 04<4:30:07, 4.78s/it] 9%|▉ | 331/3723 [27:09<4:30:53, 4.79s/it] 9%|▉ | 332/3723 [27:14<4:30:16, 4.78s/it] 9%|▉ | 333/3723 [27:18<4:30:35, 4.79s/it] 9%|▉ | 334/3723 [27:23<4:29:33, 4.77s/it] 9%|▉ | 335/3723 [27:28<4:28:37, 4.76s/it] 9%|▉ | 336/3723 [27:33<4:27:47, 4.74s/it] 9%|▉ | 337/3723 [27:37<4:26:57, 4.73s/it] 9%|▉ | 338/3723 [27:42<4:26:43, 4.73s/it] 9%|▉ | 339/3723 [27:47<4:26:20, 4.72s/it] 9%|▉ | 340/3723 [27:51<4:26:02, 4.72s/it] 9%|▉ | 340/3723 [27:52<4:26:02, 4.72s/it] 9%|▉ | 341/3723 [27:56<4:26:56, 4.74s/it] 9%|▉ | 342/3723 [28:01<4:28:07, 4.76s/it] 9%|▉ | 343/3723 [28:06<4:26:55, 4.74s/it] 9%|▉ | 344/3723 [28:10<4:26:26, 4.73s/it] 9%|▉ | 345/3723 [28:15<4:26:03, 4.73s/it] 9%|▉ | 346/3723 [28:20<4:25:23, 4.72s/it] 9%|▉ 0: {'loss': 0.8417, 'grad_norm': 1.1230958726720228, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.06} 0: {'loss': 0.8218, 'grad_norm': 1.0393042604230327, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.06} 0: | 347/3723 [28:25<4:25:35, 4.72s/it] 9%|▉ | 348/3723 [28:29<4:26:17, 4.73s/it] 9%|▉ | 349/3723 [28:34<4:26:49, 4.75s/it] 9%|▉ | 350/3723 [28:39<4:26:58, 4.75s/it] 9%|▉ | 350/3723 [28:39<4:26:58, 4.75s/it] 9%|▉ | 351/3723 [28:44<4:27:48, 4.77s/it] 9%|▉ | 352/3723 [28:49<4:27:59, 4.77s/it] 9%|▉ | 353/3723 [28:53<4:28:07, 4.77s/it] 10%|▉ | 354/3723 [28:58<4:27:38, 4.77s/it] 10%|▉ | 355/3723 [29:03<4:27:06, 4.76s/it] 10%|▉ | 356/3723 [29:08<4:26:57, 4.76s/it] 10%|▉ | 357/3723 [29:12<4:27:01, 4.76s/it] 10%|▉ | 358/3723 [29:17<4:27:12, 4.76s/it] 10%|▉ | 359/3723 [29:22<4:27:19, 4.77s/it] 10%|▉ | 360/3723 [29:27<4:27:21, 4.77s/it] 10%|▉ | 360/3723 [29:27<4:27:21, 4.77s/it] 10%|▉ | 361/3723 [29:31<4:27:22, 4.7 0: {'loss': 0.812, 'grad_norm': 1.108241387585411, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.06} 0: 7s/it] 10%|▉ | 362/3723 [29:36<4:27:19, 4.77s/it] 10%|▉ | 363/3723 [29:41<4:27:32, 4.78s/it] 10%|▉ | 364/3723 [29:46<4:26:49, 4.77s/it] 10%|▉ | 365/3723 [29:50<4:25:16, 4.74s/it] 10%|▉ | 366/3723 [29:55<4:24:56, 4.74s/it] 10%|▉ | 367/3723 [30:00<4:23:58, 4.72s/it] 10%|▉ | 368/3723 [30:04<4:23:26, 4.71s/it] 10%|▉ | 369/3723 [30:09<4:23:20, 4.71s/it] 10%|▉ | 370/3723 [30:14<4:22:39, 4.70s/it] 10%|▉ | 370/3723 [30:14<4:22:39, 4.70s/it] 10%|▉ | 371/3723 [30:19<4:22:25, 4.70s/it] 10%|▉ | 372/3723 [30:23<4:21:06, 4.68s/it] 10%|█ | 373/3723 [30:28<4:20:40, 4.67s/it] 10%|█ | 374/3723 [30:32<4:19:47, 4.65s/it] 10%|█ | 375/3723 [30:37<4:19:05, 4.64s/it] 10%|█ | 376/3723 [30:42<4:18:16, 4.63s/it] 10%|█ | 377/3723 [30:46<4:17:57, 4.63s/it] 10%|█ | 378/3723 0: {'loss': 0.7989, 'grad_norm': 1.2333585017548792, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.06} 0: {'loss': 0.8118, 'grad_norm': 1.0489734064065162, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.06} 0: [30:51<4:17:20, 4.62s/it] 10%|█ | 379/3723 [30:56<4:20:14, 4.67s/it] 10%|█ | 380/3723 [31:00<4:21:18, 4.69s/it] 10%|█ | 380/3723 [31:00<4:21:18, 4.69s/it] 10%|█ | 381/3723 [31:05<4:21:33, 4.70s/it] 10%|█ | 382/3723 [31:10<4:21:35, 4.70s/it] 10%|█ | 383/3723 [31:15<4:21:14, 4.69s/it] 10%|█ | 384/3723 [31:19<4:21:01, 4.69s/it] 10%|█ | 385/3723 [31:24<4:21:13, 4.70s/it] 10%|█ | 386/3723 [31:29<4:21:48, 4.71s/it] 10%|█ | 387/3723 [31:33<4:22:04, 4.71s/it] 10%|█ | 388/3723 [31:38<4:21:52, 4.71s/it] 10%|█ | 389/3723 [31:43<4:21:35, 4.71s/it] 10%|█ | 390/3723 [31:48<4:22:18, 4.72s/it] 10%|█ | 390/3723 [31:48<4:22:18, 4.72s/it] 11%|█ | 391/3723 [31:52<4:23:09, 4.74s/it] 11%|█ | 392/3723 [31:57<4:23:14, 4.74s/it] 11%|█ 0: {'loss': 0.8077, 'grad_norm': 1.1413843018522252, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.06} 0: | 393/3723 [32:02<4:22:17, 4.73s/it] 11%|█ | 394/3723 [32:06<4:21:57, 4.72s/it] 11%|█ | 395/3723 [32:11<4:21:29, 4.71s/it] 11%|█ | 396/3723 [32:16<4:20:26, 4.70s/it] 11%|█ | 397/3723 [32:20<4:20:08, 4.69s/it] 11%|█ | 398/3723 [32:25<4:20:03, 4.69s/it] 11%|█ | 399/3723 [32:30<4:20:00, 4.69s/it] 11%|█ | 400/3723 [32:35<4:20:06, 4.70s/it] 11%|█ | 400/3723 [32:35<4:20:06, 4.70s/it] 11%|█ | 401/3723 [32:39<4:20:24, 4.70s/it] 11%|█ | 402/3723 [32:44<4:20:21, 4.70s/it] 11%|█ | 403/3723 [32:49<4:19:54, 4.70s/it] 11%|█ | 404/3723 [32:53<4:19:25, 4.69s/it] 11%|█ | 405/3723 [32:58<4:19:04, 4.68s/it] 11%|█ | 406/3723 [33:03<4:19:01, 4.69s/it] 11%|█ | 407/3723 [33:07<4:19:04, 4.69s/it] 11%|█ | 408/3723 [33:12<4:18:52, 4.69s/it] 11%|█ | 409/3723 [33:17<4:19:13, 0: {'loss': 0.7988, 'grad_norm': 1.2615212041579735, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.07} 0: {'loss': 0.8074, 'grad_norm': 1.1822901828795997, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.07} 0: 4.69s/it] 11%|█ | 410/3723 [33:22<4:19:36, 4.70s/it] 11%|█ | 410/3723 [33:22<4:19:36, 4.70s/it] 11%|█ | 411/3723 [33:26<4:19:54, 4.71s/it] 11%|█ | 412/3723 [33:31<4:20:14, 4.72s/it] 11%|█ | 413/3723 [33:36<4:21:34, 4.74s/it] 11%|█ | 414/3723 [33:41<4:22:07, 4.75s/it] 11%|█ | 415/3723 [33:45<4:22:00, 4.75s/it] 11%|█ | 416/3723 [33:50<4:21:39, 4.75s/it] 11%|█ | 417/3723 [33:55<4:21:22, 4.74s/it] 11%|█ | 418/3723 [34:00<4:21:26, 4.75s/it] 11%|█▏ | 419/3723 [34:04<4:20:56, 4.74s/it] 11%|█▏ | 420/3723 [34:09<4:20:34, 4.73s/it] 11%|█▏ | 420/3723 [34:09<4:20:34, 4.73s/it] 11%|█▏ | 421/3723 [34:14<4:20:31, 4.73s/it] 11%|█▏ | 422/3723 [34:18<4:20:00, 4.73s/it] 11%|█▏ | 423/3723 [34:23<4:19:40, 4.72s/it] 11%|█▏ 0: {'loss': 0.794, 'grad_norm': 1.048865653902951, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.07} 0: | 424/3723 [34:28<4:19:39, 4.72s/it] 11%|█▏ | 425/3723 [34:33<4:19:45, 4.73s/it] 11%|█▏ | 426/3723 [34:37<4:20:24, 4.74s/it] 11%|█▏ | 427/3723 [34:42<4:25:11, 4.83s/it] 11%|█▏ | 428/3723 [34:47<4:24:24, 4.81s/it] 12%|█▏ | 429/3723 [34:52<4:22:59, 4.79s/it] 12%|█▏ | 430/3723 [34:57<4:22:01, 4.77s/it] 12%|█▏ | 430/3723 [34:57<4:22:01, 4.77s/it] 12%|█▏ | 431/3723 [35:01<4:21:30, 4.77s/it] 12%|█▏ | 432/3723 [35:06<4:22:00, 4.78s/it] 12%|█▏ | 433/3723 [35:11<4:22:20, 4.78s/it] 12%|█▏ | 434/3723 [35:16<4:22:16, 4.78s/it] 12%|█▏ | 435/3723 [35:21<4:22:20, 4.79s/it] 12%|█▏ | 436/3723 [35:25<4:22:31, 4.79s/it] 12%|█▏ | 437/3723 [35:30<4:22:28, 4.79s/it] 12%|█▏ | 438/3723 [35:35<4:22:11, 4.79s/it] 12%|█▏ | 439/3723 [35:40<4:21:52, 4.78s/it] 12%|█▏ 0: {'loss': 0.7804, 'grad_norm': 1.090555965238124, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.07} 0: {'loss': 0.8217, 'grad_norm': 1.0471065351469897, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.07} 0: | 440/3723 [35:45<4:22:04, 4.79s/it] 12%|█▏ | 440/3723 [35:45<4:22:04, 4.79s/it] 12%|█▏ | 441/3723 [35:49<4:22:25, 4.80s/it] 12%|█▏ | 442/3723 [35:54<4:22:18, 4.80s/it] 12%|█▏ | 443/3723 [35:59<4:21:21, 4.78s/it] 12%|█▏ | 444/3723 [36:04<4:20:27, 4.77s/it] 12%|█▏ | 445/3723 [36:08<4:19:04, 4.74s/it] 12%|█▏ | 446/3723 [36:13<4:18:09, 4.73s/it] 12%|█▏ | 447/3723 [36:18<4:18:06, 4.73s/it] 12%|█▏ | 448/3723 [36:22<4:18:03, 4.73s/it] 12%|█▏ | 449/3723 [36:27<4:18:28, 4.74s/it] 12%|█▏ | 450/3723 [36:32<4:18:37, 4.74s/it] 12%|█▏ | 450/3723 [36:32<4:18:37, 4.74s/it] 12%|█▏ | 451/3723 [36:37<4:18:52, 4.75s/it] 12%|█▏ | 452/3723 [36:41<4:18:12, 4.74s/it] 12%|█▏ | 453/3723 [36:46<4:18:28, 4.74s/it] 12%|█▏ 0: {'loss': 0.781, 'grad_norm': 1.0140528827949034, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.07} 0: | 454/3723 [36:51<4:18:32, 4.75s/it] 12%|█▏ | 455/3723 [36:56<4:18:46, 4.75s/it] 12%|█▏ | 456/3723 [37:00<4:18:58, 4.76s/it] 12%|█▏ | 457/3723 [37:05<4:22:18, 4.82s/it] 12%|█▏ | 458/3723 [37:10<4:20:57, 4.80s/it] 12%|█▏ | 459/3723 [37:15<4:19:37, 4.77s/it] 12%|█▏ | 460/3723 [37:20<4:18:42, 4.76s/it] 12%|█▏ | 460/3723 [37:20<4:18:42, 4.76s/it] 12%|█▏ | 461/3723 [37:24<4:18:38, 4.76s/it] 12%|█▏ | 462/3723 [37:29<4:18:24, 4.75s/it] 12%|█▏ | 463/3723 [37:34<4:18:34, 4.76s/it] 12%|█▏ | 464/3723 [37:39<4:18:48, 4.76s/it] 12%|█▏ | 465/3723 [37:43<4:18:15, 4.76s/it] 13%|█▎ | 466/3723 [37:48<4:17:15, 4.74s/it] 13%|█▎ | 467/3723 [37:53<4:17:30, 4.75s/it] 13%|█▎ | 468/3723 [37:58<4:21:43, 4.82s/it] 13%|█▎ | 469/3723 [38:03<4:20:36, 4.81s/it] 13%|█▎ 0: {'loss': 0.8091, 'grad_norm': 1.1798843883897028, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.08} 0: {'loss': 0.7941, 'grad_norm': 1.1377760101457313, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.08} 0: | 470/3723 [38:07<4:19:53, 4.79s/it] 13%|█▎ | 470/3723 [38:07<4:19:53, 4.79s/it] 13%|█▎ | 471/3723 [38:12<4:19:40, 4.79s/it] 13%|█▎ | 472/3723 [38:17<4:19:08, 4.78s/it] 13%|█▎ | 473/3723 [38:22<4:18:58, 4.78s/it] 13%|█▎ | 474/3723 [38:26<4:18:39, 4.78s/it] 13%|█▎ | 475/3723 [38:31<4:18:15, 4.77s/it] 13%|█▎ | 476/3723 [38:36<4:17:48, 4.76s/it] 13%|█▎ | 477/3723 [38:41<4:17:11, 4.75s/it] 13%|█▎ | 478/3723 [38:45<4:16:51, 4.75s/it] 13%|█▎ | 479/3723 [38:50<4:16:09, 4.74s/it] 13%|█▎ | 480/3723 [38:55<4:15:22, 4.72s/it] 13%|█▎ | 480/3723 [38:55<4:15:22, 4.72s/it] 13%|█▎ | 481/3723 [39:00<4:15:38, 4.73s/it] 13%|█▎ | 482/3723 [39:04<4:15:43, 4.73s/it] 13%|█▎ | 483/3723 [39:09<4:15:25, 4.73s/it] 13%|█▎ | 48 0: {'loss': 0.7956, 'grad_norm': 1.1374094781625774, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.08} 0: 4/3723 [39:14<4:15:08, 4.73s/it] 13%|█▎ | 485/3723 [39:18<4:14:32, 4.72s/it] 13%|█▎ | 486/3723 [39:23<4:14:13, 4.71s/it] 13%|█▎ | 487/3723 [39:28<4:14:05, 4.71s/it] 13%|█▎ | 488/3723 [39:33<4:14:03, 4.71s/it] 13%|█▎ | 489/3723 [39:37<4:14:12, 4.72s/it] 13%|█▎ | 490/3723 [39:42<4:15:56, 4.75s/it] 13%|█▎ | 490/3723 [39:42<4:15:56, 4.75s/it] 13%|█▎ | 491/3723 [39:47<4:23:01, 4.88s/it] 13%|█▎ | 492/3723 [39:52<4:22:51, 4.88s/it] 13%|█▎ | 493/3723 [39:57<4:22:34, 4.88s/it] 13%|█▎ | 494/3723 [40:02<4:21:48, 4.86s/it] 13%|█▎ | 495/3723 [40:07<4:21:11, 4.85s/it] 13%|█▎ | 496/3723 [40:12<4:20:15, 4.84s/it] 13%|█▎ | 497/3723 [40:16<4:19:45, 4.83s/it] 13%|█▎ | 498/3723 [40:21<4:19:11, 4.82s/it] 13%|█▎ | 499/3723 [40:26<4:18:38, 4.81s/it] 13%|█▎ | 500 0: {'loss': 0.7798, 'grad_norm': 1.096875394723807, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.08} 0: {'loss': 0.7998, 'grad_norm': 1.1847437767762612, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.08} 0: /3723 [40:31<4:18:32, 4.81s/it] 13%|█▎ | 500/3723 [40:31<4:18:32, 4.81s/it] 13%|█▎ | 501/3723 [40:36<4:18:17, 4.81s/it] 13%|█▎ | 502/3723 [40:40<4:17:18, 4.79s/it] 14%|█▎ | 503/3723 [40:45<4:16:51, 4.79s/it] 14%|█▎ | 504/3723 [40:50<4:16:32, 4.78s/it] 14%|█▎ | 505/3723 [40:55<4:16:11, 4.78s/it] 14%|█▎ | 506/3723 [40:59<4:15:13, 4.76s/it] 14%|█▎ | 507/3723 [41:04<4:14:04, 4.74s/it] 14%|█▎ | 508/3723 [41:09<4:13:12, 4.73s/it] 14%|█▎ | 509/3723 [41:13<4:12:02, 4.71s/it] 14%|█▎ | 510/3723 [41:18<4:10:55, 4.69s/it] 14%|█▎ | 510/3723 [41:18<4:10:55, 4.69s/it] 14%|█▎ | 511/3723 [41:23<4:10:47, 4.68s/it] 14%|█▍ | 512/3723 [41:27<4:10:16, 4.68s/it] 14%|█▍ | 513/3723 [41:32<4:09:46, 4.67s/it] 14%|█▍ | 514/372 0: {'loss': 0.7932, 'grad_norm': 1.0106237517934777, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.08} 0: 3 [41:37<4:09:52, 4.67s/it] 14%|█▍ | 515/3723 [41:41<4:09:33, 4.67s/it] 14%|█▍ | 516/3723 [41:46<4:09:32, 4.67s/it] 14%|█▍ | 517/3723 [41:51<4:09:26, 4.67s/it] 14%|█▍ | 518/3723 [41:55<4:09:11, 4.67s/it] 14%|█▍ | 519/3723 [42:00<4:08:36, 4.66s/it] 14%|█▍ | 520/3723 [42:05<4:08:02, 4.65s/it] 14%|█▍ | 520/3723 [42:05<4:08:02, 4.65s/it] 14%|█▍ | 521/3723 [42:09<4:08:20, 4.65s/it] 14%|█▍ | 522/3723 [42:14<4:08:35, 4.66s/it] 14%|█▍ | 523/3723 [42:19<4:08:34, 4.66s/it] 14%|█▍ | 524/3723 [42:23<4:08:25, 4.66s/it] 14%|█▍ | 525/3723 [42:28<4:08:27, 4.66s/it] 14%|█▍ | 526/3723 [42:37<5:23:29, 6.07s/it] 14%|█▍ | 527/3723 [42:42<5:05:28, 5.73s/it] 14%|█▍ | 528/3723 [42:47<4:49:15, 5.43s/it] 14%|█▍ | 529/3723 [42:52<4:37:42, 5.22s/it] 14%|█▍ | 530/3723 0: {'loss': 0.7834, 'grad_norm': 1.0335126306971294, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.09} 0: {'loss': 0.8013, 'grad_norm': 1.0374042593806105, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.09} 0: [42:56<4:29:33, 5.07s/it] 14%|█▍ | 530/3723 [42:56<4:29:33, 5.07s/it] 14%|█▍ | 531/3723 [43:01<4:29:01, 5.06s/it] 14%|█▍ | 532/3723 [43:07<4:28:57, 5.06s/it] 14%|█▍ | 533/3723 [43:12<4:28:44, 5.05s/it] 14%|█▍ | 534/3723 [43:17<4:29:13, 5.07s/it] 14%|█▍ | 535/3723 [43:22<4:28:24, 5.05s/it] 14%|█▍ | 536/3723 [43:27<4:24:31, 4.98s/it] 14%|█▍ | 537/3723 [43:31<4:21:37, 4.93s/it] 14%|█▍ | 538/3723 [43:36<4:19:27, 4.89s/it] 14%|█▍ | 539/3723 [43:41<4:17:57, 4.86s/it] 15%|█▍ | 540/3723 [43:46<4:17:00, 4.84s/it] 15%|█▍ | 540/3723 [43:46<4:17:00, 4.84s/it] 15%|█▍ | 541/3723 [43:51<4:16:35, 4.84s/it] 15%|█▍ | 542/3723 [43:55<4:16:12, 4.83s/it] 15%|█▍ | 543/3723 [44:00<4:16:21, 4.84s/it] 15%|█▍ | 544/3723 [44 0: {'loss': 0.777, 'grad_norm': 1.1729554901993764, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.09} 0: :05<4:16:24, 4.84s/it] 15%|█▍ | 545/3723 [44:10<4:16:17, 4.84s/it] 15%|█▍ | 546/3723 [44:15<4:14:29, 4.81s/it] 15%|█▍ | 547/3723 [44:19<4:13:47, 4.79s/it] 15%|█▍ | 548/3723 [44:24<4:13:08, 4.78s/it] 15%|█▍ | 549/3723 [44:29<4:12:29, 4.77s/it] 15%|█▍ | 550/3723 [44:34<4:12:04, 4.77s/it] 15%|█▍ | 550/3723 [44:34<4:12:04, 4.77s/it] 15%|█▍ | 551/3723 [44:38<4:12:07, 4.77s/it] 15%|█▍ | 552/3723 [44:43<4:11:59, 4.77s/it] 15%|█▍ | 553/3723 [44:48<4:11:45, 4.77s/it] 15%|█▍ | 554/3723 [44:53<4:11:29, 4.76s/it] 15%|█▍ | 555/3723 [44:57<4:11:46, 4.77s/it] 15%|█▍ | 556/3723 [45:02<4:11:12, 4.76s/it] 15%|█▍ | 557/3723 [45:07<4:10:55, 4.76s/it] 15%|█▍ | 558/3723 [45:12<4:10:24, 4.75s/it] 15%|█▌ | 559/3723 [45:16<4:09:45, 4.74s/it] 15%|█▌ | 560/3723 [45: 0: {'loss': 0.7765, 'grad_norm': 0.9733793562288551, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.09} 0: {'loss': 0.7719, 'grad_norm': 1.1350451114666265, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.09} 0: 21<4:09:46, 4.74s/it] 15%|█▌ | 560/3723 [45:21<4:09:46, 4.74s/it] 15%|█▌ | 561/3723 [45:26<4:10:04, 4.75s/it] 15%|█▌ | 562/3723 [45:31<4:09:59, 4.75s/it] 15%|█▌ | 563/3723 [45:35<4:09:28, 4.74s/it] 15%|█▌ | 564/3723 [45:40<4:08:53, 4.73s/it] 15%|█▌ | 565/3723 [45:45<4:07:33, 4.70s/it] 15%|█▌ | 566/3723 [45:49<4:07:18, 4.70s/it] 15%|█▌ | 567/3723 [45:54<4:07:05, 4.70s/it] 15%|█▌ | 568/3723 [45:59<4:07:07, 4.70s/it] 15%|█▌ | 569/3723 [46:04<4:07:09, 4.70s/it] 15%|█▌ | 570/3723 [46:08<4:07:01, 4.70s/it] 15%|█▌ | 570/3723 [46:08<4:07:01, 4.70s/it] 15%|█▌ | 571/3723 [46:13<4:07:07, 4.70s/it] 15%|█▌ | 572/3723 [46:18<4:06:42, 4.70s/it] 15%|█▌ | 573/3723 [46:22<4:06:46, 4.70s/it] 15%|█▌ | 574/3723 [46:27<4 0: {'loss': 0.768, 'grad_norm': 1.005701280779638, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.09} 0: :07:02, 4.71s/it] 15%|█▌ | 575/3723 [46:32<4:07:04, 4.71s/it] 15%|█▌ | 576/3723 [46:36<4:07:02, 4.71s/it] 15%|█▌ | 577/3723 [46:41<4:07:02, 4.71s/it] 16%|█▌ | 578/3723 [46:46<4:06:39, 4.71s/it] 16%|█▌ | 579/3723 [46:51<4:06:04, 4.70s/it] 16%|█▌ | 580/3723 [46:55<4:05:43, 4.69s/it] 16%|█▌ | 580/3723 [46:55<4:05:43, 4.69s/it] 16%|█▌ | 581/3723 [47:00<4:05:38, 4.69s/it] 16%|█▌ | 582/3723 [47:05<4:06:00, 4.70s/it] 16%|█▌ | 583/3723 [47:09<4:06:10, 4.70s/it] 16%|█▌ | 584/3723 [47:14<4:05:56, 4.70s/it] 16%|█▌ | 585/3723 [47:19<4:06:49, 4.72s/it] 16%|█▌ | 586/3723 [47:24<4:07:40, 4.74s/it] 16%|█▌ | 587/3723 [47:28<4:08:30, 4.75s/it] 16%|█▌ | 588/3723 [47:33<4:08:57, 4.76s/it] 16%|█▌ | 589/3723 [47:38<4:09:45, 4.78s/it] 16%|█▌ | 590/3723 [47:43<4: 0: {'loss': 0.7776, 'grad_norm': 1.0939413790217516, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.1} 0: {'loss': 0.7853, 'grad_norm': 0.962740455300278, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.1} 0: 08:47, 4.76s/it] 16%|█▌ | 590/3723 [47:43<4:08:47, 4.76s/it] 16%|█▌ | 591/3723 [47:47<4:07:45, 4.75s/it] 16%|█▌ | 592/3723 [47:52<4:06:36, 4.73s/it] 16%|█▌ | 593/3723 [47:57<4:05:57, 4.71s/it] 16%|█▌ | 594/3723 [48:01<4:05:21, 4.70s/it] 16%|█▌ | 595/3723 [48:06<4:04:45, 4.69s/it] 16%|█▌ | 596/3723 [48:11<4:06:38, 4.73s/it] 16%|█▌ | 597/3723 [48:16<4:08:04, 4.76s/it] 16%|█▌ | 598/3723 [48:21<4:08:53, 4.78s/it] 16%|█▌ | 599/3723 [48:25<4:09:30, 4.79s/it] 16%|█▌ | 600/3723 [48:30<4:08:57, 4.78s/it] 16%|█▌ | 600/3723 [48:30<4:08:57, 4.78s/it] 16%|█▌ | 601/3723 [48:35<4:08:10, 4.77s/it] 16%|█▌ | 602/3723 [48:40<4:08:47, 4.78s/it] 16%|█▌ | 603/3723 [48:45<4:09:15, 4.79s/it] 16%|█▌ | 604/3723 [48:49<4:09:2 0: {'loss': 0.7636, 'grad_norm': 0.9574428624812077, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.1} 0: 1, 4.80s/it] 16%|█▋ | 605/3723 [48:54<4:09:40, 4.80s/it] 16%|█▋ | 606/3723 [48:59<4:10:04, 4.81s/it] 16%|█▋ | 607/3723 [49:04<4:10:15, 4.82s/it] 16%|█▋ | 608/3723 [49:09<4:10:21, 4.82s/it] 16%|█▋ | 609/3723 [49:13<4:08:41, 4.79s/it] 16%|█▋ | 610/3723 [49:18<4:08:13, 4.78s/it] 16%|█▋ | 610/3723 [49:18<4:08:13, 4.78s/it] 16%|█▋ | 611/3723 [49:23<4:07:45, 4.78s/it] 16%|█▋ | 612/3723 [49:28<4:07:13, 4.77s/it] 16%|█▋ | 613/3723 [49:32<4:06:44, 4.76s/it] 16%|█▋ | 614/3723 [49:37<4:07:00, 4.77s/it] 17%|█▋ | 615/3723 [49:42<4:07:25, 4.78s/it] 17%|█▋ | 616/3723 [49:47<4:08:17, 4.79s/it] 17%|█▋ | 617/3723 [49:52<4:08:36, 4.80s/it] 17%|█▋ | 618/3723 [49:56<4:08:49, 4.81s/it] 17%|█▋ | 619/3723 [50:01<4:08:07, 4.80s/it] 17%|█▋ | 620/3723 [50:06<4:07:15 0: {'loss': 0.7669, 'grad_norm': 0.9017517288873588, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.1} 0: {'loss': 0.7762, 'grad_norm': 1.1342015233513156, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.1} 0: , 4.78s/it] 17%|█▋ | 620/3723 [50:06<4:07:15, 4.78s/it] 17%|█▋ | 621/3723 [50:11<4:07:14, 4.78s/it] 17%|█▋ | 622/3723 [50:16<4:06:34, 4.77s/it] 17%|█▋ | 623/3723 [50:20<4:05:51, 4.76s/it] 17%|█▋ | 624/3723 [50:25<4:05:02, 4.74s/it] 17%|█▋ | 625/3723 [50:30<4:04:13, 4.73s/it] 17%|█▋ | 626/3723 [50:34<4:04:26, 4.74s/it] 17%|█▋ | 627/3723 [50:39<4:04:06, 4.73s/it] 17%|█▋ | 628/3723 [50:44<4:03:25, 4.72s/it] 17%|█▋ | 629/3723 [50:48<4:02:43, 4.71s/it] 17%|█▋ | 630/3723 [50:53<4:01:58, 4.69s/it] 17%|█▋ | 630/3723 [50:53<4:01:58, 4.69s/it] 17%|█▋ | 631/3723 [50:58<4:01:43, 4.69s/it] 17%|█▋ | 632/3723 [51:03<4:01:45, 4.69s/it] 17%|█▋ | 633/3723 [51:07<4:01:28, 4.69s/it] 17%|█▋ | 634/3723 [51:12<4:01:16, 4 0: {'loss': 0.7893, 'grad_norm': 1.1901937614064781, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.1} 0: .69s/it] 17%|█▋ | 635/3723 [51:17<4:01:02, 4.68s/it] 17%|█▋ | 636/3723 [51:21<4:00:48, 4.68s/it] 17%|█▋ | 637/3723 [51:26<4:00:50, 4.68s/it] 17%|█▋ | 638/3723 [51:31<4:00:43, 4.68s/it] 17%|█▋ | 639/3723 [51:35<4:00:36, 4.68s/it] 17%|█▋ | 640/3723 [51:40<4:00:38, 4.68s/it] 17%|█▋ | 640/3723 [51:40<4:00:38, 4.68s/it] 17%|█▋ | 641/3723 [51:45<4:01:02, 4.69s/it] 17%|█▋ | 642/3723 [51:49<4:00:57, 4.69s/it] 17%|█▋ | 643/3723 [51:54<4:00:37, 4.69s/it] 17%|█▋ | 644/3723 [51:59<4:00:38, 4.69s/it] 17%|█▋ | 645/3723 [52:03<4:00:26, 4.69s/it] 17%|█▋ | 646/3723 [52:08<4:00:26, 4.69s/it] 17%|█▋ | 647/3723 [52:13<3:59:59, 4.68s/it] 17%|█▋ | 648/3723 [52:18<4:00:28, 4.69s/it] 17%|█▋ | 649/3723 [52:22<4:00:07, 4.69s/it] 17%|█▋ | 650/3723 [52:27<3:59:54, 4. 0: {'loss': 0.7713, 'grad_norm': 1.148340495972678, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.1} 0: {'loss': 0.7774, 'grad_norm': 1.0746543503624761, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.11} 0: 68s/it] 17%|█▋ | 650/3723 [52:27<3:59:54, 4.68s/it] 17%|█▋ | 651/3723 [52:32<4:00:17, 4.69s/it] 18%|█▊ | 652/3723 [52:36<4:00:41, 4.70s/it] 18%|█▊ | 653/3723 [52:41<4:00:42, 4.70s/it] 18%|█▊ | 654/3723 [52:46<4:00:29, 4.70s/it] 18%|█▊ | 655/3723 [52:50<4:00:10, 4.70s/it] 18%|█▊ | 656/3723 [52:55<3:59:52, 4.69s/it] 18%|█▊ | 657/3723 [53:00<4:00:01, 4.70s/it] 18%|█▊ | 658/3723 [53:04<3:59:48, 4.69s/it] 18%|█▊ | 659/3723 [53:09<3:59:43, 4.69s/it] 18%|█▊ | 660/3723 [53:14<3:58:59, 4.68s/it] 18%|█▊ | 660/3723 [53:14<3:58:59, 4.68s/it] 18%|█▊ | 661/3723 [53:18<3:58:26, 4.67s/it] 18%|█▊ | 662/3723 [53:23<3:57:23, 4.65s/it] 18%|█▊ | 663/3723 [53:28<3:57:37, 4.66s/it] 18%|█▊ | 664/3723 [53:32<3:57:51, 4.67s/ 0: {'loss': 0.7633, 'grad_norm': 1.1585517987352456, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.11} 0: it] 18%|█▊ | 665/3723 [53:37<3:57:54, 4.67s/it] 18%|█▊ | 666/3723 [53:42<3:58:31, 4.68s/it] 18%|█▊ | 667/3723 [53:47<3:59:50, 4.71s/it] 18%|█▊ | 668/3723 [53:51<4:00:38, 4.73s/it] 18%|█▊ | 669/3723 [53:56<4:00:52, 4.73s/it] 18%|█▊ | 670/3723 [54:01<4:01:11, 4.74s/it] 18%|█▊ | 670/3723 [54:01<4:01:11, 4.74s/it] 18%|█▊ | 671/3723 [54:06<4:01:37, 4.75s/it] 18%|█▊ | 672/3723 [54:10<4:00:18, 4.73s/it] 18%|█▊ | 673/3723 [54:15<3:58:53, 4.70s/it] 18%|█▊ | 674/3723 [54:20<3:58:09, 4.69s/it] 18%|█▊ | 675/3723 [54:24<3:57:04, 4.67s/it] 18%|█▊ | 676/3723 [54:29<3:57:26, 4.68s/it] 18%|█▊ | 677/3723 [54:34<3:57:42, 4.68s/it] 18%|█▊ | 678/3723 [54:38<3:57:32, 4.68s/it] 18%|█▊ | 679/3723 [54:43<3:57:18, 4.68s/it] 18%|█▊ | 680/3723 [54:48<3:57:18, 4.68s/i 0: {'loss': 0.777, 'grad_norm': 0.9942085786663855, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.11} 0: {'loss': 0.7574, 'grad_norm': 1.096760820189726, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.11} 0: t] 18%|█▊ | 680/3723 [54:48<3:57:18, 4.68s/it] 18%|█▊ | 681/3723 [54:52<3:57:44, 4.69s/it] 18%|█▊ | 682/3723 [54:57<3:57:34, 4.69s/it] 18%|█▊ | 683/3723 [55:02<3:57:19, 4.68s/it] 18%|█▊ | 684/3723 [55:06<3:57:17, 4.68s/it] 18%|█▊ | 685/3723 [55:11<3:57:02, 4.68s/it] 18%|█▊ | 686/3723 [55:16<3:57:07, 4.68s/it] 18%|█▊ | 687/3723 [55:20<3:56:41, 4.68s/it] 18%|█▊ | 688/3723 [55:25<3:56:24, 4.67s/it] 19%|█▊ | 689/3723 [55:30<3:55:28, 4.66s/it] 19%|█▊ | 690/3723 [55:34<3:55:21, 4.66s/it] 19%|█▊ | 690/3723 [55:34<3:55:21, 4.66s/it] 19%|█▊ | 691/3723 [55:39<3:54:43, 4.64s/it] 19%|█▊ | 692/3723 [55:44<3:53:52, 4.63s/it] 19%|█▊ | 693/3723 [55:48<3:53:35, 4.63s/it] 19%|█▊ | 694/3723 [55:53<3:54:01, 4.64s/it] 0: {'loss': 0.7691, 'grad_norm': 0.9668031126240589, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.11} 0: 19%|█▊ | 695/3723 [55:58<3:54:24, 4.64s/it] 19%|█▊ | 696/3723 [56:02<3:54:52, 4.66s/it] 19%|█▊ | 697/3723 [56:07<3:54:57, 4.66s/it] 19%|█▊ | 698/3723 [56:12<3:55:06, 4.66s/it] 19%|█▉ | 699/3723 [56:16<3:55:00, 4.66s/it] 19%|█▉ | 700/3723 [56:21<3:55:08, 4.67s/it] 19%|█▉ | 700/3723 [56:21<3:55:08, 4.67s/it] 19%|█▉ | 701/3723 [56:26<3:55:32, 4.68s/it] 19%|█▉ | 702/3723 [56:30<3:55:32, 4.68s/it] 19%|█▉ | 703/3723 [56:35<3:55:21, 4.68s/it] 19%|█▉ | 704/3723 [56:40<3:55:27, 4.68s/it] 19%|█▉ | 705/3723 [56:44<3:55:13, 4.68s/it] 19%|█▉ | 706/3723 [56:49<3:55:12, 4.68s/it] 19%|█▉ | 707/3723 [56:54<3:55:02, 4.68s/it] 19%|█▉ | 708/3723 [56:58<3:55:15, 4.68s/it] 19%|█▉ | 709/3723 [57:03<3:55:21, 4.69s/it] 19%|█▉ | 710/3723 [57:08<3:55:40, 4.69s/it] 0: {'loss': 0.7755, 'grad_norm': 1.2433689633987164, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.11} 0: {'loss': 0.7509, 'grad_norm': 1.2828489758601593, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.12} 0: 19%|█▉ | 710/3723 [57:08<3:55:40, 4.69s/it] 19%|█▉ | 711/3723 [57:12<3:56:15, 4.71s/it] 19%|█▉ | 712/3723 [57:17<3:56:02, 4.70s/it] 19%|█▉ | 713/3723 [57:22<3:55:26, 4.69s/it] 19%|█▉ | 714/3723 [57:26<3:54:41, 4.68s/it] 19%|█▉ | 715/3723 [57:31<3:53:56, 4.67s/it] 19%|█▉ | 716/3723 [57:36<3:53:50, 4.67s/it] 19%|█▉ | 717/3723 [57:40<3:53:31, 4.66s/it] 19%|█▉ | 718/3723 [57:45<3:53:25, 4.66s/it] 19%|█▉ | 719/3723 [57:50<3:53:22, 4.66s/it] 19%|█▉ | 720/3723 [57:55<3:55:08, 4.70s/it] 19%|█▉ | 720/3723 [57:55<3:55:08, 4.70s/it] 19%|█▉ | 721/3723 [57:59<3:56:50, 4.73s/it] 19%|█▉ | 722/3723 [58:04<3:56:22, 4.73s/it] 19%|█▉ | 723/3723 [58:09<3:55:18, 4.71s/it] 19%|█▉ | 724/3723 [58:13<3:55:14, 4.71s/it] 19%| 0: {'loss': 0.7564, 'grad_norm': 1.0263873324203807, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.12} 0: ▉ | 725/3723 [58:18<3:55:05, 4.70s/it] 20%|█▉ | 726/3723 [58:23<3:55:02, 4.71s/it] 20%|█▉ | 727/3723 [58:28<3:54:40, 4.70s/it] 20%|█▉ | 728/3723 [58:32<3:56:18, 4.73s/it] 20%|█▉ | 729/3723 [58:37<3:57:37, 4.76s/it] 20%|█▉ | 730/3723 [58:42<3:58:25, 4.78s/it] 20%|█▉ | 730/3723 [58:42<3:58:25, 4.78s/it] 20%|█▉ | 731/3723 [58:47<3:59:23, 4.80s/it] 20%|█▉ | 732/3723 [58:52<4:00:17, 4.82s/it] 20%|█▉ | 733/3723 [58:57<4:03:08, 4.88s/it] 20%|█▉ | 734/3723 [59:02<4:02:01, 4.86s/it] 20%|█▉ | 735/3723 [59:06<4:00:53, 4.84s/it] 20%|█▉ | 736/3723 [59:11<3:59:56, 4.82s/it] 20%|█▉ | 737/3723 [59:16<3:59:20, 4.81s/it] 20%|█▉ | 738/3723 [59:21<3:58:55, 4.80s/it] 20%|█▉ | 739/3723 [59:25<3:58:32, 4.80s/it] 20%|█▉ | 740/3723 [59:30<3:56:17, 4.75s/it] 0: {'loss': 0.78, 'grad_norm': 1.0190251681966236, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.12} 0: {'loss': 0.7689, 'grad_norm': 1.0857872202509946, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.12} 0: 20%|█▉ | 740/3723 [59:30<3:56:17, 4.75s/it] 20%|█▉ | 741/3723 [59:35<3:54:04, 4.71s/it] 20%|█▉ | 742/3723 [59:39<3:52:18, 4.68s/it] 20%|█▉ | 743/3723 [59:44<3:51:01, 4.65s/it] 20%|█▉ | 744/3723 [59:48<3:49:43, 4.63s/it] 20%|██ | 745/3723 [59:53<3:49:10, 4.62s/it] 20%|██ | 746/3723 [59:58<3:48:43, 4.61s/it] 20%|██ | 747/3723 [1:00:02<3:49:09, 4.62s/it] 20%|██ | 748/3723 [1:00:07<3:48:59, 4.62s/it] 20%|██ | 749/3723 [1:00:12<3:48:43, 4.61s/it] 20%|██ | 750/3723 [1:00:16<3:48:00, 4.60s/it] 20%|██ | 750/3723 [1:00:16<3:48:00, 4.60s/it] 20%|██ | 751/3723 [1:00:21<3:48:04, 4.60s/it] 20%|██ | 752/3723 [1:00:25<3:48:05, 4.61s/it] 20%|██ | 753/3723 [1:00:30<3:48:34, 4.62s/it] 20%|██ | 754/3723 [1:00:35<3:48:42, 4 0: {'loss': 0.7519, 'grad_norm': 1.067824011245136, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.12} 0: .62s/it] 20%|██ | 755/3723 [1:00:39<3:48:58, 4.63s/it] 20%|██ | 756/3723 [1:00:44<3:49:23, 4.64s/it] 20%|██ | 757/3723 [1:00:49<3:49:26, 4.64s/it] 20%|██ | 758/3723 [1:00:53<3:48:57, 4.63s/it] 20%|██ | 759/3723 [1:00:58<3:48:47, 4.63s/it] 20%|██ | 760/3723 [1:01:02<3:48:30, 4.63s/it] 20%|██ | 760/3723 [1:01:02<3:48:30, 4.63s/it] 20%|██ | 761/3723 [1:01:07<3:48:42, 4.63s/it] 20%|██ | 762/3723 [1:01:12<3:48:29, 4.63s/it] 20%|██ | 763/3723 [1:01:16<3:48:07, 4.62s/it] 21%|██ | 764/3723 [1:01:21<3:48:09, 4.63s/it] 21%|██ | 765/3723 [1:01:26<3:47:38, 4.62s/it] 21%|██ | 766/3723 [1:01:30<3:48:15, 4.63s/it] 21%|██ | 767/3723 [1:01:35<3:48:40, 4.64s/it] 21%|██ | 768/3723 [1:01:40<3:49:00, 4.65s/it] 21%|██ | 769/3723 [1:01:44<3:48:34, 4.64s/it] 21%|██ 0: {'loss': 0.7532, 'grad_norm': 0.96688484395279, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.12} 0: {'loss': 0.7403, 'grad_norm': 0.9067710638911621, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.13} 0: | 770/3723 [1:01:49<3:48:06, 4.63s/it] 21%|██ | 770/3723 [1:01:49<3:48:06, 4.63s/it] 21%|██ | 771/3723 [1:01:53<3:48:45, 4.65s/it] 21%|██ | 772/3723 [1:01:58<3:51:33, 4.71s/it] 21%|██ | 773/3723 [1:02:03<3:52:56, 4.74s/it] 21%|██ | 774/3723 [1:02:08<3:51:33, 4.71s/it] 21%|██ | 775/3723 [1:02:12<3:50:33, 4.69s/it] 21%|██ | 776/3723 [1:02:17<3:49:55, 4.68s/it] 21%|██ | 777/3723 [1:02:22<3:50:14, 4.69s/it] 21%|██ | 778/3723 [1:02:26<3:49:35, 4.68s/it] 21%|██ | 779/3723 [1:02:31<3:48:58, 4.67s/it] 21%|██ | 780/3723 [1:02:36<3:48:45, 4.66s/it] 21%|██ | 780/3723 [1:02:36<3:48:45, 4.66s/it] 21%|██ | 781/3723 [1:02:40<3:48:27, 4.66s/it] 21%|██ | 782/3723 [1:02:45<3:48:12, 4.66s/it] 21%|██ | 783/3723 [1:02:50<3:4 0: {'loss': 0.7649, 'grad_norm': 0.9856011357561554, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.13} 0: 8:13, 4.66s/it] 21%|██ | 784/3723 [1:02:54<3:47:41, 4.65s/it] 21%|██ | 785/3723 [1:02:59<3:47:24, 4.64s/it] 21%|██ | 786/3723 [1:03:04<3:47:05, 4.64s/it] 21%|██ | 787/3723 [1:03:08<3:47:00, 4.64s/it] 21%|██ | 788/3723 [1:03:13<3:46:45, 4.64s/it] 21%|██ | 789/3723 [1:03:17<3:46:54, 4.64s/it] 21%|██ | 790/3723 [1:03:22<3:46:37, 4.64s/it] 21%|██ | 790/3723 [1:03:22<3:46:37, 4.64s/it] 21%|██ | 791/3723 [1:03:27<3:46:27, 4.63s/it] 21%|██▏ | 792/3723 [1:03:31<3:46:10, 4.63s/it] 21%|██▏ | 793/3723 [1:03:36<3:46:07, 4.63s/it] 21%|██▏ | 794/3723 [1:03:41<3:46:16, 4.64s/it] 21%|██▏ | 795/3723 [1:03:45<3:45:35, 4.62s/it] 21%|██▏ | 796/3723 [1:03:50<3:45:21, 4.62s/it] 21%|██▏ | 797/3723 [1:03:54<3:45:17, 4.62s/it] 21%|██▏ | 798/3723 [1:03:59<3:45:16, 4.6 0: {'loss': 0.7595, 'grad_norm': 1.1387666975769444, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.13} 0: {'loss': 0.7611, 'grad_norm': 1.1106421044812071, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.13} 0: 2s/it] 21%|██▏ | 799/3723 [1:04:04<3:45:51, 4.63s/it] 21%|██▏ | 800/3723 [1:04:08<3:46:29, 4.65s/it] 21%|██▏ | 800/3723 [1:04:08<3:46:29, 4.65s/it] 22%|██▏ | 801/3723 [1:04:13<3:47:21, 4.67s/it] 22%|██▏ | 802/3723 [1:04:18<3:47:00, 4.66s/it] 22%|██▏ | 803/3723 [1:04:22<3:46:29, 4.65s/it] 22%|██▏ | 804/3723 [1:04:27<3:47:17, 4.67s/it] 22%|██▏ | 805/3723 [1:04:32<3:47:26, 4.68s/it] 22%|██▏ | 806/3723 [1:04:36<3:46:53, 4.67s/it] 22%|██▏ | 807/3723 [1:04:41<3:46:37, 4.66s/it] 22%|██▏ | 808/3723 [1:04:46<3:45:44, 4.65s/it] 22%|██▏ | 809/3723 [1:04:50<3:45:06, 4.63s/it] 22%|██▏ | 810/3723 [1:04:55<3:45:06, 4.64s/it] 22%|██▏ | 810/3723 [1:04:55<3:45:06, 4.64s/it] 22%|██▏ | 811/3723 [1:05:00<3:44:53, 0: {'loss': 0.7552, 'grad_norm': 1.0300280586712922, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.13} 0: 4.63s/it] 22%|██▏ | 812/3723 [1:05:04<3:45:07, 4.64s/it] 22%|██▏ | 813/3723 [1:05:09<3:45:13, 4.64s/it] 22%|██▏ | 814/3723 [1:05:14<3:45:10, 4.64s/it] 22%|██▏ | 815/3723 [1:05:18<3:47:36, 4.70s/it] 22%|██▏ | 816/3723 [1:05:23<3:50:38, 4.76s/it] 22%|██▏ | 817/3723 [1:05:30<4:11:57, 5.20s/it] 22%|██▏ | 818/3723 [1:05:34<4:04:48, 5.06s/it] 22%|██▏ | 819/3723 [1:05:39<3:59:30, 4.95s/it] 22%|██▏ | 820/3723 [1:05:44<3:57:27, 4.91s/it] 22%|██▏ | 820/3723 [1:05:44<3:57:27, 4.91s/it] 22%|██▏ | 821/3723 [1:05:49<3:55:18, 4.86s/it] 22%|██▏ | 822/3723 [1:05:53<3:52:40, 4.81s/it] 22%|██▏ | 823/3723 [1:05:58<3:50:32, 4.77s/it] 22%|██▏ | 824/3723 [1:06:03<3:49:01, 4.74s/it] 22%|██▏ | 825/3723 [1:06:07<3:49:08, 4.74s/it] 22%|██▏ | 826/3723 [1:06:12<3: 0: {'loss': 0.7656, 'grad_norm': 1.115536122864361, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.13} 0: 49:26, 4.75s/it] 22%|██▏ | 827/3723 [1:06:17<3:49:32, 4.76s/it] 22%|██▏ | 828/3723 [1:06:22<3:49:17, 4.75s/it] 22%|██▏ | 829/3723 [1:06:26<3:49:45, 4.76s/it] 22%|██▏ | 830/3723 [1:06:31<3:49:58, 4.77s/it] 22%|██▏ | 830/3723 [1:06:31<3:49:58, 4.77s/it] 22%|██▏ | 831/3723 [1:06:36<3:50:12, 4.78s/it] 22%|██▏ | 832/3723 [1:06:41<3:49:51, 4.77s/it] 22%|██▏ | 833/3723 [1:06:45<3:48:35, 4.75s/it] 22%|██▏ | 834/3723 [1:06:50<3:47:56, 4.73s/it] 22%|██▏ | 835/3723 [1:06:55<3:47:18, 4.72s/it] 22%|██▏ | 836/3723 [1:07:00<3:47:01, 4.72s/it] 22%|██▏ | 837/3723 [1:07:04<3:46:54, 4.72s/it] 23%|██▎ | 838/3723 [1:07:09<3:46:47, 4.72s/it] 23%|██▎ | 839/3723 [1:07:14<3:46:30, 4.71s/it] 23%|██▎ | 840/3723 [1:07:18<3:46:19, 4.71s/it] 0: {'loss': 0.7621, 'grad_norm': 1.0292315550042848, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.14} 0: {'loss': 0.7472, 'grad_norm': 0.930308464256181, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.14} 0: 23%|██▎ | 840/3723 [1:07:18<3:46:19, 4.71s/it] 23%|██▎ | 841/3723 [1:07:23<3:46:22, 4.71s/it] 23%|██▎ | 842/3723 [1:07:28<3:46:22, 4.71s/it] 23%|██▎ | 843/3723 [1:07:33<3:46:28, 4.72s/it] 23%|██▎ | 844/3723 [1:07:37<3:46:46, 4.73s/it] 23%|██▎ | 845/3723 [1:07:42<3:46:37, 4.72s/it] 23%|██▎ | 846/3723 [1:07:47<3:46:29, 4.72s/it] 23%|██▎ | 847/3723 [1:07:52<3:47:59, 4.76s/it] 23%|██▎ | 848/3723 [1:07:56<3:48:59, 4.78s/it] 23%|██▎ | 849/3723 [1:08:01<3:48:34, 4.77s/it] 23%|██▎ | 850/3723 [1:08:06<3:47:57, 4.76s/it] 23%|██▎ | 850/3723 [1:08:06<3:47:57, 4.76s/it] 23%|██▎ | 851/3723 [1:08:11<3:47:54, 4.76s/it] 23%|██▎ | 852/3723 [1:08:15<3:47:23, 4.75s/it] 23%|██▎ | 853/3723 [1:08:20<3:46:51, 4.74s/it] 23%|██▎ | 854/3723 0: {'loss': 0.7667, 'grad_norm': 1.1323839431945077, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.14} 0: [1:08:25<3:46:13, 4.73s/it] 23%|██▎ | 855/3723 [1:08:29<3:45:39, 4.72s/it] 23%|██▎ | 856/3723 [1:08:34<3:45:03, 4.71s/it] 23%|██▎ | 857/3723 [1:08:39<3:45:02, 4.71s/it] 23%|██▎ | 858/3723 [1:08:44<3:44:47, 4.71s/it] 23%|██▎ | 859/3723 [1:08:48<3:44:33, 4.70s/it] 23%|██▎ | 860/3723 [1:08:53<3:44:14, 4.70s/it] 23%|██▎ | 860/3723 [1:08:53<3:44:14, 4.70s/it] 23%|██▎ | 861/3723 [1:08:58<3:44:19, 4.70s/it] 23%|██▎ | 862/3723 [1:09:02<3:44:14, 4.70s/it] 23%|██▎ | 863/3723 [1:09:07<3:44:01, 4.70s/it] 23%|██▎ | 864/3723 [1:09:12<3:44:03, 4.70s/it] 23%|██▎ | 865/3723 [1:09:16<3:44:11, 4.71s/it] 23%|██▎ | 866/3723 [1:09:21<3:44:16, 4.71s/it] 23%|██▎ | 867/3723 [1:09:26<3:44:14, 4.71s/it] 23%|██▎ | 868/3723 [1:09:31<3:44:23, 4.72s/it] 23%|██▎ | 8 0: {'loss': 0.7452, 'grad_norm': 0.9259288842628004, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.14} 0: {'loss': 0.7509, 'grad_norm': 1.0068010671221423, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.14} 0: 69/3723 [1:09:35<3:44:35, 4.72s/it] 23%|██▎ | 870/3723 [1:09:40<3:44:39, 4.72s/it] 23%|██▎ | 870/3723 [1:09:40<3:44:39, 4.72s/it] 23%|██▎ | 871/3723 [1:09:45<3:44:51, 4.73s/it] 23%|██▎ | 872/3723 [1:09:50<3:47:48, 4.79s/it] 23%|██▎ | 873/3723 [1:09:55<3:47:11, 4.78s/it] 23%|██▎ | 874/3723 [1:09:59<3:46:36, 4.77s/it] 24%|██▎ | 875/3723 [1:10:04<3:45:47, 4.76s/it] 24%|██▎ | 876/3723 [1:10:09<3:45:25, 4.75s/it] 24%|██▎ | 877/3723 [1:10:13<3:45:17, 4.75s/it] 24%|██▎ | 878/3723 [1:10:18<3:44:51, 4.74s/it] 24%|██▎ | 879/3723 [1:10:23<3:45:55, 4.77s/it] 24%|██▎ | 880/3723 [1:10:28<3:45:28, 4.76s/it] 24%|██▎ | 880/3723 [1:10:28<3:45:28, 4.76s/it] 24%|██▎ | 881/3723 [1:10:33<3:45:15, 4.76s/it] 24%|██▎ 0: {'loss': 0.7721, 'grad_norm': 1.010398184391979, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.14} 0: | 882/3723 [1:10:37<3:44:51, 4.75s/it] 24%|██▎ | 883/3723 [1:10:42<3:44:42, 4.75s/it] 24%|██▎ | 884/3723 [1:10:47<3:44:35, 4.75s/it] 24%|██▍ | 885/3723 [1:10:52<3:44:39, 4.75s/it] 24%|██▍ | 886/3723 [1:10:56<3:44:29, 4.75s/it] 24%|██▍ | 887/3723 [1:11:01<3:44:08, 4.74s/it] 24%|██▍ | 888/3723 [1:11:06<3:43:37, 4.73s/it] 24%|██▍ | 889/3723 [1:11:10<3:43:21, 4.73s/it] 24%|██▍ | 890/3723 [1:11:15<3:42:53, 4.72s/it] 24%|██▍ | 890/3723 [1:11:15<3:42:53, 4.72s/it] 24%|██▍ | 891/3723 [1:11:20<3:42:43, 4.72s/it] 24%|██▍ | 892/3723 [1:11:25<3:42:05, 4.71s/it] 24%|██▍ | 893/3723 [1:11:29<3:41:50, 4.70s/it] 24%|██▍ | 894/3723 [1:11:34<3:43:56, 4.75s/it] 24%|██▍ | 895/3723 [1:11:39<3:44:29, 4.76s/it] 24%|██▍ | 896/3723 [1:11:44<3:43:15, 4.74s/it] 24%|██ 0: {'loss': 0.7597, 'grad_norm': 1.005460174235457, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.15} 0: {'loss': 0.7575, 'grad_norm': 0.9469835483569223, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.15} 0: | 897/3723 [1:11:48<3:42:01, 4.71s/it] 24%|██▍ | 898/3723 [1:11:53<3:41:19, 4.70s/it] 24%|██▍ | 899/3723 [1:11:58<3:40:57, 4.69s/it] 24%|██▍ | 900/3723 [1:12:02<3:40:54, 4.70s/it] 24%|██▍ | 900/3723 [1:12:02<3:40:54, 4.70s/it] 24%|██▍ | 901/3723 [1:12:07<3:41:09, 4.70s/it] 24%|██▍ | 902/3723 [1:12:12<3:41:09, 4.70s/it] 24%|██▍ | 903/3723 [1:12:16<3:41:33, 4.71s/it] 24%|██▍ | 904/3723 [1:12:21<3:41:11, 4.71s/it] 24%|██▍ | 905/3723 [1:12:26<3:40:56, 4.70s/it] 24%|██▍ | 906/3723 [1:12:30<3:40:47, 4.70s/it] 24%|██▍ | 907/3723 [1:12:35<3:40:52, 4.71s/it] 24%|██▍ | 908/3723 [1:12:40<3:40:36, 4.70s/it] 24%|██▍ | 909/3723 [1:12:45<3:39:58, 4.69s/it] 24%|██▍ | 910/3723 [1:12:49<3:39:24, 4.68s/it] 24%|█ 0: {'loss': 0.7435, 'grad_norm': 0.9164285793766749, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.15} 0: ▍ | 910/3723 [1:12:49<3:39:24, 4.68s/it] 24%|██▍ | 911/3723 [1:12:54<3:39:39, 4.69s/it] 24%|██▍ | 912/3723 [1:12:59<3:39:45, 4.69s/it] 25%|██▍ | 913/3723 [1:13:03<3:39:22, 4.68s/it] 25%|██▍ | 914/3723 [1:13:08<3:38:59, 4.68s/it] 25%|██▍ | 915/3723 [1:13:13<3:38:21, 4.67s/it] 25%|██▍ | 916/3723 [1:13:17<3:38:36, 4.67s/it] 25%|██▍ | 917/3723 [1:13:22<3:39:13, 4.69s/it] 25%|██▍ | 918/3723 [1:13:27<3:39:12, 4.69s/it] 25%|██▍ | 919/3723 [1:13:31<3:39:12, 4.69s/it] 25%|██▍ | 920/3723 [1:13:36<3:39:17, 4.69s/it] 25%|██▍ | 920/3723 [1:13:36<3:39:17, 4.69s/it] 25%|██▍ | 921/3723 [1:13:41<3:38:58, 4.69s/it] 25%|██▍ | 922/3723 [1:13:45<3:38:37, 4.68s/it] 25%|██▍ | 923/3723 [1:13:50<3:38:17, 4.68s/it] 25%|██▍ | 924/3723 [1:13:55<3:38:12, 4.68s/it] 2 0: {'loss': 0.7399, 'grad_norm': 1.0056517323484384, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.15} 0: 5%|██▍ | 925/3723 [1:13:59<3:37:53, 4.67s/it] 25%|██▍ | 926/3723 [1:14:04<3:37:43, 4.67s/it] 25%|██▍ | 927/3723 [1:14:09<3:37:31, 4.67s/it] 25%|██▍ | 928/3723 [1:14:13<3:36:59, 4.66s/it] 25%|██▍ | 929/3723 [1:14:18<3:36:35, 4.65s/it] 25%|██▍ | 930/3723 [1:14:23<3:36:37, 4.65s/it] 25%|██▍ | 930/3723 [1:14:23<3:36:37, 4.65s/it] 25%|██▌ | 931/3723 [1:14:27<3:36:29, 4.65s/it] 25%|██▌ | 932/3723 [1:14:32<3:36:25, 4.65s/it] 25%|██▌ | 933/3723 [1:14:37<3:36:24, 4.65s/it] 25%|██▌ | 934/3723 [1:14:41<3:36:51, 4.67s/it] 25%|██▌ | 935/3723 [1:14:46<3:36:52, 4.67s/it] 25%|██▌ | 936/3723 [1:14:51<3:37:06, 4.67s/it] 25%|██▌ | 937/3723 [1:14:55<3:37:25, 4.68s/it] 25%|██▌ | 938/3723 [1:15:00<3:37:40, 4.69s/it] 25%|██▌ | 939/3723 [1:15:05<3:37:27, 4.69 0: {'loss': 0.7593, 'grad_norm': 0.955400815304853, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.15} 0: {'loss': 0.7368, 'grad_norm': 0.9216466209679349, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.15} 0: s/it] 25%|██▌ | 940/3723 [1:15:09<3:37:29, 4.69s/it] 25%|██▌ | 940/3723 [1:15:10<3:37:29, 4.69s/it] 25%|██▌ | 941/3723 [1:15:14<3:37:40, 4.69s/it] 25%|██▌ | 942/3723 [1:15:19<3:37:35, 4.69s/it] 25%|██▌ | 943/3723 [1:15:24<3:37:25, 4.69s/it] 25%|██▌ | 944/3723 [1:15:28<3:37:32, 4.70s/it] 25%|██▌ | 945/3723 [1:15:33<3:37:14, 4.69s/it] 25%|██▌ | 946/3723 [1:15:38<3:37:08, 4.69s/it] 25%|██▌ | 947/3723 [1:15:42<3:37:17, 4.70s/it] 25%|██▌ | 948/3723 [1:15:47<3:37:07, 4.69s/it] 25%|██▌ | 949/3723 [1:15:52<3:37:01, 4.69s/it] 26%|██▌ | 950/3723 [1:15:56<3:37:16, 4.70s/it] 26%|██▌ | 950/3723 [1:15:56<3:37:16, 4.70s/it] 26%|██▌ | 951/3723 [1:16:01<3:37:16, 4.70s/it] 26%|██▌ | 952/3723 [1:16:06<3:37:16, 4 0: {'loss': 0.7322, 'grad_norm': 0.9884198144373314, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.15} 0: .70s/it] 26%|██▌ | 953/3723 [1:16:11<3:37:15, 4.71s/it] 26%|██▌ | 954/3723 [1:16:15<3:37:48, 4.72s/it] 26%|██▌ | 955/3723 [1:16:20<3:37:54, 4.72s/it] 26%|██▌ | 956/3723 [1:16:25<3:37:27, 4.72s/it] 26%|██▌ | 957/3723 [1:16:29<3:36:40, 4.70s/it] 26%|██▌ | 958/3723 [1:16:34<3:36:12, 4.69s/it] 26%|██▌ | 959/3723 [1:16:39<3:36:04, 4.69s/it] 26%|██▌ | 960/3723 [1:16:43<3:36:07, 4.69s/it] 26%|██▌ | 960/3723 [1:16:44<3:36:07, 4.69s/it] 26%|██▌ | 961/3723 [1:16:48<3:37:19, 4.72s/it] 26%|██▌ | 962/3723 [1:16:53<3:38:03, 4.74s/it] 26%|██▌ | 963/3723 [1:16:58<3:38:40, 4.75s/it] 26%|██▌ | 964/3723 [1:17:03<3:38:55, 4.76s/it] 26%|██▌ | 965/3723 [1:17:07<3:38:55, 4.76s/it] 26%|██▌ | 966/3723 [1:17:12<3:39:33, 4.78s/it] 26%|██▌ | 967/3723 [1:17:17<3:3 0: {'loss': 0.7433, 'grad_norm': 1.0472255751523771, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.16} 0: {'loss': 0.7436, 'grad_norm': 2.8127876425598366, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.16} 0: 8:31, 4.76s/it] 26%|██▌ | 968/3723 [1:17:22<3:37:21, 4.73s/it] 26%|██▌ | 969/3723 [1:17:26<3:36:37, 4.72s/it] 26%|██▌ | 970/3723 [1:17:31<3:35:52, 4.70s/it] 26%|██▌ | 970/3723 [1:17:31<3:35:52, 4.70s/it] 26%|██▌ | 971/3723 [1:17:36<3:35:33, 4.70s/it] 26%|██▌ | 972/3723 [1:17:40<3:35:13, 4.69s/it] 26%|██▌ | 973/3723 [1:17:45<3:35:42, 4.71s/it] 26%|██▌ | 974/3723 [1:17:50<3:35:38, 4.71s/it] 26%|██▌ | 975/3723 [1:17:54<3:35:30, 4.71s/it] 26%|██▌ | 976/3723 [1:17:59<3:35:23, 4.70s/it] 26%|██▌ | 977/3723 [1:18:04<3:34:59, 4.70s/it] 26%|██▋ | 978/3723 [1:18:09<3:35:09, 4.70s/it] 26%|██▋ | 979/3723 [1:18:13<3:34:52, 4.70s/it] 26%|██▋ | 980/3723 [1:18:18<3:34:41, 4.70s/it] 26%|██▋ | 980/3723 [1:18:18< 0: {'loss': 0.7591, 'grad_norm': 0.9230203739283798, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.16} 0: 3:34:41, 4.70s/it] 26%|██▋ | 981/3723 [1:18:23<3:34:14, 4.69s/it] 26%|██▋ | 982/3723 [1:18:27<3:33:44, 4.68s/it] 26%|██▋ | 983/3723 [1:18:32<3:32:58, 4.66s/it] 26%|██▋ | 984/3723 [1:18:37<3:32:27, 4.65s/it] 26%|██▋ | 985/3723 [1:18:41<3:32:10, 4.65s/it] 26%|██▋ | 986/3723 [1:18:46<3:31:51, 4.64s/it] 27%|██▋ | 987/3723 [1:18:50<3:31:25, 4.64s/it] 27%|██▋ | 988/3723 [1:18:55<3:31:45, 4.65s/it] 27%|██▋ | 989/3723 [1:19:00<3:32:41, 4.67s/it] 27%|██▋ | 990/3723 [1:19:05<3:33:07, 4.68s/it] 27%|██▋ | 990/3723 [1:19:05<3:33:07, 4.68s/it] 27%|██▋ | 991/3723 [1:19:09<3:33:16, 4.68s/it] 27%|██▋ | 992/3723 [1:19:14<3:33:33, 4.69s/it] 27%|██▋ | 993/3723 [1:19:19<3:33:31, 4.69s/it] 27%|██▋ | 994/3723 [1:19:23<3:33:18, 4.69s/it] 27%|██▋ | 995/3723 [ 0: {'loss': 0.764, 'grad_norm': 0.968875581980853, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.16} 0: 1:19:28<3:33:12, 4.69s/it] 27%|██▋ | 996/3723 [1:19:33<3:32:52, 4.68s/it] 27%|██▋ | 997/3723 [1:19:37<3:32:54, 4.69s/it] 27%|██▋ | 998/3723 [1:19:42<3:33:07, 4.69s/it] 27%|██▋ | 999/3723 [1:19:47<3:33:15, 4.70s/it] 27%|██▋ | 1000/3723 [1:19:51<3:32:52, 4.69s/it] 27%|██▋ | 1000/3723 [1:19:51<3:32:52, 4.69s/it] 27%|██▋ | 1001/3723 [1:19:56<3:33:06, 4.70s/it] 27%|██▋ | 1002/3723 [1:20:01<3:32:55, 4.70s/it] 27%|██▋ | 1003/3723 [1:20:06<3:32:57, 4.70s/it] 27%|██▋ | 1004/3723 [1:20:10<3:32:44, 4.69s/it] 27%|██▋ | 1005/3723 [1:20:15<3:32:32, 4.69s/it] 27%|██▋ | 1006/3723 [1:20:20<3:32:36, 4.70s/it] 27%|██▋ | 1007/3723 [1:20:24<3:32:34, 4.70s/it] 27%|██▋ | 1008/3723 [1:20:29<3:32:10, 4.69s/it] 27%|██▋ | 1009/3723 [1:20:34<3:31:27, 4.67s/it] 27%|██ 0: {'loss': 0.76, 'grad_norm': 1.0107779550423182, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.16} 0: {'loss': 0.727, 'grad_norm': 0.9467648821945406, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.16} 0: | 1010/3723 [1:20:38<3:30:50, 4.66s/it] 27%|██▋ | 1010/3723 [1:20:38<3:30:50, 4.66s/it] 27%|██▋ | 1011/3723 [1:20:43<3:30:16, 4.65s/it] 27%|██▋ | 1012/3723 [1:20:48<3:29:38, 4.64s/it] 27%|██▋ | 1013/3723 [1:20:52<3:29:17, 4.63s/it] 27%|██▋ | 1014/3723 [1:20:57<3:29:18, 4.64s/it] 27%|██▋ | 1015/3723 [1:21:01<3:28:53, 4.63s/it] 27%|██▋ | 1016/3723 [1:21:06<3:28:30, 4.62s/it] 27%|██▋ | 1017/3723 [1:21:11<3:28:46, 4.63s/it] 27%|██▋ | 1018/3723 [1:21:15<3:28:36, 4.63s/it] 27%|██▋ | 1019/3723 [1:21:20<3:28:24, 4.62s/it] 27%|██▋ | 1020/3723 [1:21:25<3:28:36, 4.63s/it] 27%|██▋ | 1020/3723 [1:21:25<3:28:36, 4.63s/it] 27%|██▋ | 1021/3723 [1:21:29<3:28:56, 4.64s/it] 27%|██▋ | 1022/3723 [1:21:34<3:28:40, 4.6 0: {'loss': 0.7487, 'grad_norm': 1.0467973888390953, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.17} 0: 4s/it] 27%|██▋ | 1023/3723 [1:21:38<3:28:22, 4.63s/it] 28%|██▊ | 1024/3723 [1:21:43<3:28:09, 4.63s/it] 28%|██▊ | 1025/3723 [1:21:48<3:28:10, 4.63s/it] 28%|██▊ | 1026/3723 [1:21:52<3:28:17, 4.63s/it] 28%|██▊ | 1027/3723 [1:21:57<3:28:06, 4.63s/it] 28%|██▊ | 1028/3723 [1:22:02<3:28:06, 4.63s/it] 28%|██▊ | 1029/3723 [1:22:06<3:27:44, 4.63s/it] 28%|██▊ | 1030/3723 [1:22:11<3:27:49, 4.63s/it] 28%|██▊ | 1030/3723 [1:22:11<3:27:49, 4.63s/it] 28%|██▊ | 1031/3723 [1:22:15<3:27:55, 4.63s/it] 28%|██▊ | 1032/3723 [1:22:20<3:28:01, 4.64s/it] 28%|██▊ | 1033/3723 [1:22:25<3:28:10, 4.64s/it] 28%|██▊ | 1034/3723 [1:22:29<3:28:17, 4.65s/it] 28%|██▊ | 1035/3723 [1:22:34<3:28:26, 4.65s/it] 28%|██▊ | 1036/3723 [1:22:39<3:28:23, 4.65s/it] 28%|██▊ | 1037/37 0: {'loss': 0.7497, 'grad_norm': 0.980772468558916, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.17} 0: {'loss': 0.7428, 'grad_norm': 0.9830922749469903, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.17} 0: 23 [1:22:43<3:28:45, 4.66s/it] 28%|██▊ | 1038/3723 [1:22:48<3:28:47, 4.67s/it] 28%|██▊ | 1039/3723 [1:22:53<3:28:52, 4.67s/it] 28%|██▊ | 1040/3723 [1:22:57<3:29:06, 4.68s/it] 28%|██▊ | 1040/3723 [1:22:58<3:29:06, 4.68s/it] 28%|██▊ | 1041/3723 [1:23:02<3:29:16, 4.68s/it] 28%|██▊ | 1042/3723 [1:23:07<3:29:24, 4.69s/it] 28%|██▊ | 1043/3723 [1:23:12<3:29:32, 4.69s/it] 28%|██▊ | 1044/3723 [1:23:16<3:32:06, 4.75s/it] 28%|██▊ | 1045/3723 [1:23:21<3:31:13, 4.73s/it] 28%|██▊ | 1046/3723 [1:23:26<3:30:17, 4.71s/it] 28%|██▊ | 1047/3723 [1:23:30<3:29:38, 4.70s/it] 28%|██▊ | 1048/3723 [1:23:35<3:29:08, 4.69s/it] 28%|██▊ | 1049/3723 [1:23:40<3:28:52, 4.69s/it] 28%|██▊ | 1050/3723 [1:23:45<3:28:56, 4.69s/it] 28%|█ 0: {'loss': 0.7327, 'grad_norm': 0.9499449850354889, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.17} 0: ▊ | 1050/3723 [1:23:45<3:28:56, 4.69s/it] 28%|██▊ | 1051/3723 [1:23:49<3:28:29, 4.68s/it] 28%|██▊ | 1052/3723 [1:23:54<3:28:28, 4.68s/it] 28%|██▊ | 1053/3723 [1:23:59<3:28:09, 4.68s/it] 28%|██▊ | 1054/3723 [1:24:03<3:28:06, 4.68s/it] 28%|██▊ | 1055/3723 [1:24:08<3:28:16, 4.68s/it] 28%|██▊ | 1056/3723 [1:24:13<3:28:27, 4.69s/it] 28%|██▊ | 1057/3723 [1:24:17<3:28:08, 4.68s/it] 28%|██▊ | 1058/3723 [1:24:22<3:28:01, 4.68s/it] 28%|██▊ | 1059/3723 [1:24:27<3:28:07, 4.69s/it] 28%|██▊ | 1060/3723 [1:24:31<3:28:16, 4.69s/it] 28%|██▊ | 1060/3723 [1:24:31<3:28:16, 4.69s/it] 28%|██▊ | 1061/3723 [1:24:36<3:27:52, 4.69s/it] 29%|██▊ | 1062/3723 [1:24:41<3:27:28, 4.68s/it] 29%|██▊ | 1063/3723 [1:24:45<3:27:27, 4.68s/it] 29%|██▊ | 1064/3723 [1:24:50<3:27: 0: {'loss': 0.7306, 'grad_norm': 0.9004344905958201, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.17} 0: 14, 4.68s/it] 29%|██▊ | 1065/3723 [1:24:55<3:27:38, 4.69s/it] 29%|██▊ | 1066/3723 [1:24:59<3:27:33, 4.69s/it] 29%|██▊ | 1067/3723 [1:25:04<3:27:03, 4.68s/it] 29%|██▊ | 1068/3723 [1:25:09<3:26:50, 4.67s/it] 29%|██▊ | 1069/3723 [1:25:13<3:26:32, 4.67s/it] 29%|██▊ | 1070/3723 [1:25:18<3:26:46, 4.68s/it] 29%|██▊ | 1070/3723 [1:25:18<3:26:46, 4.68s/it] 29%|██▉ | 1071/3723 [1:25:23<3:26:38, 4.68s/it] 29%|██▉ | 1072/3723 [1:25:27<3:26:31, 4.67s/it] 29%|██▉ | 1073/3723 [1:25:32<3:26:16, 4.67s/it] 29%|██▉ | 1074/3723 [1:25:37<3:26:04, 4.67s/it] 29%|██▉ | 1075/3723 [1:25:41<3:26:04, 4.67s/it] 29%|██▉ | 1076/3723 [1:25:46<3:26:02, 4.67s/it] 29%|██▉ | 1077/3723 [1:25:51<3:26:05, 4.67s/it] 29%|██▉ | 1078/3723 [1:25:56<3:25:54, 4.67s/it] 29%|██▉ | 0: {'loss': 0.7359, 'grad_norm': 0.8620170591782175, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.17} 0: {'loss': 0.7244, 'grad_norm': 0.9735726328437027, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.18} 0: 1079/3723 [1:26:00<3:25:48, 4.67s/it] 29%|██▉ | 1080/3723 [1:26:05<3:25:21, 4.66s/it] 29%|██▉ | 1080/3723 [1:26:05<3:25:21, 4.66s/it] 29%|██▉ | 1081/3723 [1:26:09<3:24:59, 4.66s/it] 29%|██▉ | 1082/3723 [1:26:14<3:24:19, 4.64s/it] 29%|██▉ | 1083/3723 [1:26:19<3:24:14, 4.64s/it] 29%|██▉ | 1084/3723 [1:26:23<3:23:39, 4.63s/it] 29%|██▉ | 1085/3723 [1:26:28<3:23:28, 4.63s/it] 29%|██▉ | 1086/3723 [1:26:33<3:23:21, 4.63s/it] 29%|██▉ | 1087/3723 [1:26:37<3:23:44, 4.64s/it] 29%|██▉ | 1088/3723 [1:26:42<3:23:40, 4.64s/it] 29%|██▉ | 1089/3723 [1:26:47<3:24:04, 4.65s/it] 29%|██▉ | 1090/3723 [1:26:51<3:24:05, 4.65s/it] 29%|██▉ | 1090/3723 [1:26:51<3:24:05, 4.65s/it] 29%|██▉ | 1091/3723 [1:26:56<3:24:29, 4.66s/it] 2 0: {'loss': 0.7292, 'grad_norm': 0.9431565947873687, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.18} 0: 9%|██▉ | 1092/3723 [1:27:01<3:24:41, 4.67s/it] 29%|██▉ | 1093/3723 [1:27:05<3:24:49, 4.67s/it] 29%|██▉ | 1094/3723 [1:27:10<3:24:51, 4.68s/it] 29%|██▉ | 1095/3723 [1:27:15<3:25:13, 4.69s/it] 29%|██▉ | 1096/3723 [1:27:19<3:25:19, 4.69s/it] 29%|██▉ | 1097/3723 [1:27:24<3:25:52, 4.70s/it] 29%|██▉ | 1098/3723 [1:27:29<3:26:26, 4.72s/it] 30%|██▉ | 1099/3723 [1:27:34<3:26:45, 4.73s/it] 30%|██▉ | 1100/3723 [1:27:38<3:25:45, 4.71s/it] 30%|██▉ | 1100/3723 [1:27:38<3:25:45, 4.71s/it] 30%|██▉ | 1101/3723 [1:27:43<3:25:25, 4.70s/it] 30%|██▉ | 1102/3723 [1:27:48<3:25:01, 4.69s/it] 30%|██▉ | 1103/3723 [1:27:52<3:24:35, 4.69s/it] 30%|██▉ | 1104/3723 [1:27:57<3:25:13, 4.70s/it] 30%|██▉ | 1105/3723 [1:28:02<3:24:27, 4.69s/it] 30%|██▉ | 1106/3723 [1:28: 0: {'loss': 0.7367, 'grad_norm': 0.9564775770458037, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.18} 0: 06<3:23:51, 4.67s/it] 30%|██▉ | 1107/3723 [1:28:11<3:23:22, 4.66s/it] 30%|██▉ | 1108/3723 [1:28:16<3:23:10, 4.66s/it] 30%|██▉ | 1109/3723 [1:28:20<3:23:23, 4.67s/it] 30%|██▉ | 1110/3723 [1:28:25<3:23:05, 4.66s/it] 30%|██▉ | 1110/3723 [1:28:25<3:23:05, 4.66s/it] 30%|██▉ | 1111/3723 [1:28:30<3:23:35, 4.68s/it] 30%|██▉ | 1112/3723 [1:28:34<3:22:50, 4.66s/it] 30%|██▉ | 1113/3723 [1:28:39<3:22:25, 4.65s/it] 30%|██▉ | 1114/3723 [1:28:44<3:22:14, 4.65s/it] 30%|██▉ | 1115/3723 [1:28:48<3:21:42, 4.64s/it] 30%|██▉ | 1116/3723 [1:28:53<3:22:02, 4.65s/it] 30%|███ | 1117/3723 [1:28:57<3:21:34, 4.64s/it] 30%|███ | 1118/3723 [1:29:02<3:21:54, 4.65s/it] 30%|███ | 1119/3723 [1:29:07<3:21:56, 4.65s/it] 30%|███ | 1120/3723 [1:29:11<3:21:41, 4.65s/it] 0: {'loss': 0.7272, 'grad_norm': 0.9278031853669448, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.18} 0: {'loss': 0.7296, 'grad_norm': 1.0818680910571798, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.18} 0: 30%|███ | 1120/3723 [1:29:11<3:21:41, 4.65s/it] 30%|███ | 1121/3723 [1:29:16<3:21:16, 4.64s/it] 30%|███ | 1122/3723 [1:29:21<3:21:16, 4.64s/it] 30%|███ | 1123/3723 [1:29:25<3:21:18, 4.65s/it] 30%|███ | 1124/3723 [1:29:30<3:20:50, 4.64s/it] 30%|███ | 1125/3723 [1:29:35<3:20:31, 4.63s/it] 30%|███ | 1126/3723 [1:29:39<3:21:44, 4.66s/it] 30%|███ | 1127/3723 [1:29:44<3:22:49, 4.69s/it] 30%|███ | 1128/3723 [1:29:49<3:23:39, 4.71s/it] 30%|███ | 1129/3723 [1:29:54<3:24:25, 4.73s/it] 30%|███ | 1130/3723 [1:29:58<3:24:35, 4.73s/it] 30%|███ | 1130/3723 [1:29:58<3:24:35, 4.73s/it] 30%|███ | 1131/3723 [1:30:03<3:24:52, 4.74s/it] 30%|███ | 1132/3723 [1:30:08<3:24:45, 4.74s/it] 30%|███ | 1133/3723 [1:30:13<3:24:37, 4.74 0: {'loss': 0.7294, 'grad_norm': 0.9329012871426992, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.18} 0: s/it] 30%|███ | 1134/3723 [1:30:17<3:24:13, 4.73s/it] 30%|███ | 1135/3723 [1:30:22<3:24:33, 4.74s/it] 31%|███ | 1136/3723 [1:30:27<3:24:38, 4.75s/it] 31%|███ | 1137/3723 [1:30:32<3:24:24, 4.74s/it] 31%|███ | 1138/3723 [1:30:36<3:24:15, 4.74s/it] 31%|███ | 1139/3723 [1:30:41<3:23:42, 4.73s/it] 31%|███ | 1140/3723 [1:30:46<3:22:42, 4.71s/it] 31%|███ | 1140/3723 [1:30:46<3:22:42, 4.71s/it] 31%|███ | 1141/3723 [1:30:50<3:22:04, 4.70s/it] 31%|███ | 1142/3723 [1:30:55<3:22:03, 4.70s/it] 31%|███ | 1143/3723 [1:31:00<3:22:19, 4.71s/it] 31%|███ | 1144/3723 [1:31:04<3:22:18, 4.71s/it] 31%|███ | 1145/3723 [1:31:09<3:22:11, 4.71s/it] 31%|███ | 1146/3723 [1:31:14<3:22:26, 4.71s/it] 31%|███ | 1147/3723 [1:31:19<3:23:53, 4.75s/it] 31%|███ | 1148/372 0: {'loss': 0.7482, 'grad_norm': 0.9057630019652699, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.19} 0: {'loss': 0.7441, 'grad_norm': 0.8153957922048403, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.19} 0: 3 [1:31:24<3:25:44, 4.79s/it] 31%|███ | 1149/3723 [1:31:29<3:30:41, 4.91s/it] 31%|███ | 1150/3723 [1:31:34<3:30:10, 4.90s/it] 31%|███ | 1150/3723 [1:31:34<3:30:10, 4.90s/it] 31%|███ | 1151/3723 [1:31:39<3:29:42, 4.89s/it] 31%|███ | 1152/3723 [1:31:43<3:27:36, 4.85s/it] 31%|███ | 1153/3723 [1:31:48<3:26:00, 4.81s/it] 31%|███ | 1154/3723 [1:31:53<3:24:49, 4.78s/it] 31%|███ | 1155/3723 [1:31:57<3:23:11, 4.75s/it] 31%|███ | 1156/3723 [1:32:02<3:22:18, 4.73s/it] 31%|███ | 1157/3723 [1:32:07<3:21:48, 4.72s/it] 31%|███ | 1158/3723 [1:32:11<3:20:59, 4.70s/it] 31%|███ | 1159/3723 [1:32:16<3:20:27, 4.69s/it] 31%|███ | 1160/3723 [1:32:21<3:19:20, 4.67s/it] 31%|███ | 1160/3723 [1:32:21<3:19:20, 4.67s/it] 31%|██ 0: {'loss': 0.7524, 'grad_norm': 1.0416773595211097, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.19} 0: █ | 1161/3723 [1:32:25<3:18:43, 4.65s/it] 31%|███ | 1162/3723 [1:32:30<3:18:22, 4.65s/it] 31%|███ | 1163/3723 [1:32:35<3:18:09, 4.64s/it] 31%|███▏ | 1164/3723 [1:32:39<3:17:45, 4.64s/it] 31%|███▏ | 1165/3723 [1:32:44<3:17:29, 4.63s/it] 31%|███▏ | 1166/3723 [1:32:48<3:16:57, 4.62s/it] 31%|███▏ | 1167/3723 [1:32:53<3:16:25, 4.61s/it] 31%|███▏ | 1168/3723 [1:32:58<3:16:09, 4.61s/it] 31%|███▏ | 1169/3723 [1:33:02<3:15:45, 4.60s/it] 31%|███▏ | 1170/3723 [1:33:07<3:15:30, 4.59s/it] 31%|███▏ | 1170/3723 [1:33:07<3:15:30, 4.59s/it] 31%|███▏ | 1171/3723 [1:33:11<3:15:35, 4.60s/it] 31%|███▏ | 1172/3723 [1:33:16<3:15:19, 4.59s/it] 32%|███▏ | 1173/3723 [1:33:21<3:15:00, 4.59s/it] 32%|███▏ | 1174/3723 [1:33:25<3:14:59, 4.59s/it] 32%|███▏ | 0: {'loss': 0.7348, 'grad_norm': 0.9164050984813328, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.19} 0: 1175/3723 [1:33:30<3:14:44, 4.59s/it] 32%|███▏ | 1176/3723 [1:33:34<3:14:53, 4.59s/it] 32%|███▏ | 1177/3723 [1:33:39<3:15:00, 4.60s/it] 32%|███▏ | 1178/3723 [1:33:44<3:15:19, 4.60s/it] 32%|███▏ | 1179/3723 [1:33:48<3:15:51, 4.62s/it] 32%|███▏ | 1180/3723 [1:33:53<3:15:31, 4.61s/it] 32%|███▏ | 1180/3723 [1:33:53<3:15:31, 4.61s/it] 32%|███▏ | 1181/3723 [1:33:57<3:15:41, 4.62s/it] 32%|███▏ | 1182/3723 [1:34:02<3:15:17, 4.61s/it] 32%|███▏ | 1183/3723 [1:34:07<3:14:51, 4.60s/it] 32%|███▏ | 1184/3723 [1:34:11<3:15:04, 4.61s/it] 32%|███▏ | 1185/3723 [1:34:16<3:15:11, 4.61s/it] 32%|███▏ | 1186/3723 [1:34:20<3:15:05, 4.61s/it] 32%|███▏ | 1187/3723 [1:34:25<3:14:45, 4.61s/it] 32%|███▏ | 1188/3723 [1:34:30<3:14:50, 4.61s/it] 32%|███▏ | 1189/3 0: {'loss': 0.7466, 'grad_norm': 0.8895454886185938, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.19} 0: {'loss': 0.7275, 'grad_norm': 0.9951462100967345, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.19} 0: 723 [1:34:34<3:15:01, 4.62s/it] 32%|███▏ | 1190/3723 [1:34:39<3:14:59, 4.62s/it] 32%|███▏ | 1190/3723 [1:34:39<3:14:59, 4.62s/it] 32%|███▏ | 1191/3723 [1:34:44<3:15:13, 4.63s/it] 32%|███▏ | 1192/3723 [1:34:48<3:15:12, 4.63s/it] 32%|███▏ | 1193/3723 [1:34:53<3:15:06, 4.63s/it] 32%|███▏ | 1194/3723 [1:34:57<3:15:07, 4.63s/it] 32%|███▏ | 1195/3723 [1:35:02<3:15:06, 4.63s/it] 32%|███▏ | 1196/3723 [1:35:07<3:15:05, 4.63s/it] 32%|███▏ | 1197/3723 [1:35:11<3:15:04, 4.63s/it] 32%|███▏ | 1198/3723 [1:35:16<3:15:02, 4.63s/it] 32%|███▏ | 1199/3723 [1:35:21<3:15:03, 4.64s/it] 32%|███▏ | 1200/3723 [1:35:25<3:14:45, 4.63s/it] 32%|███▏ | 1200/3723 [1:35:25<3:14:45, 4.63s/it] 32%|███▏ | 1201/3723 [1:35:30<3 0: {'loss': 0.7429, 'grad_norm': 0.8939605724136499, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.19} 0: :14:51, 4.64s/it] 32%|███▏ | 1202/3723 [1:35:35<3:14:48, 4.64s/it] 32%|███▏ | 1203/3723 [1:35:39<3:14:53, 4.64s/it] 32%|███▏ | 1204/3723 [1:35:44<3:15:02, 4.65s/it] 32%|███▏ | 1205/3723 [1:35:49<3:15:11, 4.65s/it] 32%|███▏ | 1206/3723 [1:35:53<3:14:46, 4.64s/it] 32%|███▏ | 1207/3723 [1:35:58<3:14:18, 4.63s/it] 32%|███▏ | 1208/3723 [1:36:02<3:14:27, 4.64s/it] 32%|███▏ | 1209/3723 [1:36:07<3:14:26, 4.64s/it] 33%|███▎ | 1210/3723 [1:36:12<3:14:03, 4.63s/it] 33%|███▎ | 1210/3723 [1:36:12<3:14:03, 4.63s/it] 33%|███▎ | 1211/3723 [1:36:16<3:13:56, 4.63s/it] 33%|███▎ | 1212/3723 [1:36:21<3:13:55, 4.63s/it] 33%|███▎ | 1213/3723 [1:36:26<3:13:35, 4.63s/it] 33%|███▎ | 1214/3723 [1:36:30<3:13:33, 4.63s/it] 33%|███▎ | 1215/3723 [1:36:35<3:13:18, 0: {'loss': 0.7343, 'grad_norm': 0.8871448512512669, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.2} 0: 4.62s/it] 33%|███▎ | 1216/3723 [1:36:39<3:13:25, 4.63s/it] 33%|███▎ | 1217/3723 [1:36:44<3:13:09, 4.62s/it] 33%|███▎ | 1218/3723 [1:36:49<3:12:50, 4.62s/it] 33%|███▎ | 1219/3723 [1:36:53<3:12:23, 4.61s/it] 33%|███▎ | 1220/3723 [1:36:58<3:12:29, 4.61s/it] 33%|███▎ | 1220/3723 [1:36:58<3:12:29, 4.61s/it] 33%|███▎ | 1221/3723 [1:37:03<3:12:37, 4.62s/it] 33%|███▎ | 1222/3723 [1:37:07<3:12:37, 4.62s/it] 33%|███▎ | 1223/3723 [1:37:12<3:12:22, 4.62s/it] 33%|███▎ | 1224/3723 [1:37:16<3:12:14, 4.62s/it] 33%|███▎ | 1225/3723 [1:37:21<3:12:21, 4.62s/it] 33%|███▎ | 1226/3723 [1:37:26<3:12:14, 4.62s/it] 33%|███▎ | 1227/3723 [1:37:30<3:11:41, 4.61s/it] 33%|███▎ | 1228/3723 [1:37:35<3:11:19, 4.60s/it] 33%|███▎ | 1229/3723 [1:37:39<3:10:58, 4.59s 0: {'loss': 0.7217, 'grad_norm': 0.9693395898083766, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.2} 0: {'loss': 0.7285, 'grad_norm': 0.9013433873944164, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.2} 0: /it] 33%|███▎ | 1230/3723 [1:37:44<3:11:12, 4.60s/it] 33%|███▎ | 1230/3723 [1:37:44<3:11:12, 4.60s/it] 33%|███▎ | 1231/3723 [1:37:49<3:11:39, 4.61s/it] 33%|███▎ | 1232/3723 [1:37:53<3:11:19, 4.61s/it] 33%|███▎ | 1233/3723 [1:37:58<3:11:30, 4.61s/it] 33%|███▎ | 1234/3723 [1:38:02<3:11:23, 4.61s/it] 33%|███▎ | 1235/3723 [1:38:07<3:11:20, 4.61s/it] 33%|███▎ | 1236/3723 [1:38:12<3:11:11, 4.61s/it] 33%|███▎ | 1237/3723 [1:38:16<3:10:52, 4.61s/it] 33%|███▎ | 1238/3723 [1:38:21<3:10:27, 4.60s/it] 33%|███▎ | 1239/3723 [1:38:25<3:10:26, 4.60s/it] 33%|███▎ | 1240/3723 [1:38:30<3:10:27, 4.60s/it] 33%|███▎ | 1240/3723 [1:38:30<3:10:27, 4.60s/it] 33%|███▎ | 1241/3723 [1:38:35<3:10:51, 4.61s/it] 33%|█ 0: {'loss': 0.723, 'grad_norm': 0.911146475180512, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.2} 0: █▎ | 1242/3723 [1:38:39<3:10:54, 4.62s/it] 33%|███▎ | 1243/3723 [1:38:44<3:10:58, 4.62s/it] 33%|███▎ | 1244/3723 [1:38:49<3:10:50, 4.62s/it] 33%|███▎ | 1245/3723 [1:38:53<3:10:47, 4.62s/it] 33%|███▎ | 1246/3723 [1:38:58<3:10:51, 4.62s/it] 33%|███▎ | 1247/3723 [1:39:02<3:10:46, 4.62s/it] 34%|███▎ | 1248/3723 [1:39:07<3:10:37, 4.62s/it] 34%|███▎ | 1249/3723 [1:39:12<3:10:33, 4.62s/it] 34%|███▎ | 1250/3723 [1:39:16<3:10:22, 4.62s/it] 34%|███▎ | 1250/3723 [1:39:16<3:10:22, 4.62s/it] 34%|███▎ | 1251/3723 [1:39:21<3:09:44, 4.61s/it] 34%|███▎ | 1252/3723 [1:39:25<3:09:34, 4.60s/it] 34%|███▎ | 1253/3723 [1:39:30<3:09:45, 4.61s/it] 34%|███▎ | 1254/3723 [1:39:35<3:09:40, 4.61s/it] 34%|███▎ | 1255/3723 [1:39:39<3:09:44, 4.61s/it] 34%|███ 0: {'loss': 0.7479, 'grad_norm': 0.8958090800566282, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.2} 0: | 1256/3723 [1:39:44<3:09:28, 4.61s/it] 34%|███▍ | 1257/3723 [1:39:49<3:08:53, 4.60s/it] 34%|███▍ | 1258/3723 [1:39:53<3:08:17, 4.58s/it] 34%|███▍ | 1259/3723 [1:39:58<3:07:58, 4.58s/it] 34%|███▍ | 1260/3723 [1:40:05<3:37:33, 5.30s/it] 34%|███▍ | 1260/3723 [1:40:05<3:37:33, 5.30s/it] 34%|███▍ | 1261/3723 [1:40:09<3:29:27, 5.10s/it] 34%|███▍ | 1262/3723 [1:40:14<3:22:50, 4.95s/it] 34%|███▍ | 1263/3723 [1:40:18<3:18:34, 4.84s/it] 34%|███▍ | 1264/3723 [1:40:23<3:16:16, 4.79s/it] 34%|███▍ | 1265/3723 [1:40:28<3:14:49, 4.76s/it] 34%|███▍ | 1266/3723 [1:40:33<3:20:59, 4.91s/it] 34%|███▍ | 1267/3723 [1:40:38<3:17:49, 4.83s/it] 34%|███▍ | 1268/3723 [1:40:42<3:15:34, 4.78s/it] 34%|███▍ | 1269/3723 [1:40:47<3:13:57, 4.74s/it] 34%|███▍ 0: {'loss': 0.7257, 'grad_norm': 0.9016498257282325, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.2} 0: {'loss': 0.715, 'grad_norm': 1.0114365816800095, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.21} 0: | 1270/3723 [1:40:52<3:13:02, 4.72s/it] 34%|███▍ | 1270/3723 [1:40:52<3:13:02, 4.72s/it] 34%|███▍ | 1271/3723 [1:40:56<3:12:15, 4.70s/it] 34%|███▍ | 1272/3723 [1:41:01<3:11:36, 4.69s/it] 34%|███▍ | 1273/3723 [1:41:06<3:11:16, 4.68s/it] 34%|███▍ | 1274/3723 [1:41:10<3:10:55, 4.68s/it] 34%|███▍ | 1275/3723 [1:41:15<3:10:42, 4.67s/it] 34%|███▍ | 1276/3723 [1:41:20<3:10:29, 4.67s/it] 34%|███▍ | 1277/3723 [1:41:24<3:10:20, 4.67s/it] 34%|███▍ | 1278/3723 [1:41:29<3:10:28, 4.67s/it] 34%|███▍ | 1279/3723 [1:41:34<3:10:31, 4.68s/it] 34%|███▍ | 1280/3723 [1:41:38<3:10:24, 4.68s/it] 34%|███▍ | 1280/3723 [1:41:38<3:10:24, 4.68s/it] 34%|███▍ | 1281/3723 [1:41:43<3:10:13, 4.67s/it] 34%|███▍ | 1282/3723 [1 0: {'loss': 0.7178, 'grad_norm': 0.8194896238526527, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.21} 0: :41:48<3:09:53, 4.67s/it] 34%|███▍ | 1283/3723 [1:41:52<3:09:51, 4.67s/it] 34%|███▍ | 1284/3723 [1:41:57<3:09:54, 4.67s/it] 35%|███▍ | 1285/3723 [1:42:02<3:09:53, 4.67s/it] 35%|███▍ | 1286/3723 [1:42:06<3:09:51, 4.67s/it] 35%|███▍ | 1287/3723 [1:42:11<3:13:42, 4.77s/it] 35%|███▍ | 1288/3723 [1:42:16<3:12:18, 4.74s/it] 35%|███▍ | 1289/3723 [1:42:21<3:11:16, 4.71s/it] 35%|███▍ | 1290/3723 [1:42:25<3:10:45, 4.70s/it] 35%|███▍ | 1290/3723 [1:42:25<3:10:45, 4.70s/it] 35%|███▍ | 1291/3723 [1:42:30<3:10:19, 4.70s/it] 35%|███▍ | 1292/3723 [1:42:35<3:10:06, 4.69s/it] 35%|███▍ | 1293/3723 [1:42:39<3:09:04, 4.67s/it] 35%|███▍ | 1294/3723 [1:42:44<3:08:33, 4.66s/it] 35%|███▍ | 1295/3723 [1:42:49<3:12:17, 4.75s/it] 35%|███▍ | 1296/3723 [1:42:54< 0: {'loss': 0.7205, 'grad_norm': 0.877556288192682, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.21} 0: 3:10:20, 4.71s/it] 35%|███▍ | 1297/3723 [1:42:58<3:08:36, 4.66s/it] 35%|███▍ | 1298/3723 [1:43:03<3:07:26, 4.64s/it] 35%|███▍ | 1299/3723 [1:43:07<3:06:27, 4.62s/it] 35%|███▍ | 1300/3723 [1:43:12<3:05:54, 4.60s/it] 35%|███▍ | 1300/3723 [1:43:12<3:05:54, 4.60s/it] 35%|███▍ | 1301/3723 [1:43:16<3:05:51, 4.60s/it] 35%|███▍ | 1302/3723 [1:43:21<3:05:54, 4.61s/it] 35%|███▍ | 1303/3723 [1:43:26<3:06:20, 4.62s/it] 35%|███▌ | 1304/3723 [1:43:30<3:06:10, 4.62s/it] 35%|███▌ | 1305/3723 [1:43:35<3:06:43, 4.63s/it] 35%|███▌ | 1306/3723 [1:43:40<3:07:23, 4.65s/it] 35%|███▌ | 1307/3723 [1:43:44<3:07:46, 4.66s/it] 35%|███▌ | 1308/3723 [1:43:49<3:07:47, 4.67s/it] 35%|███▌ | 1309/3723 [1:43:54<3:07:37, 4.66s/it] 35%|███▌ | 1310/3723 [1:43:58<3:07:22 0: {'loss': 0.7183, 'grad_norm': 1.066306493094566, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.21} 0: {'loss': 0.7339, 'grad_norm': 0.9027734261922332, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.21} 0: , 4.66s/it] 35%|███▌ | 1310/3723 [1:43:58<3:07:22, 4.66s/it] 35%|███▌ | 1311/3723 [1:44:03<3:07:20, 4.66s/it] 35%|███▌ | 1312/3723 [1:44:08<3:07:07, 4.66s/it] 35%|███▌ | 1313/3723 [1:44:12<3:06:47, 4.65s/it] 35%|███▌ | 1314/3723 [1:44:17<3:07:48, 4.68s/it] 35%|███▌ | 1315/3723 [1:44:22<3:08:38, 4.70s/it] 35%|███▌ | 1316/3723 [1:44:27<3:09:16, 4.72s/it] 35%|███▌ | 1317/3723 [1:44:31<3:09:39, 4.73s/it] 35%|███▌ | 1318/3723 [1:44:36<3:10:00, 4.74s/it] 35%|███▌ | 1319/3723 [1:44:41<3:10:07, 4.75s/it] 35%|███▌ | 1320/3723 [1:44:46<3:10:09, 4.75s/it] 35%|███▌ | 1320/3723 [1:44:46<3:10:09, 4.75s/it] 35%|███▌ | 1321/3723 [1:44:50<3:10:15, 4.75s/it] 36%|███▌ | 1322/3723 [1:44:55<3:10:03, 4.75s/it] 0: {'loss': 0.7161, 'grad_norm': 0.9757674047877708, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.21} 0: 36%|███▌ | 1323/3723 [1:45:00<3:10:03, 4.75s/it] 36%|███▌ | 1324/3723 [1:45:05<3:10:03, 4.75s/it] 36%|███▌ | 1325/3723 [1:45:09<3:10:09, 4.76s/it] 36%|███▌ | 1326/3723 [1:45:14<3:11:29, 4.79s/it] 36%|███▌ | 1327/3723 [1:45:19<3:12:45, 4.83s/it] 36%|███▌ | 1328/3723 [1:45:24<3:13:28, 4.85s/it] 36%|███▌ | 1329/3723 [1:45:29<3:12:18, 4.82s/it] 36%|███▌ | 1330/3723 [1:45:33<3:10:27, 4.78s/it] 36%|███▌ | 1330/3723 [1:45:34<3:10:27, 4.78s/it] 36%|███▌ | 1331/3723 [1:45:38<3:09:11, 4.75s/it] 36%|███▌ | 1332/3723 [1:45:43<3:08:05, 4.72s/it] 36%|███▌ | 1333/3723 [1:45:47<3:07:17, 4.70s/it] 36%|███▌ | 1334/3723 [1:45:52<3:06:43, 4.69s/it] 36%|███▌ | 1335/3723 [1:45:57<3:06:32, 4.69s/it] 36%|███▌ | 1336/3723 [1:46:02<3:06:17, 4.68s/it] 36%|█ 0: {'loss': 0.735, 'grad_norm': 0.9252722979804332, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.22} 0: ██▌ | 1337/3723 [1:46:06<3:05:53, 4.67s/it] 36%|███▌ | 1338/3723 [1:46:11<3:05:49, 4.68s/it] 36%|███▌ | 1339/3723 [1:46:16<3:05:42, 4.67s/it] 36%|███▌ | 1340/3723 [1:46:20<3:05:40, 4.67s/it] 36%|███▌ | 1340/3723 [1:46:20<3:05:40, 4.67s/it] 36%|███▌ | 1341/3723 [1:46:25<3:05:46, 4.68s/it] 36%|███▌ | 1342/3723 [1:46:30<3:05:26, 4.67s/it] 36%|███▌ | 1343/3723 [1:46:34<3:04:45, 4.66s/it] 36%|███▌ | 1344/3723 [1:46:39<3:03:55, 4.64s/it] 36%|███▌ | 1345/3723 [1:46:43<3:05:00, 4.67s/it] 36%|███▌ | 1346/3723 [1:46:48<3:06:08, 4.70s/it] 36%|███▌ | 1347/3723 [1:46:53<3:06:47, 4.72s/it] 36%|███▌ | 1348/3723 [1:46:58<3:06:21, 4.71s/it] 36%|███▌ | 1349/3723 [1:47:02<3:04:58, 4.67s/it] 36%|███▋ | 1350/3723 [1:47:07<3:04:29, 4.66s/it] 0: {'loss': 0.7151, 'grad_norm': 0.9593787684934734, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.22} 0: {'loss': 0.7155, 'grad_norm': 0.8701106309767118, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.22} 0: 36%|███▋ | 1350/3723 [1:47:07<3:04:29, 4.66s/it] 36%|███▋ | 1351/3723 [1:47:12<3:04:16, 4.66s/it] 36%|███▋ | 1352/3723 [1:47:16<3:03:48, 4.65s/it] 36%|███▋ | 1353/3723 [1:47:21<3:04:58, 4.68s/it] 36%|███▋ | 1354/3723 [1:47:26<3:04:30, 4.67s/it] 36%|███▋ | 1355/3723 [1:47:30<3:04:03, 4.66s/it] 36%|███▋ | 1356/3723 [1:47:35<3:04:04, 4.67s/it] 36%|███▋ | 1357/3723 [1:47:40<3:03:17, 4.65s/it] 36%|███▋ | 1358/3723 [1:47:44<3:03:07, 4.65s/it] 37%|███▋ | 1359/3723 [1:47:49<3:03:16, 4.65s/it] 37%|███▋ | 1360/3723 [1:47:54<3:03:10, 4.65s/it] 37%|███▋ | 1360/3723 [1:47:54<3:03:10, 4.65s/it] 37%|███▋ | 1361/3723 [1:47:58<3:03:26, 4.66s/it] 37%|███▋ | 1362/3723 [1:48:03<3:02:46, 4.64s/it] 37%|███▋ | 1363 0: {'loss': 0.7141, 'grad_norm': 0.9231839390965166, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.22} 0: /3723 [1:48:07<3:02:24, 4.64s/it] 37%|███▋ | 1364/3723 [1:48:12<3:01:54, 4.63s/it] 37%|███▋ | 1365/3723 [1:48:17<3:01:28, 4.62s/it] 37%|███▋ | 1366/3723 [1:48:21<3:01:02, 4.61s/it] 37%|███▋ | 1367/3723 [1:48:26<3:00:55, 4.61s/it] 37%|███▋ | 1368/3723 [1:48:30<3:01:19, 4.62s/it] 37%|███▋ | 1369/3723 [1:48:35<3:01:20, 4.62s/it] 37%|███▋ | 1370/3723 [1:48:40<3:01:14, 4.62s/it] 37%|███▋ | 1370/3723 [1:48:40<3:01:14, 4.62s/it] 37%|███▋ | 1371/3723 [1:48:44<3:01:27, 4.63s/it] 37%|███▋ | 1372/3723 [1:48:49<3:01:12, 4.62s/it] 37%|███▋ | 1373/3723 [1:48:54<3:00:52, 4.62s/it] 37%|███▋ | 1374/3723 [1:48:58<3:00:37, 4.61s/it] 37%|███▋ | 1375/3723 [1:49:03<2:59:55, 4.60s/it] 37%|███▋ | 1376/3723 [1:49:07<2:59:30, 4.59s/it] 37%|███▋ | 1377/3723 [ 0: {'loss': 0.7252, 'grad_norm': 1.062808003431976, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.22} 0: 1:49:12<2:59:04, 4.58s/it] 37%|███▋ | 1378/3723 [1:49:16<2:59:14, 4.59s/it] 37%|███▋ | 1379/3723 [1:49:21<2:59:04, 4.58s/it] 37%|███▋ | 1380/3723 [1:49:26<2:59:07, 4.59s/it] 37%|███▋ | 1380/3723 [1:49:26<2:59:07, 4.59s/it] 37%|███▋ | 1381/3723 [1:49:30<2:59:13, 4.59s/it] 37%|███▋ | 1382/3723 [1:49:35<2:58:59, 4.59s/it] 37%|███▋ | 1383/3723 [1:49:39<2:59:05, 4.59s/it] 37%|███▋ | 1384/3723 [1:49:44<2:58:53, 4.59s/it] 37%|███▋ | 1385/3723 [1:49:49<2:58:55, 4.59s/it] 37%|███▋ | 1386/3723 [1:49:53<2:59:21, 4.60s/it] 37%|███▋ | 1387/3723 [1:49:58<2:59:18, 4.61s/it] 37%|███▋ | 1388/3723 [1:50:02<2:58:54, 4.60s/it] 37%|███▋ | 1389/3723 [1:50:07<2:58:55, 4.60s/it] 37%|███▋ | 1390/3723 [1:50:12<2:59:16, 4.61s/it] 0: {'loss': 0.7105, 'grad_norm': 0.9048330788467592, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.22} 0: {'loss': 0.7255, 'grad_norm': 1.0558761724958179, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.23} 0: 37%|███▋ | 1390/3723 [1:50:12<2:59:16, 4.61s/it] 37%|███▋ | 1391/3723 [1:50:16<2:59:14, 4.61s/it] 37%|███▋ | 1392/3723 [1:50:21<2:59:18, 4.62s/it] 37%|███▋ | 1393/3723 [1:50:25<2:59:00, 4.61s/it] 37%|███▋ | 1394/3723 [1:50:30<3:00:34, 4.65s/it] 37%|███▋ | 1395/3723 [1:50:35<3:00:37, 4.66s/it] 37%|███▋ | 1396/3723 [1:50:40<3:02:22, 4.70s/it] 38%|███▊ | 1397/3723 [1:50:44<3:02:35, 4.71s/it] 38%|███▊ | 1398/3723 [1:50:49<3:01:24, 4.68s/it] 38%|███▊ | 1399/3723 [1:50:54<3:00:53, 4.67s/it] 38%|███▊ | 1400/3723 [1:50:58<3:00:06, 4.65s/it] 38%|███▊ | 1400/3723 [1:50:58<3:00:06, 4.65s/it] 38%|███▊ | 1401/3723 [1:51:03<2:59:34, 4.64s/it] 38%|███▊ | 1402/3723 [1:51:08<2:59:08, 4.63s/it] 38%|███▊ | 1403/3723 [1:51:12<2:58:46, 4.6 0: {'loss': 0.7116, 'grad_norm': 0.9193318355676108, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.23} 0: 2s/it] 38%|███▊ | 1404/3723 [1:51:17<2:59:30, 4.64s/it] 38%|███▊ | 1405/3723 [1:51:22<3:00:57, 4.68s/it] 38%|███▊ | 1406/3723 [1:51:26<3:01:00, 4.69s/it] 38%|███▊ | 1407/3723 [1:51:31<3:00:19, 4.67s/it] 38%|███▊ | 1408/3723 [1:51:36<2:59:21, 4.65s/it] 38%|███▊ | 1409/3723 [1:51:40<2:58:48, 4.64s/it] 38%|███▊ | 1410/3723 [1:51:45<2:58:07, 4.62s/it] 38%|███▊ | 1410/3723 [1:51:45<2:58:07, 4.62s/it] 38%|███▊ | 1411/3723 [1:51:49<2:57:52, 4.62s/it] 38%|███▊ | 1412/3723 [1:51:54<2:57:48, 4.62s/it] 38%|███▊ | 1413/3723 [1:51:59<2:57:52, 4.62s/it] 38%|███▊ | 1414/3723 [1:52:03<2:57:20, 4.61s/it] 38%|███▊ | 1415/3723 [1:52:08<2:57:30, 4.61s/it] 38%|███▊ | 1416/3723 [1:52:12<2:57:25, 4.61s/it] 38%|███▊ | 1417/3723 [1:52:17<2:57:03, 4.61s/it] 0: {'loss': 0.7143, 'grad_norm': 1.3773593249717304, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.23} 0: {'loss': 0.7424, 'grad_norm': 0.9054272092732112, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.23} 0: 38%|███▊ | 1418/3723 [1:52:22<2:56:45, 4.60s/it] 38%|███▊ | 1419/3723 [1:52:26<2:56:16, 4.59s/it] 38%|███▊ | 1420/3723 [1:52:31<2:56:25, 4.60s/it] 38%|███▊ | 1420/3723 [1:52:31<2:56:25, 4.60s/it] 38%|███▊ | 1421/3723 [1:52:35<2:56:54, 4.61s/it] 38%|███▊ | 1422/3723 [1:52:40<2:56:50, 4.61s/it] 38%|███▊ | 1423/3723 [1:52:45<2:56:52, 4.61s/it] 38%|███▊ | 1424/3723 [1:52:49<2:56:45, 4.61s/it] 38%|███▊ | 1425/3723 [1:52:54<2:56:49, 4.62s/it] 38%|███▊ | 1426/3723 [1:52:58<2:56:43, 4.62s/it] 38%|███▊ | 1427/3723 [1:53:03<2:57:00, 4.63s/it] 38%|███▊ | 1428/3723 [1:53:08<2:57:05, 4.63s/it] 38%|███▊ | 1429/3723 [1:53:12<2:57:14, 4.64s/it] 38%|███▊ | 1430/3723 [1:53:17<2:57:18, 4.64s/it] 38%|███ 0: {'loss': 0.7241, 'grad_norm': 0.8523956139243641, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.23} 0: ▊ | 1430/3723 [1:53:17<2:57:18, 4.64s/it] 38%|███▊ | 1431/3723 [1:53:22<2:57:41, 4.65s/it] 38%|███▊ | 1432/3723 [1:53:26<2:57:51, 4.66s/it] 38%|███▊ | 1433/3723 [1:53:31<2:57:37, 4.65s/it] 39%|███▊ | 1434/3723 [1:53:36<2:57:21, 4.65s/it] 39%|███▊ | 1435/3723 [1:53:40<2:56:47, 4.64s/it] 39%|███▊ | 1436/3723 [1:53:45<2:56:31, 4.63s/it] 39%|███▊ | 1437/3723 [1:53:50<2:56:42, 4.64s/it] 39%|███▊ | 1438/3723 [1:53:54<2:56:54, 4.65s/it] 39%|███▊ | 1439/3723 [1:53:59<2:56:50, 4.65s/it] 39%|███▊ | 1440/3723 [1:54:04<2:56:21, 4.64s/it] 39%|███▊ | 1440/3723 [1:54:04<2:56:21, 4.64s/it] 39%|███▊ | 1441/3723 [1:54:08<2:56:09, 4.63s/it] 39%|███▊ | 1442/3723 [1:54:13<2:55:44, 4.62s/it] 39%|███▉ | 1443/3723 [1:54:17<2:55:29, 4.62s/it] 39%|███▉ 0: {'loss': 0.7288, 'grad_norm': 0.8473091308464251, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.23} 0: | 1444/3723 [1:54:22<2:55:25, 4.62s/it] 39%|███▉ | 1445/3723 [1:54:27<2:55:42, 4.63s/it] 39%|███▉ | 1446/3723 [1:54:31<2:55:42, 4.63s/it] 39%|███▉ | 1447/3723 [1:54:36<2:55:34, 4.63s/it] 39%|███▉ | 1448/3723 [1:54:40<2:55:33, 4.63s/it] 39%|███▉ | 1449/3723 [1:54:45<2:55:34, 4.63s/it] 39%|███▉ | 1450/3723 [1:54:50<2:55:22, 4.63s/it] 39%|███▉ | 1450/3723 [1:54:50<2:55:22, 4.63s/it] 39%|███▉ | 1451/3723 [1:54:54<2:55:29, 4.63s/it] 39%|███▉ | 1452/3723 [1:54:59<2:55:14, 4.63s/it] 39%|███▉ | 1453/3723 [1:55:04<2:55:12, 4.63s/it] 39%|███▉ | 1454/3723 [1:55:08<2:55:12, 4.63s/it] 39%|███▉ | 1455/3723 [1:55:13<2:55:29, 4.64s/it] 39%|███▉ | 1456/3723 [1:55:18<2:55:43, 4.65s/it] 39%|███▉ | 1457/3723 [1:55:22<2:55:06, 4.64s/it] 39%|███▉ | 145 0: {'loss': 0.71, 'grad_norm': 0.935162165806545, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.24} 0: {'loss': 0.7364, 'grad_norm': 0.9274516513271921, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.24} 0: 8/3723 [1:55:27<2:54:34, 4.62s/it] 39%|███▉ | 1459/3723 [1:55:31<2:53:56, 4.61s/it] 39%|███▉ | 1460/3723 [1:55:36<2:53:28, 4.60s/it] 39%|███▉ | 1460/3723 [1:55:36<2:53:28, 4.60s/it] 39%|███▉ | 1461/3723 [1:55:41<2:53:51, 4.61s/it] 39%|███▉ | 1462/3723 [1:55:45<2:53:24, 4.60s/it] 39%|███▉ | 1463/3723 [1:55:50<2:53:14, 4.60s/it] 39%|███▉ | 1464/3723 [1:55:54<2:53:17, 4.60s/it] 39%|███▉ | 1465/3723 [1:55:59<2:53:11, 4.60s/it] 39%|███▉ | 1466/3723 [1:56:04<2:52:52, 4.60s/it] 39%|███▉ | 1467/3723 [1:56:08<2:52:40, 4.59s/it] 39%|███▉ | 1468/3723 [1:56:13<2:53:05, 4.61s/it] 39%|███▉ | 1469/3723 [1:56:17<2:52:49, 4.60s/it] 39%|███▉ | 1470/3723 [1:56:22<2:53:01, 4.61s/it] 39%|███▉ | 1470/3723 [1:56:2 0: {'loss': 0.7254, 'grad_norm': 0.9378747587825189, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.24} 0: 2<2:53:01, 4.61s/it] 40%|███▉ | 1471/3723 [1:56:27<2:53:38, 4.63s/it] 40%|███▉ | 1472/3723 [1:56:31<2:53:26, 4.62s/it] 40%|███▉ | 1473/3723 [1:56:36<2:53:38, 4.63s/it] 40%|███▉ | 1474/3723 [1:56:41<2:53:26, 4.63s/it] 40%|███▉ | 1475/3723 [1:56:45<2:53:50, 4.64s/it] 40%|███▉ | 1476/3723 [1:56:50<2:53:57, 4.65s/it] 40%|███▉ | 1477/3723 [1:56:55<2:53:56, 4.65s/it] 40%|███▉ | 1478/3723 [1:56:59<2:54:13, 4.66s/it] 40%|███▉ | 1479/3723 [1:57:04<2:54:25, 4.66s/it] 40%|███▉ | 1480/3723 [1:57:09<2:54:34, 4.67s/it] 40%|███▉ | 1480/3723 [1:57:09<2:54:34, 4.67s/it] 40%|███▉ | 1481/3723 [1:57:13<2:54:33, 4.67s/it] 40%|███▉ | 1482/3723 [1:57:18<2:54:05, 4.66s/it] 40%|███▉ | 1483/3723 [1:57:23<2:54:01, 4.66s/it] 40%|███▉ | 1484/3723 [1:57:27<2:53: 0: {'loss': 0.7046, 'grad_norm': 0.9090332105057263, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.24} 0: 46, 4.66s/it] 40%|███▉ | 1485/3723 [1:57:32<2:53:33, 4.65s/it] 40%|███▉ | 1486/3723 [1:57:37<2:53:23, 4.65s/it] 40%|███▉ | 1487/3723 [1:57:41<2:53:16, 4.65s/it] 40%|███▉ | 1488/3723 [1:57:46<2:53:09, 4.65s/it] 40%|███▉ | 1489/3723 [1:57:50<2:52:45, 4.64s/it] 40%|████ | 1490/3723 [1:57:55<2:52:30, 4.64s/it] 40%|████ | 1490/3723 [1:57:55<2:52:30, 4.64s/it] 40%|████ | 1491/3723 [1:58:00<2:52:40, 4.64s/it] 40%|████ | 1492/3723 [1:58:04<2:52:31, 4.64s/it] 40%|████ | 1493/3723 [1:58:09<2:52:23, 4.64s/it] 40%|████ | 1494/3723 [1:58:14<2:52:20, 4.64s/it] 40%|████ | 1495/3723 [1:58:18<2:51:47, 4.63s/it] 40%|████ | 1496/3723 [1:58:23<2:51:31, 4.62s/it] 40%|████ | 1497/3723 [1:58:27<2:51:12, 4.61s/it] 40%|████ | 1498/3723 [1:58:32<2:50:42, 4. 0: {'loss': 0.7348, 'grad_norm': 0.89673359340811, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.24} 0: {'loss': 0.7194, 'grad_norm': 0.9672472265916219, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.24} 0: 60s/it] 40%|████ | 1499/3723 [1:58:37<2:50:17, 4.59s/it] 40%|████ | 1500/3723 [1:58:41<2:49:50, 4.58s/it] 40%|████ | 1500/3723 [1:58:41<2:49:50, 4.58s/it] 40%|████ | 1501/3723 [1:58:46<2:49:40, 4.58s/it] 40%|████ | 1502/3723 [1:58:50<2:49:24, 4.58s/it] 40%|████ | 1503/3723 [1:58:55<2:49:38, 4.59s/it] 40%|████ | 1504/3723 [1:59:00<2:50:06, 4.60s/it] 40%|████ | 1505/3723 [1:59:04<2:50:18, 4.61s/it] 40%|████ | 1506/3723 [1:59:09<2:50:52, 4.62s/it] 40%|████ | 1507/3723 [1:59:13<2:51:09, 4.63s/it] 41%|████ | 1508/3723 [1:59:18<2:51:24, 4.64s/it] 41%|████ | 1509/3723 [1:59:23<2:51:16, 4.64s/it] 41%|████ | 1510/3723 [1:59:27<2:51:10, 4.64s/it] 41%|████ | 1510/3723 [1:59:27<2:51:10, 4.64s/it] 41%| 0: {'loss': 0.7189, 'grad_norm': 0.888149090065113, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.24} 0: ███ | 1511/3723 [1:59:32<2:51:21, 4.65s/it] 41%|████ | 1512/3723 [1:59:37<2:51:14, 4.65s/it] 41%|████ | 1513/3723 [1:59:41<2:50:53, 4.64s/it] 41%|████ | 1514/3723 [1:59:46<2:50:55, 4.64s/it] 41%|████ | 1515/3723 [1:59:51<2:51:05, 4.65s/it] 41%|████ | 1516/3723 [1:59:55<2:51:04, 4.65s/it] 41%|████ | 1517/3723 [2:00:00<2:51:00, 4.65s/it] 41%|████ | 1518/3723 [2:00:05<2:50:42, 4.65s/it] 41%|████ | 1519/3723 [2:00:09<2:50:37, 4.64s/it] 41%|████ | 1520/3723 [2:00:14<2:50:33, 4.65s/it] 41%|████ | 1520/3723 [2:00:14<2:50:33, 4.65s/it] 41%|████ | 1521/3723 [2:00:19<2:50:41, 4.65s/it] 41%|████ | 1522/3723 [2:00:23<2:50:42, 4.65s/it] 41%|████ | 1523/3723 [2:00:28<2:50:29, 4.65s/it] 41%|████ | 1524/3723 [2:00:32<2:50:10, 4.64s/it] 41%|██ 0: {'loss': 0.7246, 'grad_norm': 0.8750515842718576, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.25} 0: █ | 1525/3723 [2:00:37<2:50:23, 4.65s/it] 41%|████ | 1526/3723 [2:00:42<2:49:57, 4.64s/it] 41%|████ | 1527/3723 [2:00:46<2:49:38, 4.64s/it] 41%|████ | 1528/3723 [2:00:51<2:49:30, 4.63s/it] 41%|████ | 1529/3723 [2:00:56<2:48:44, 4.61s/it] 41%|████ | 1530/3723 [2:01:00<2:48:11, 4.60s/it] 41%|████ | 1530/3723 [2:01:00<2:48:11, 4.60s/it] 41%|████ | 1531/3723 [2:01:05<2:48:13, 4.60s/it] 41%|████ | 1532/3723 [2:01:09<2:48:03, 4.60s/it] 41%|████ | 1533/3723 [2:01:14<2:47:53, 4.60s/it] 41%|████ | 1534/3723 [2:01:19<2:47:44, 4.60s/it] 41%|████ | 1535/3723 [2:01:23<2:47:42, 4.60s/it] 41%|████▏ | 1536/3723 [2:01:28<2:47:33, 4.60s/it] 41%|████▏ | 1537/3723 [2:01:32<2:47:07, 4.59s/it] 41%|████▏ | 1538/3723 [2:01:37<2:46:41, 4.58s/it] 41%|███ 0: {'loss': 0.7263, 'grad_norm': 0.8498913505117374, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.25} 0: {'loss': 0.7114, 'grad_norm': 1.0147304149533272, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.25} 0: █▏ | 1539/3723 [2:01:41<2:46:36, 4.58s/it] 41%|████▏ | 1540/3723 [2:01:46<2:46:17, 4.57s/it] 41%|████▏ | 1540/3723 [2:01:46<2:46:17, 4.57s/it] 41%|████▏ | 1541/3723 [2:01:51<2:46:30, 4.58s/it] 41%|████▏ | 1542/3723 [2:01:55<2:46:17, 4.57s/it] 41%|████▏ | 1543/3723 [2:02:00<2:46:23, 4.58s/it] 41%|████▏ | 1544/3723 [2:02:04<2:46:47, 4.59s/it] 41%|████▏ | 1545/3723 [2:02:09<2:47:05, 4.60s/it] 42%|████▏ | 1546/3723 [2:02:14<2:46:49, 4.60s/it] 42%|████▏ | 1547/3723 [2:02:18<2:46:29, 4.59s/it] 42%|████▏ | 1548/3723 [2:02:23<2:46:29, 4.59s/it] 42%|████▏ | 1549/3723 [2:02:27<2:46:15, 4.59s/it] 42%|████▏ | 1550/3723 [2:02:32<2:46:20, 4.59s/it] 42%|████▏ | 1550/3723 [2:02:32<2:46:20, 4.59s/it] 0: {'loss': 0.7164, 'grad_norm': 0.886339098939481, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.25} 0: 42%|████▏ | 1551/3723 [2:02:37<2:46:33, 4.60s/it] 42%|████▏ | 1552/3723 [2:02:41<2:46:12, 4.59s/it] 42%|████▏ | 1553/3723 [2:02:46<2:46:12, 4.60s/it] 42%|████▏ | 1554/3723 [2:02:50<2:46:12, 4.60s/it] 42%|████▏ | 1555/3723 [2:02:55<2:46:06, 4.60s/it] 42%|████▏ | 1556/3723 [2:02:59<2:45:37, 4.59s/it] 42%|████▏ | 1557/3723 [2:03:04<2:45:16, 4.58s/it] 42%|████▏ | 1558/3723 [2:03:09<2:45:10, 4.58s/it] 42%|████▏ | 1559/3723 [2:03:13<2:45:10, 4.58s/it] 42%|████▏ | 1560/3723 [2:03:18<2:47:32, 4.65s/it] 42%|████▏ | 1560/3723 [2:03:18<2:47:32, 4.65s/it] 42%|████▏ | 1561/3723 [2:03:23<2:47:03, 4.64s/it] 42%|████▏ | 1562/3723 [2:03:27<2:46:29, 4.62s/it] 42%|████▏ | 1563/3723 [2:03:32<2:46:05, 4.61s/it] 42%|████▏ | 1564/3723 [2:03: 0: {'loss': 0.7388, 'grad_norm': 0.9610509944550988, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.25} 0: 36<2:46:07, 4.62s/it] 42%|████▏ | 1565/3723 [2:03:41<2:46:33, 4.63s/it] 42%|████▏ | 1566/3723 [2:03:46<2:46:25, 4.63s/it] 42%|████▏ | 1567/3723 [2:03:50<2:46:13, 4.63s/it] 42%|████▏ | 1568/3723 [2:03:55<2:46:15, 4.63s/it] 42%|████▏ | 1569/3723 [2:04:00<2:47:23, 4.66s/it] 42%|████▏ | 1570/3723 [2:04:04<2:47:08, 4.66s/it] 42%|████▏ | 1570/3723 [2:04:04<2:47:08, 4.66s/it] 42%|████▏ | 1571/3723 [2:04:09<2:47:03, 4.66s/it] 42%|████▏ | 1572/3723 [2:04:14<2:47:06, 4.66s/it] 42%|████▏ | 1573/3723 [2:04:18<2:46:59, 4.66s/it] 42%|████▏ | 1574/3723 [2:04:23<2:49:09, 4.72s/it] 42%|████▏ | 1575/3723 [2:04:28<2:47:42, 4.68s/it] 42%|████▏ | 1576/3723 [2:04:32<2:46:56, 4.67s/it] 42%|████▏ | 1577/3723 [2:04:37<2:46:39, 4.66s/it] 42%|████▏ 0: {'loss': 0.7028, 'grad_norm': 0.842333958153763, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.25} 0: {'loss': 0.6998, 'grad_norm': 0.9291043986892626, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.26} 0: | 1578/3723 [2:04:42<2:46:21, 4.65s/it] 42%|████▏ | 1579/3723 [2:04:46<2:46:16, 4.65s/it] 42%|████▏ | 1580/3723 [2:04:51<2:46:04, 4.65s/it] 42%|████▏ | 1580/3723 [2:04:51<2:46:04, 4.65s/it] 42%|████▏ | 1581/3723 [2:04:56<2:46:01, 4.65s/it] 42%|████▏ | 1582/3723 [2:05:00<2:45:57, 4.65s/it] 43%|████▎ | 1583/3723 [2:05:05<2:45:39, 4.64s/it] 43%|████▎ | 1584/3723 [2:05:10<2:45:29, 4.64s/it] 43%|████▎ | 1585/3723 [2:05:14<2:46:36, 4.68s/it] 43%|████▎ | 1586/3723 [2:05:19<2:46:12, 4.67s/it] 43%|████▎ | 1587/3723 [2:05:24<2:45:54, 4.66s/it] 43%|████▎ | 1588/3723 [2:05:28<2:45:43, 4.66s/it] 43%|████▎ | 1589/3723 [2:05:33<2:45:26, 4.65s/it] 43%|████▎ | 1590/3723 [2:05:38<2:45:21, 4.65s/it] 43%| 0: {'loss': 0.7089, 'grad_norm': 0.9236791050125038, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.26} 0: ███▎ | 1590/3723 [2:05:38<2:45:21, 4.65s/it] 43%|████▎ | 1591/3723 [2:05:42<2:45:20, 4.65s/it] 43%|████▎ | 1592/3723 [2:05:47<2:45:07, 4.65s/it] 43%|████▎ | 1593/3723 [2:05:51<2:44:50, 4.64s/it] 43%|████▎ | 1594/3723 [2:05:56<2:44:30, 4.64s/it] 43%|████▎ | 1595/3723 [2:06:01<2:44:09, 4.63s/it] 43%|████▎ | 1596/3723 [2:06:05<2:43:47, 4.62s/it] 43%|████▎ | 1597/3723 [2:06:10<2:43:52, 4.62s/it] 43%|████▎ | 1598/3723 [2:06:15<2:43:53, 4.63s/it] 43%|████▎ | 1599/3723 [2:06:19<2:43:52, 4.63s/it] 43%|████▎ | 1600/3723 [2:06:24<2:43:36, 4.62s/it] 43%|████▎ | 1600/3723 [2:06:24<2:43:36, 4.62s/it] 43%|████▎ | 1601/3723 [2:06:29<2:44:26, 4.65s/it] 43%|████▎ | 1602/3723 [2:06:33<2:44:47, 4.66s/it] 43%|████▎ | 1603/3723 [2:06:38<2:4 0: {'loss': 0.7087, 'grad_norm': 0.8943019203775638, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.26} 0: 5:15, 4.68s/it] 43%|████▎ | 1604/3723 [2:06:43<2:45:49, 4.70s/it] 43%|████▎ | 1605/3723 [2:06:47<2:46:19, 4.71s/it] 43%|████▎ | 1606/3723 [2:06:52<2:46:38, 4.72s/it] 43%|████▎ | 1607/3723 [2:06:57<2:46:30, 4.72s/it] 43%|████▎ | 1608/3723 [2:07:02<2:45:53, 4.71s/it] 43%|████▎ | 1609/3723 [2:07:06<2:45:17, 4.69s/it] 43%|████▎ | 1610/3723 [2:07:11<2:45:15, 4.69s/it] 43%|████▎ | 1610/3723 [2:07:11<2:45:15, 4.69s/it] 43%|████▎ | 1611/3723 [2:07:16<2:45:08, 4.69s/it] 43%|████▎ | 1612/3723 [2:07:20<2:45:24, 4.70s/it] 43%|████▎ | 1613/3723 [2:07:25<2:45:37, 4.71s/it] 43%|████▎ | 1614/3723 [2:07:30<2:46:35, 4.74s/it] 43%|████▎ | 1615/3723 [2:07:35<2:47:18, 4.76s/it] 43%|████▎ | 1616/3723 [2:07:39<2:47:26, 4.77s/it] 43%|████▎ | 0: {'loss': 0.7002, 'grad_norm': 1.0385970186455526, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.26} 0: 1617/3723 [2:07:44<2:47:28, 4.77s/it] 43%|████▎ | 1618/3723 [2:07:49<2:48:15, 4.80s/it] 43%|████▎ | 1619/3723 [2:07:54<2:48:04, 4.79s/it] 44%|████▎ | 1620/3723 [2:07:59<2:47:46, 4.79s/it] 44%|████▎ | 1620/3723 [2:07:59<2:47:46, 4.79s/it] 44%|████▎ | 1621/3723 [2:08:04<2:48:17, 4.80s/it] 44%|████▎ | 1622/3723 [2:08:08<2:48:30, 4.81s/it] 44%|████▎ | 1623/3723 [2:08:13<2:48:49, 4.82s/it] 44%|████▎ | 1624/3723 [2:08:18<2:48:52, 4.83s/it] 44%|████▎ | 1625/3723 [2:08:23<2:48:45, 4.83s/it] 44%|████▎ | 1626/3723 [2:08:28<2:47:53, 4.80s/it] 44%|████▎ | 1627/3723 [2:08:32<2:46:56, 4.78s/it] 44%|████▎ | 1628/3723 [2:08:37<2:46:14, 4.76s/it] 44%|████▍ | 1629/3723 [2:08:42<2:45:50, 4.75s/it] 44%|████▍ | 1630/3723 [2:08:47<2:46:13, 4.77s/it] 0: {'loss': 0.7038, 'grad_norm': 0.8887187410773796, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.26} 0: {'loss': 0.7137, 'grad_norm': 0.9671206964776897, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.26} 0: 44%|████▍ | 1630/3723 [2:08:47<2:46:13, 4.77s/it] 44%|████▍ | 1631/3723 [2:08:51<2:46:34, 4.78s/it] 44%|████▍ | 1632/3723 [2:08:56<2:46:16, 4.77s/it] 44%|████▍ | 1633/3723 [2:09:01<2:45:48, 4.76s/it] 44%|████▍ | 1634/3723 [2:09:06<2:45:18, 4.75s/it] 44%|████▍ | 1635/3723 [2:09:10<2:45:18, 4.75s/it] 44%|████▍ | 1636/3723 [2:09:15<2:45:08, 4.75s/it] 44%|████▍ | 1637/3723 [2:09:20<2:45:13, 4.75s/it] 44%|████▍ | 1638/3723 [2:09:25<2:45:15, 4.76s/it] 44%|████▍ | 1639/3723 [2:09:29<2:45:10, 4.76s/it] 44%|████▍ | 1640/3723 [2:09:34<2:45:00, 4.75s/it] 44%|████▍ | 1640/3723 [2:09:34<2:45:00, 4.75s/it] 44%|████▍ | 1641/3723 [2:09:39<2:45:03, 4.76s/it] 44%|████▍ | 1642/3723 [2:09:44<2:45:23, 0: {'loss': 0.6999, 'grad_norm': 1.0075240554972094, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.27} 0: 4.77s/it] 44%|████▍ | 1643/3723 [2:09:48<2:45:16, 4.77s/it] 44%|████▍ | 1644/3723 [2:09:53<2:45:20, 4.77s/it] 44%|████▍ | 1645/3723 [2:09:58<2:45:10, 4.77s/it] 44%|████▍ | 1646/3723 [2:10:03<2:44:48, 4.76s/it] 44%|████▍ | 1647/3723 [2:10:07<2:44:24, 4.75s/it] 44%|████▍ | 1648/3723 [2:10:12<2:44:00, 4.74s/it] 44%|████▍ | 1649/3723 [2:10:17<2:43:34, 4.73s/it] 44%|████▍ | 1650/3723 [2:10:22<2:43:37, 4.74s/it] 44%|████▍ | 1650/3723 [2:10:22<2:43:37, 4.74s/it] 44%|████▍ | 1651/3723 [2:10:26<2:44:01, 4.75s/it] 44%|████▍ | 1652/3723 [2:10:31<2:45:43, 4.80s/it] 44%|████▍ | 1653/3723 [2:10:36<2:45:05, 4.79s/it] 44%|████▍ | 1654/3723 [2:10:41<2:43:37, 4.75s/it] 44%|████▍ | 1655/3723 [2:10:45<2:42:45, 4.72s/it] 44%|████▍ | 1656/ 0: {'loss': 0.7044, 'grad_norm': 0.8943988875201934, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.27} 0: 3723 [2:10:50<2:43:08, 4.74s/it] 45%|████▍ | 1657/3723 [2:10:55<2:43:05, 4.74s/it] 45%|████▍ | 1658/3723 [2:11:00<2:42:02, 4.71s/it] 45%|████▍ | 1659/3723 [2:11:04<2:41:32, 4.70s/it] 45%|████▍ | 1660/3723 [2:11:09<2:41:41, 4.70s/it] 45%|████▍ | 1660/3723 [2:11:09<2:41:41, 4.70s/it] 45%|████▍ | 1661/3723 [2:11:14<2:43:15, 4.75s/it] 45%|████▍ | 1662/3723 [2:11:19<2:43:54, 4.77s/it] 45%|████▍ | 1663/3723 [2:11:23<2:44:44, 4.80s/it] 45%|████▍ | 1664/3723 [2:11:28<2:45:46, 4.83s/it] 45%|████▍ | 1665/3723 [2:11:33<2:46:03, 4.84s/it] 45%|████▍ | 1666/3723 [2:11:38<2:45:03, 4.81s/it] 45%|████▍ | 1667/3723 [2:11:43<2:43:29, 4.77s/it] 45%|████▍ | 1668/3723 [2:11:47<2:43:45, 4.78s/it] 45%|████▍ | 1669/3723 [2:11:52<2:43:57, 4.79s/it] 45%|█ 0: {'loss': 0.6987, 'grad_norm': 0.8740443936481933, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.27} 0: {'loss': 0.7097, 'grad_norm': 0.89252623033561, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.27} 0: ██▍ | 1670/3723 [2:11:57<2:44:22, 4.80s/it] 45%|████▍ | 1670/3723 [2:11:57<2:44:22, 4.80s/it] 45%|████▍ | 1671/3723 [2:12:02<2:44:19, 4.80s/it] 45%|████▍ | 1672/3723 [2:12:07<2:44:14, 4.80s/it] 45%|████▍ | 1673/3723 [2:12:12<2:44:13, 4.81s/it] 45%|████▍ | 1674/3723 [2:12:16<2:44:24, 4.81s/it] 45%|████▍ | 1675/3723 [2:12:21<2:44:44, 4.83s/it] 45%|████▌ | 1676/3723 [2:12:26<2:45:48, 4.86s/it] 45%|████▌ | 1677/3723 [2:12:31<2:44:58, 4.84s/it] 45%|████▌ | 1678/3723 [2:12:36<2:45:18, 4.85s/it] 45%|████▌ | 1679/3723 [2:12:41<2:45:39, 4.86s/it] 45%|████▌ | 1680/3723 [2:12:46<2:46:12, 4.88s/it] 45%|████▌ | 1680/3723 [2:12:46<2:46:12, 4.88s/it] 45%|████▌ | 1681/3723 [2:12:51<2:47:02, 4.91s 0: {'loss': 0.6998, 'grad_norm': 0.910136797764842, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.27} 0: /it] 45%|████▌ | 1682/3723 [2:12:55<2:46:36, 4.90s/it] 45%|████▌ | 1683/3723 [2:13:00<2:45:49, 4.88s/it] 45%|████▌ | 1684/3723 [2:13:05<2:45:12, 4.86s/it] 45%|████▌ | 1685/3723 [2:13:10<2:44:53, 4.85s/it] 45%|████▌ | 1686/3723 [2:13:15<2:44:41, 4.85s/it] 45%|████▌ | 1687/3723 [2:13:20<2:44:33, 4.85s/it] 45%|████▌ | 1688/3723 [2:13:24<2:43:27, 4.82s/it] 45%|████▌ | 1689/3723 [2:13:29<2:42:31, 4.79s/it] 45%|████▌ | 1690/3723 [2:13:34<2:42:10, 4.79s/it] 45%|████▌ | 1690/3723 [2:13:34<2:42:10, 4.79s/it] 45%|████▌ | 1691/3723 [2:13:39<2:42:03, 4.79s/it] 45%|████▌ | 1692/3723 [2:13:43<2:41:23, 4.77s/it] 45%|████▌ | 1693/3723 [2:13:48<2:41:00, 4.76s/it] 46%|████▌ | 1694/3723 [2:13:53<2:42:03, 4.79s/it] 46%|████▌ | 1695/3723 [ 0: {'loss': 0.7107, 'grad_norm': 0.8239373558787471, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.27} 0: 2:13:58<2:42:28, 4.81s/it] 46%|████▌ | 1696/3723 [2:14:03<2:42:44, 4.82s/it] 46%|████▌ | 1697/3723 [2:14:08<2:42:49, 4.82s/it] 46%|████▌ | 1698/3723 [2:14:12<2:43:06, 4.83s/it] 46%|████▌ | 1699/3723 [2:14:17<2:43:12, 4.84s/it] 46%|████▌ | 1700/3723 [2:14:22<2:42:38, 4.82s/it] 46%|████▌ | 1700/3723 [2:14:22<2:42:38, 4.82s/it] 46%|████▌ | 1701/3723 [2:14:27<2:44:04, 4.87s/it] 46%|████▌ | 1702/3723 [2:14:32<2:45:46, 4.92s/it] 46%|████▌ | 1703/3723 [2:14:37<2:46:05, 4.93s/it] 46%|████▌ | 1704/3723 [2:14:42<2:46:23, 4.94s/it] 46%|████▌ | 1705/3723 [2:14:47<2:46:39, 4.95s/it] 46%|████▌ | 1706/3723 [2:14:52<2:46:45, 4.96s/it] 46%|████▌ | 1707/3723 [2:14:57<2:46:49, 4.97s/it] 46%|████▌ | 1708/3723 [2:15:02<2:46:41, 4.96s/it] 46%|███ 0: {'loss': 0.7117, 'grad_norm': 1.0459522684853702, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.28} 0: {'loss': 0.7096, 'grad_norm': 0.9247761945647065, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.28} 0: ▌ | 1709/3723 [2:15:07<2:46:44, 4.97s/it] 46%|████▌ | 1710/3723 [2:15:12<2:46:40, 4.97s/it] 46%|████▌ | 1710/3723 [2:15:12<2:46:40, 4.97s/it] 46%|████▌ | 1711/3723 [2:15:17<2:44:52, 4.92s/it] 46%|████▌ | 1712/3723 [2:15:21<2:41:44, 4.83s/it] 46%|████▌ | 1713/3723 [2:15:26<2:39:24, 4.76s/it] 46%|████▌ | 1714/3723 [2:15:31<2:40:31, 4.79s/it] 46%|████▌ | 1715/3723 [2:15:35<2:39:19, 4.76s/it] 46%|████▌ | 1716/3723 [2:15:40<2:38:33, 4.74s/it] 46%|████▌ | 1717/3723 [2:15:45<2:38:06, 4.73s/it] 46%|████▌ | 1718/3723 [2:15:50<2:37:44, 4.72s/it] 46%|████▌ | 1719/3723 [2:15:54<2:37:33, 4.72s/it] 46%|████▌ | 1720/3723 [2:15:59<2:37:27, 4.72s/it] 46%|████▌ | 1720/3723 [2:15:59<2:37:27, 4.72s/it] 0: {'loss': 0.707, 'grad_norm': 0.878414990350605, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.28} 0: 46%|████▌ | 1721/3723 [2:16:04<2:37:24, 4.72s/it] 46%|████▋ | 1722/3723 [2:16:08<2:36:58, 4.71s/it] 46%|████▋ | 1723/3723 [2:16:13<2:36:30, 4.70s/it] 46%|████▋ | 1724/3723 [2:16:18<2:36:27, 4.70s/it] 46%|████▋ | 1725/3723 [2:16:22<2:36:54, 4.71s/it] 46%|████▋ | 1726/3723 [2:16:27<2:37:21, 4.73s/it] 46%|████▋ | 1727/3723 [2:16:32<2:37:42, 4.74s/it] 46%|████▋ | 1728/3723 [2:16:37<2:37:33, 4.74s/it] 46%|████▋ | 1729/3723 [2:16:41<2:37:03, 4.73s/it] 46%|████▋ | 1730/3723 [2:16:46<2:36:51, 4.72s/it] 46%|████▋ | 1730/3723 [2:16:46<2:36:51, 4.72s/it] 46%|████▋ | 1731/3723 [2:16:51<2:37:06, 4.73s/it] 47%|████▋ | 1732/3723 [2:16:56<2:36:59, 4.73s/it] 47%|████▋ | 1733/3723 [2:17:00<2:37:02, 4.73s/it] 47%|████▋ | 1734/3723 [2:17:0 0: {'loss': 0.7128, 'grad_norm': 0.942728269994045, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.28} 0: 5<2:37:16, 4.74s/it] 47%|████▋ | 1735/3723 [2:17:10<2:37:19, 4.75s/it] 47%|████▋ | 1736/3723 [2:17:15<2:37:20, 4.75s/it] 47%|████▋ | 1737/3723 [2:17:19<2:37:16, 4.75s/it] 47%|████▋ | 1738/3723 [2:17:24<2:39:08, 4.81s/it] 47%|████▋ | 1739/3723 [2:17:29<2:38:04, 4.78s/it] 47%|████▋ | 1740/3723 [2:17:34<2:37:25, 4.76s/it] 47%|████▋ | 1740/3723 [2:17:34<2:37:25, 4.76s/it] 47%|████▋ | 1741/3723 [2:17:39<2:37:16, 4.76s/it] 47%|████▋ | 1742/3723 [2:17:43<2:36:56, 4.75s/it] 47%|████▋ | 1743/3723 [2:17:48<2:36:49, 4.75s/it] 47%|████▋ | 1744/3723 [2:17:53<2:36:30, 4.74s/it] 47%|████▋ | 1745/3723 [2:17:58<2:36:42, 4.75s/it] 47%|████▋ | 1746/3723 [2:18:02<2:36:56, 4.76s/it] 47%|████▋ | 1747/3723 [2:18:07<2:36:49, 4.76s/it] 47%|████▋ 0: {'loss': 0.6927, 'grad_norm': 0.8647441703529836, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.28} 0: {'loss': 0.7098, 'grad_norm': 1.039195611368304, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.28} 0: | 1748/3723 [2:18:12<2:36:46, 4.76s/it] 47%|████▋ | 1749/3723 [2:18:17<2:36:10, 4.75s/it] 47%|████▋ | 1750/3723 [2:18:21<2:35:37, 4.73s/it] 47%|████▋ | 1750/3723 [2:18:21<2:35:37, 4.73s/it] 47%|████▋ | 1751/3723 [2:18:26<2:35:22, 4.73s/it] 47%|████▋ | 1752/3723 [2:18:31<2:34:49, 4.71s/it] 47%|████▋ | 1753/3723 [2:18:35<2:34:16, 4.70s/it] 47%|████▋ | 1754/3723 [2:18:40<2:33:44, 4.68s/it] 47%|████▋ | 1755/3723 [2:18:45<2:33:28, 4.68s/it] 47%|████▋ | 1756/3723 [2:18:49<2:33:48, 4.69s/it] 47%|████▋ | 1757/3723 [2:18:54<2:33:49, 4.69s/it] 47%|████▋ | 1758/3723 [2:18:59<2:34:11, 4.71s/it] 47%|████▋ | 1759/3723 [2:19:04<2:34:44, 4.73s/it] 47%|████▋ | 1760/3723 [2:19:08<2:34:57, 4.74s/it] 47%| 0: {'loss': 0.7147, 'grad_norm': 0.8638904062246574, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.29} 0: ███▋ | 1760/3723 [2:19:08<2:34:57, 4.74s/it] 47%|████▋ | 1761/3723 [2:19:13<2:34:49, 4.73s/it] 47%|████▋ | 1762/3723 [2:19:18<2:34:29, 4.73s/it] 47%|████▋ | 1763/3723 [2:19:22<2:34:13, 4.72s/it] 47%|████▋ | 1764/3723 [2:19:27<2:33:58, 4.72s/it] 47%|████▋ | 1765/3723 [2:19:32<2:35:16, 4.76s/it] 47%|████▋ | 1766/3723 [2:19:37<2:36:46, 4.81s/it] 47%|████▋ | 1767/3723 [2:19:42<2:37:03, 4.82s/it] 47%|████▋ | 1768/3723 [2:19:47<2:36:53, 4.82s/it] 48%|████▊ | 1769/3723 [2:19:51<2:36:42, 4.81s/it] 48%|████▊ | 1770/3723 [2:19:56<2:36:25, 4.81s/it] 48%|████▊ | 1770/3723 [2:19:56<2:36:25, 4.81s/it] 48%|████▊ | 1771/3723 [2:20:01<2:36:43, 4.82s/it] 48%|████▊ | 1772/3723 [2:20:06<2:35:29, 4.78s/it] 48%|████▊ | 1773/3723 [2:20:10<2:34 0: {'loss': 0.6896, 'grad_norm': 0.8495508888372795, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.29} 0: :16, 4.75s/it] 48%|████▊ | 1774/3723 [2:20:15<2:33:30, 4.73s/it] 48%|████▊ | 1775/3723 [2:20:20<2:33:04, 4.71s/it] 48%|████▊ | 1776/3723 [2:20:24<2:32:43, 4.71s/it] 48%|████▊ | 1777/3723 [2:20:29<2:31:59, 4.69s/it] 48%|████▊ | 1778/3723 [2:20:34<2:31:30, 4.67s/it] 48%|████▊ | 1779/3723 [2:20:38<2:30:57, 4.66s/it] 48%|████▊ | 1780/3723 [2:20:43<2:30:47, 4.66s/it] 48%|████▊ | 1780/3723 [2:20:43<2:30:47, 4.66s/it] 48%|████▊ | 1781/3723 [2:20:48<2:33:14, 4.73s/it] 48%|████▊ | 1782/3723 [2:20:53<2:34:12, 4.77s/it] 48%|████▊ | 1783/3723 [2:20:58<2:33:59, 4.76s/it] 48%|████▊ | 1784/3723 [2:21:02<2:34:33, 4.78s/it] 48%|████▊ | 1785/3723 [2:21:07<2:35:27, 4.81s/it] 48%|████▊ | 1786/3723 [2:21:12<2:35:36, 4.82s/it] 48%|████▊ | 0: {'loss': 0.6965, 'grad_norm': 0.8612632631532747, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.29} 0: 1787/3723 [2:21:17<2:35:17, 4.81s/it] 48%|████▊ | 1788/3723 [2:21:22<2:34:56, 4.80s/it] 48%|████▊ | 1789/3723 [2:21:26<2:34:40, 4.80s/it] 48%|████▊ | 1790/3723 [2:21:31<2:34:33, 4.80s/it] 48%|████▊ | 1790/3723 [2:21:31<2:34:33, 4.80s/it] 48%|████▊ | 1791/3723 [2:21:36<2:33:54, 4.78s/it] 48%|████▊ | 1792/3723 [2:21:41<2:32:58, 4.75s/it] 48%|████▊ | 1793/3723 [2:21:45<2:32:39, 4.75s/it] 48%|████▊ | 1794/3723 [2:21:50<2:32:24, 4.74s/it] 48%|████▊ | 1795/3723 [2:21:55<2:32:13, 4.74s/it] 48%|████▊ | 1796/3723 [2:22:00<2:32:02, 4.73s/it] 48%|████▊ | 1797/3723 [2:22:04<2:32:07, 4.74s/it] 48%|████▊ | 1798/3723 [2:22:09<2:32:05, 4.74s/it] 48%|████▊ | 1799/3723 [2:22:14<2:32:21, 4.75s/it] 48%|████▊ | 1800/3723 [2:22:19<2:32:37, 4.76s/it] 0: {'loss': 0.717, 'grad_norm': 0.9444077366958208, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.29} 0: {'loss': 0.7129, 'grad_norm': 0.898267930948978, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.29} 0: 48%|████▊ | 1800/3723 [2:22:19<2:32:37, 4.76s/it] 48%|████▊ | 1801/3723 [2:22:23<2:33:25, 4.79s/it] 48%|████▊ | 1802/3723 [2:22:28<2:33:25, 4.79s/it] 48%|████▊ | 1803/3723 [2:22:33<2:33:15, 4.79s/it] 48%|████▊ | 1804/3723 [2:22:38<2:32:42, 4.77s/it] 48%|████▊ | 1805/3723 [2:22:42<2:31:41, 4.75s/it] 49%|████▊ | 1806/3723 [2:22:47<2:31:00, 4.73s/it] 49%|████▊ | 1807/3723 [2:22:52<2:30:24, 4.71s/it] 49%|████▊ | 1808/3723 [2:22:57<2:30:11, 4.71s/it] 49%|████▊ | 1809/3723 [2:23:01<2:30:00, 4.70s/it] 49%|████▊ | 1810/3723 [2:23:06<2:29:59, 4.70s/it] 49%|████▊ | 1810/3723 [2:23:06<2:29:59, 4.70s/it] 49%|████▊ | 1811/3723 [2:23:11<2:29:53, 4.70s/it] 49%|████▊ | 1812/3723 [2:23:15<2:30:03, 0: {'loss': 0.7092, 'grad_norm': 0.9045382306649526, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.29} 0: 4.71s/it] 49%|████▊ | 1813/3723 [2:23:20<2:30:01, 4.71s/it] 49%|████▊ | 1814/3723 [2:23:25<2:30:07, 4.72s/it] 49%|████▉ | 1815/3723 [2:23:30<2:30:59, 4.75s/it] 49%|████▉ | 1816/3723 [2:23:34<2:31:32, 4.77s/it] 49%|████▉ | 1817/3723 [2:23:39<2:31:09, 4.76s/it] 49%|████▉ | 1818/3723 [2:23:44<2:31:37, 4.78s/it] 49%|████▉ | 1819/3723 [2:23:49<2:31:59, 4.79s/it] 49%|████▉ | 1820/3723 [2:23:54<2:32:21, 4.80s/it] 49%|████▉ | 1820/3723 [2:23:54<2:32:21, 4.80s/it] 49%|████▉ | 1821/3723 [2:23:58<2:32:10, 4.80s/it] 49%|████▉ | 1822/3723 [2:24:03<2:32:25, 4.81s/it] 49%|████▉ | 1823/3723 [2:24:08<2:32:25, 4.81s/it] 49%|████▉ | 1824/3723 [2:24:13<2:31:29, 4.79s/it] 49%|████▉ | 1825/3723 [2:24:18<2:30:49, 4.77s/it] 49%|████▉ | 1826/3 0: {'loss': 0.6953, 'grad_norm': 1.0116463774605875, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.29} 0: 723 [2:24:22<2:30:18, 4.75s/it] 49%|████▉ | 1827/3723 [2:24:27<2:29:45, 4.74s/it] 49%|████▉ | 1828/3723 [2:24:32<2:29:22, 4.73s/it] 49%|████▉ | 1829/3723 [2:24:36<2:29:04, 4.72s/it] 49%|████▉ | 1830/3723 [2:24:41<2:28:45, 4.72s/it] 49%|████▉ | 1830/3723 [2:24:41<2:28:45, 4.72s/it] 49%|████▉ | 1831/3723 [2:24:46<2:28:34, 4.71s/it] 49%|████▉ | 1832/3723 [2:24:50<2:28:23, 4.71s/it] 49%|████▉ | 1833/3723 [2:24:55<2:28:20, 4.71s/it] 49%|████▉ | 1834/3723 [2:25:00<2:28:24, 4.71s/it] 49%|████▉ | 1835/3723 [2:25:05<2:28:01, 4.70s/it] 49%|████▉ | 1836/3723 [2:25:09<2:27:39, 4.69s/it] 49%|████▉ | 1837/3723 [2:25:14<2:27:34, 4.69s/it] 49%|████▉ | 1838/3723 [2:25:19<2:27:32, 4.70s/it] 49%|████▉ | 1839/3723 [2:25:23<2:27:31, 4.70s/it] 49%|█ 0: {'loss': 0.7164, 'grad_norm': 0.8424925208821779, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.3} 0: {'loss': 0.7012, 'grad_norm': 0.8565304610693067, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.3} 0: ██▉ | 1840/3723 [2:25:28<2:27:31, 4.70s/it] 49%|████▉ | 1840/3723 [2:25:28<2:27:31, 4.70s/it] 49%|████▉ | 1841/3723 [2:25:33<2:27:31, 4.70s/it] 49%|████▉ | 1842/3723 [2:25:37<2:27:28, 4.70s/it] 50%|████▉ | 1843/3723 [2:25:42<2:27:33, 4.71s/it] 50%|████▉ | 1844/3723 [2:25:47<2:27:41, 4.72s/it] 50%|████▉ | 1845/3723 [2:25:52<2:27:43, 4.72s/it] 50%|████▉ | 1846/3723 [2:25:56<2:27:38, 4.72s/it] 50%|████▉ | 1847/3723 [2:26:01<2:27:37, 4.72s/it] 50%|████▉ | 1848/3723 [2:26:06<2:27:36, 4.72s/it] 50%|████▉ | 1849/3723 [2:26:11<2:27:34, 4.72s/it] 50%|████▉ | 1850/3723 [2:26:15<2:27:27, 4.72s/it] 50%|████▉ | 1850/3723 [2:26:15<2:27:27, 4.72s/it] 50%|████▉ | 1851/3723 [2:26:20<2:27:35, 4.73s/ 0: {'loss': 0.7034, 'grad_norm': 0.9000405923587387, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.3} 0: it] 50%|████▉ | 1852/3723 [2:26:25<2:27:26, 4.73s/it] 50%|████▉ | 1853/3723 [2:26:30<2:27:22, 4.73s/it] 50%|████▉ | 1854/3723 [2:26:34<2:27:13, 4.73s/it] 50%|████▉ | 1855/3723 [2:26:39<2:27:04, 4.72s/it] 50%|████▉ | 1856/3723 [2:26:44<2:26:55, 4.72s/it] 50%|████▉ | 1857/3723 [2:26:48<2:26:45, 4.72s/it] 50%|████▉ | 1858/3723 [2:26:53<2:26:28, 4.71s/it] 50%|████▉ | 1859/3723 [2:26:58<2:26:17, 4.71s/it] 50%|████▉ | 1860/3723 [2:27:03<2:26:39, 4.72s/it] 50%|████▉ | 1860/3723 [2:27:03<2:26:39, 4.72s/it] 50%|████▉ | 1861/3723 [2:27:07<2:27:15, 4.75s/it] 50%|█████ | 1862/3723 [2:27:12<2:27:29, 4.76s/it] 50%|█████ | 1863/3723 [2:27:17<2:27:37, 4.76s/it] 50%|█████ | 1864/3723 [2:27:22<2:27:35, 4.76s/it] 50%|█████ | 1865/3723 [2 0: {'loss': 0.6985, 'grad_norm': 0.8991624511736482, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.3} 0: :27:26<2:27:38, 4.77s/it] 50%|█████ | 1866/3723 [2:27:31<2:27:46, 4.77s/it] 50%|█████ | 1867/3723 [2:27:36<2:27:51, 4.78s/it] 50%|█████ | 1868/3723 [2:27:41<2:27:36, 4.77s/it] 50%|█████ | 1869/3723 [2:27:45<2:26:46, 4.75s/it] 50%|█████ | 1870/3723 [2:27:50<2:26:17, 4.74s/it] 50%|█████ | 1870/3723 [2:27:50<2:26:17, 4.74s/it] 50%|█████ | 1871/3723 [2:27:55<2:26:14, 4.74s/it] 50%|█████ | 1872/3723 [2:28:00<2:26:13, 4.74s/it] 50%|█████ | 1873/3723 [2:28:04<2:25:57, 4.73s/it] 50%|█████ | 1874/3723 [2:28:09<2:26:00, 4.74s/it] 50%|█████ | 1875/3723 [2:28:14<2:25:34, 4.73s/it] 50%|█████ | 1876/3723 [2:28:19<2:25:33, 4.73s/it] 50%|█████ | 1877/3723 [2:28:23<2:25:20, 4.72s/it] 50%|█████ | 1878/3723 [2:28:28<2:25:11, 4.72s/it] 50%|███ 0: {'loss': 0.7162, 'grad_norm': 0.9101681677115901, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.3} 0: {'loss': 0.6772, 'grad_norm': 0.8371508075492988, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.3} 0: █ | 1879/3723 [2:28:33<2:24:20, 4.70s/it] 50%|█████ | 1880/3723 [2:28:37<2:23:31, 4.67s/it] 50%|█████ | 1880/3723 [2:28:37<2:23:31, 4.67s/it] 51%|█████ | 1881/3723 [2:28:42<2:23:03, 4.66s/it] 51%|█████ | 1882/3723 [2:28:46<2:22:42, 4.65s/it] 51%|█████ | 1883/3723 [2:28:51<2:22:08, 4.63s/it] 51%|█████ | 1884/3723 [2:28:56<2:21:46, 4.63s/it] 51%|█████ | 1885/3723 [2:29:00<2:21:53, 4.63s/it] 51%|█████ | 1886/3723 [2:29:05<2:22:05, 4.64s/it] 51%|█████ | 1887/3723 [2:29:10<2:22:15, 4.65s/it] 51%|█████ | 1888/3723 [2:29:14<2:22:09, 4.65s/it] 51%|█████ | 1889/3723 [2:29:19<2:21:46, 4.64s/it] 51%|█████ | 1890/3723 [2:29:24<2:22:30, 4.66s/it] 51%|█████ | 1890/3723 [2:29:24<2:22:30, 4.66s/it] 5 0: {'loss': 0.704, 'grad_norm': 0.9954843904120542, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.31} 0: 1%|█████ | 1891/3723 [2:29:28<2:22:36, 4.67s/it] 51%|█████ | 1892/3723 [2:29:33<2:22:28, 4.67s/it] 51%|█████ | 1893/3723 [2:29:38<2:22:06, 4.66s/it] 51%|█████ | 1894/3723 [2:29:42<2:22:03, 4.66s/it] 51%|█████ | 1895/3723 [2:29:47<2:22:11, 4.67s/it] 51%|█████ | 1896/3723 [2:29:52<2:21:57, 4.66s/it] 51%|█████ | 1897/3723 [2:29:56<2:22:03, 4.67s/it] 51%|█████ | 1898/3723 [2:30:01<2:22:16, 4.68s/it] 51%|█████ | 1899/3723 [2:30:06<2:22:35, 4.69s/it] 51%|█████ | 1900/3723 [2:30:10<2:22:37, 4.69s/it] 51%|█████ | 1900/3723 [2:30:10<2:22:37, 4.69s/it] 51%|█████ | 1901/3723 [2:30:15<2:22:38, 4.70s/it] 51%|█████ | 1902/3723 [2:30:20<2:22:17, 4.69s/it] 51%|█████ | 1903/3723 [2:30:25<2:22:12, 4.69s/it] 51%|█████ | 1904/3723 [2:30:29 0: {'loss': 0.7009, 'grad_norm': 0.8460916799939532, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.31} 0: <2:21:56, 4.68s/it] 51%|█████ | 1905/3723 [2:30:34<2:22:02, 4.69s/it] 51%|█████ | 1906/3723 [2:30:39<2:22:18, 4.70s/it] 51%|█████ | 1907/3723 [2:30:43<2:22:26, 4.71s/it] 51%|█████ | 1908/3723 [2:30:48<2:22:54, 4.72s/it] 51%|█████▏ | 1909/3723 [2:30:53<2:22:51, 4.73s/it] 51%|█████▏ | 1910/3723 [2:30:57<2:22:06, 4.70s/it] 51%|█████▏ | 1910/3723 [2:30:57<2:22:06, 4.70s/it] 51%|█████▏ | 1911/3723 [2:31:02<2:21:51, 4.70s/it] 51%|█████▏ | 1912/3723 [2:31:07<2:21:29, 4.69s/it] 51%|█████▏ | 1913/3723 [2:31:11<2:21:05, 4.68s/it] 51%|█████▏ | 1914/3723 [2:31:16<2:20:55, 4.67s/it] 51%|█████▏ | 1915/3723 [2:31:21<2:20:48, 4.67s/it] 51%|█████▏ | 1916/3723 [2:31:25<2:20:44, 4.67s/it] 51%|█████▏ | 1917/3723 [2:31:30<2:20:44, 4.68s/it] 5 0: {'loss': 0.6817, 'grad_norm': 0.9024216078752956, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.31} 0: 2%|█████▏ | 1918/3723 [2:31:35<2:20:34, 4.67s/it] 52%|█████▏ | 1919/3723 [2:31:39<2:20:20, 4.67s/it] 52%|█████▏ | 1920/3723 [2:31:44<2:20:18, 4.67s/it] 52%|█████▏ | 1920/3723 [2:31:44<2:20:18, 4.67s/it] 52%|█████▏ | 1921/3723 [2:31:49<2:20:26, 4.68s/it] 52%|█████▏ | 1922/3723 [2:31:54<2:20:10, 4.67s/it] 52%|█████▏ | 1923/3723 [2:31:58<2:20:05, 4.67s/it] 52%|█████▏ | 1924/3723 [2:32:03<2:20:03, 4.67s/it] 52%|█████▏ | 1925/3723 [2:32:08<2:20:33, 4.69s/it] 52%|█████▏ | 1926/3723 [2:32:12<2:20:56, 4.71s/it] 52%|█████▏ | 1927/3723 [2:32:17<2:21:12, 4.72s/it] 52%|█████▏ | 1928/3723 [2:32:22<2:21:20, 4.72s/it] 52%|█████▏ | 1929/3723 [2:32:27<2:21:24, 4.73s/it] 52%|█████▏ | 1930/3723 [2:32:31<2:21:46, 4.74s/it] 0: {'loss': 0.6887, 'grad_norm': 0.8090997436145096, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.31} 0: {'loss': 0.6987, 'grad_norm': 0.8621058879871942, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.31} 0: 52%|█████▏ | 1930/3723 [2:32:31<2:21:46, 4.74s/it] 52%|█████▏ | 1931/3723 [2:32:36<2:22:10, 4.76s/it] 52%|█████▏ | 1932/3723 [2:32:41<2:21:58, 4.76s/it] 52%|█████▏ | 1933/3723 [2:32:46<2:21:38, 4.75s/it] 52%|█████▏ | 1934/3723 [2:32:50<2:20:56, 4.73s/it] 52%|█████▏ | 1935/3723 [2:32:55<2:21:02, 4.73s/it] 52%|█████▏ | 1936/3723 [2:33:00<2:20:59, 4.73s/it] 52%|█████▏ | 1937/3723 [2:33:05<2:21:00, 4.74s/it] 52%|█████▏ | 1938/3723 [2:33:09<2:20:57, 4.74s/it] 52%|█████▏ | 1939/3723 [2:33:14<2:20:45, 4.73s/it] 52%|█████▏ | 1940/3723 [2:33:19<2:21:06, 4.75s/it] 52%|█████▏ | 1940/3723 [2:33:19<2:21:06, 4.75s/it] 52%|█████▏ | 1941/3723 [2:33:24<2:21:20, 4.76s/it] 52%|█████▏ | 1942/3723 [2:3 0: {'loss': 0.6861, 'grad_norm': 0.8824148225928011, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.31} 0: 3:28<2:21:25, 4.76s/it] 52%|█████▏ | 1943/3723 [2:33:33<2:21:30, 4.77s/it] 52%|█████▏ | 1944/3723 [2:33:38<2:21:27, 4.77s/it] 52%|█████▏ | 1945/3723 [2:33:43<2:21:25, 4.77s/it] 52%|█████▏ | 1946/3723 [2:33:47<2:21:15, 4.77s/it] 52%|█████▏ | 1947/3723 [2:33:52<2:21:46, 4.79s/it] 52%|█████▏ | 1948/3723 [2:33:57<2:21:32, 4.78s/it] 52%|█████▏ | 1949/3723 [2:34:02<2:21:16, 4.78s/it] 52%|█████▏ | 1950/3723 [2:34:07<2:21:03, 4.77s/it] 52%|█████▏ | 1950/3723 [2:34:07<2:21:03, 4.77s/it] 52%|█████▏ | 1951/3723 [2:34:11<2:21:10, 4.78s/it] 52%|█████▏ | 1952/3723 [2:34:16<2:21:02, 4.78s/it] 52%|█████▏ | 1953/3723 [2:34:21<2:21:05, 4.78s/it] 52%|█████▏ | 1954/3723 [2:34:26<2:21:05, 4.79s/it] 53%|█████▎ | 1955/3723 [2:34:30<2:21:00, 0: {'loss': 0.7125, 'grad_norm': 0.8614530216974285, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.32} 0: 4.79s/it] 53%|█████▎ | 1956/3723 [2:34:35<2:20:49, 4.78s/it] 53%|█████▎ | 1957/3723 [2:34:40<2:20:39, 4.78s/it] 53%|█████▎ | 1958/3723 [2:34:45<2:20:27, 4.77s/it] 53%|█████▎ | 1959/3723 [2:34:50<2:20:13, 4.77s/it] 53%|█████▎ | 1960/3723 [2:34:54<2:19:53, 4.76s/it] 53%|█████▎ | 1960/3723 [2:34:54<2:19:53, 4.76s/it] 53%|█████▎ | 1961/3723 [2:34:59<2:19:55, 4.76s/it] 53%|█████▎ | 1962/3723 [2:35:04<2:19:46, 4.76s/it] 53%|█████▎ | 1963/3723 [2:35:09<2:19:02, 4.74s/it] 53%|█████▎ | 1964/3723 [2:35:13<2:18:11, 4.71s/it] 53%|█████▎ | 1965/3723 [2:35:18<2:17:36, 4.70s/it] 53%|█████▎ | 1966/3723 [2:35:22<2:17:15, 4.69s/it] 53%|█████▎ | 1967/3723 [2:35:27<2:16:52, 4.68s/it] 53%|█████▎ | 1968/3723 [2:35:32<2:16:35, 4.67s/it] 53%| 0: {'loss': 0.7009, 'grad_norm': 0.9720336342931799, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.32} 0: {'loss': 0.7013, 'grad_norm': 15.48700551482027, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.32} 0: █████▎ | 1969/3723 [2:35:36<2:16:18, 4.66s/it] 53%|█████▎ | 1970/3723 [2:35:41<2:16:19, 4.67s/it] 53%|█████▎ | 1970/3723 [2:35:41<2:16:19, 4.67s/it] 53%|█████▎ | 1971/3723 [2:35:46<2:16:34, 4.68s/it] 53%|█████▎ | 1972/3723 [2:35:50<2:16:24, 4.67s/it] 53%|█████▎ | 1973/3723 [2:35:55<2:16:10, 4.67s/it] 53%|█████▎ | 1974/3723 [2:36:00<2:15:57, 4.66s/it] 53%|█████▎ | 1975/3723 [2:36:04<2:15:51, 4.66s/it] 53%|█████▎ | 1976/3723 [2:36:09<2:15:42, 4.66s/it] 53%|█████▎ | 1977/3723 [2:36:14<2:15:29, 4.66s/it] 53%|█████▎ | 1978/3723 [2:36:18<2:15:24, 4.66s/it] 53%|█████▎ | 1979/3723 [2:36:23<2:17:35, 4.73s/it] 53%|█████▎ | 1980/3723 [2:36:28<2:18:58, 4.78s/it] 53%|█████▎ | 19 0: {'loss': 0.6928, 'grad_norm': 1.0400429141242709, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.32} 0: 80/3723 [2:36:28<2:18:58, 4.78s/it] 53%|█████▎ | 1981/3723 [2:36:34<2:24:22, 4.97s/it] 53%|█████▎ | 1982/3723 [2:36:39<2:23:34, 4.95s/it] 53%|█████▎ | 1983/3723 [2:36:43<2:22:13, 4.90s/it] 53%|█████▎ | 1984/3723 [2:36:48<2:20:08, 4.84s/it] 53%|█████▎ | 1985/3723 [2:36:53<2:18:34, 4.78s/it] 53%|█████▎ | 1986/3723 [2:36:57<2:17:24, 4.75s/it] 53%|█████▎ | 1987/3723 [2:37:02<2:16:25, 4.72s/it] 53%|█████▎ | 1988/3723 [2:37:07<2:16:03, 4.71s/it] 53%|█████▎ | 1989/3723 [2:37:11<2:15:35, 4.69s/it] 53%|█████▎ | 1990/3723 [2:37:16<2:15:35, 4.69s/it] 53%|█████▎ | 1990/3723 [2:37:16<2:15:35, 4.69s/it] 53%|█████▎ | 1991/3723 [2:37:21<2:15:44, 4.70s/it] 54%|█████▎ | 1992/3723 [2:37:25<2:15:47, 4.71s/it] 54%|█████▎ | 1993/3723 [2:37:3 0: {'loss': 0.7213, 'grad_norm': 0.8920295507282286, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.32} 0: 0<2:15:26, 4.70s/it] 54%|█████▎ | 1994/3723 [2:37:35<2:15:03, 4.69s/it] 54%|█████▎ | 1995/3723 [2:37:39<2:14:57, 4.69s/it] 54%|█████▎ | 1996/3723 [2:37:44<2:14:39, 4.68s/it] 54%|█████▎ | 1997/3723 [2:37:49<2:14:17, 4.67s/it] 54%|█████▎ | 1998/3723 [2:37:53<2:14:20, 4.67s/it] 54%|█████▎ | 1999/3723 [2:37:58<2:14:10, 4.67s/it] 54%|█████▎ | 2000/3723 [2:38:03<2:13:55, 4.66s/it] 54%|█████▎ | 2000/3723 [2:38:03<2:13:55, 4.66s/it] 54%|█████▎ | 2001/3723 [2:38:07<2:13:47, 4.66s/it] 54%|█████▍ | 2002/3723 [2:38:12<2:13:45, 4.66s/it] 54%|█████▍ | 2003/3723 [2:38:17<2:13:59, 4.67s/it] 54%|█████▍ | 2004/3723 [2:38:21<2:14:06, 4.68s/it] 54%|█████▍ | 2005/3723 [2:38:26<2:14:09, 4.69s/it] 54%|█████▍ | 2006/3723 [2:38:31<2:14:23, 4.7 0: {'loss': 0.6892, 'grad_norm': 0.8107153403962339, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.32} 0: 0s/it] 54%|█████▍ | 2007/3723 [2:38:36<2:14:10, 4.69s/it] 54%|█████▍ | 2008/3723 [2:38:40<2:14:05, 4.69s/it] 54%|█████▍ | 2009/3723 [2:38:45<2:14:08, 4.70s/it] 54%|█████▍ | 2010/3723 [2:38:50<2:14:18, 4.70s/it] 54%|█████▍ | 2010/3723 [2:38:50<2:14:18, 4.70s/it] 54%|█████▍ | 2011/3723 [2:38:54<2:13:54, 4.69s/it] 54%|█████▍ | 2012/3723 [2:38:59<2:13:16, 4.67s/it] 54%|█████▍ | 2013/3723 [2:39:04<2:12:54, 4.66s/it] 54%|█████▍ | 2014/3723 [2:39:08<2:12:59, 4.67s/it] 54%|█████▍ | 2015/3723 [2:39:13<2:12:56, 4.67s/it] 54%|█████▍ | 2016/3723 [2:39:18<2:13:04, 4.68s/it] 54%|█████▍ | 2017/3723 [2:39:22<2:13:05, 4.68s/it] 54%|█████▍ | 2018/3723 [2:39:27<2:13:10, 4.69s/it] 54%|█████▍ | 2019/3723 [2:39:32<2:13:00, 4.68s/it] 54%|█ 0: {'loss': 0.6793, 'grad_norm': 0.8037503550491605, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.33} 0: {'loss': 0.6799, 'grad_norm': 0.8744054110689313, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.33} 0: ████▍ | 2020/3723 [2:39:36<2:13:05, 4.69s/it] 54%|█████▍ | 2020/3723 [2:39:36<2:13:05, 4.69s/it] 54%|█████▍ | 2021/3723 [2:39:41<2:13:20, 4.70s/it] 54%|█████▍ | 2022/3723 [2:39:46<2:13:13, 4.70s/it] 54%|█████▍ | 2023/3723 [2:39:51<2:13:33, 4.71s/it] 54%|█████▍ | 2024/3723 [2:39:55<2:13:21, 4.71s/it] 54%|█████▍ | 2025/3723 [2:40:00<2:13:20, 4.71s/it] 54%|█████▍ | 2026/3723 [2:40:05<2:13:44, 4.73s/it] 54%|█████▍ | 2027/3723 [2:40:10<2:14:09, 4.75s/it] 54%|█████▍ | 2028/3723 [2:40:14<2:14:22, 4.76s/it] 54%|█████▍ | 2029/3723 [2:40:19<2:14:10, 4.75s/it] 55%|█████▍ | 2030/3723 [2:40:24<2:13:52, 4.74s/it] 55%|█████▍ | 2030/3723 [2:40:24<2:13:52, 4.74s/it] 55%|█████▍ | 2031/ 0: {'loss': 0.6991, 'grad_norm': 1.099206320108073, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.33} 0: 3723 [2:40:29<2:13:35, 4.74s/it] 55%|█████▍ | 2032/3723 [2:40:33<2:13:06, 4.72s/it] 55%|█████▍ | 2033/3723 [2:40:38<2:12:44, 4.71s/it] 55%|█████▍ | 2034/3723 [2:40:43<2:12:18, 4.70s/it] 55%|█████▍ | 2035/3723 [2:40:47<2:12:08, 4.70s/it] 55%|█████▍ | 2036/3723 [2:40:52<2:11:55, 4.69s/it] 55%|█████▍ | 2037/3723 [2:40:57<2:11:39, 4.69s/it] 55%|█████▍ | 2038/3723 [2:41:01<2:11:29, 4.68s/it] 55%|█████▍ | 2039/3723 [2:41:06<2:11:14, 4.68s/it] 55%|█████▍ | 2040/3723 [2:41:11<2:11:02, 4.67s/it] 55%|█████▍ | 2040/3723 [2:41:11<2:11:02, 4.67s/it] 55%|█████▍ | 2041/3723 [2:41:15<2:10:55, 4.67s/it] 55%|█████▍ | 2042/3723 [2:41:20<2:10:41, 4.67s/it] 55%|█████▍ | 2043/3723 [2:41:25<2:10:34, 4.66s/it] 55%|█████▍ | 2044/3723 [2:41:29<2 0: {'loss': 0.6854, 'grad_norm': 0.9649939980478377, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.33} 0: :10:09, 4.65s/it] 55%|█████▍ | 2045/3723 [2:41:34<2:09:47, 4.64s/it] 55%|█████▍ | 2046/3723 [2:41:39<2:09:38, 4.64s/it] 55%|█████▍ | 2047/3723 [2:41:43<2:09:59, 4.65s/it] 55%|█████▌ | 2048/3723 [2:41:48<2:10:06, 4.66s/it] 55%|█████▌ | 2049/3723 [2:41:53<2:10:38, 4.68s/it] 55%|█████▌ | 2050/3723 [2:41:57<2:11:42, 4.72s/it] 55%|█████▌ | 2050/3723 [2:41:57<2:11:42, 4.72s/it] 55%|█████▌ | 2051/3723 [2:42:02<2:12:08, 4.74s/it] 55%|█████▌ | 2052/3723 [2:42:07<2:11:53, 4.74s/it] 55%|█████▌ | 2053/3723 [2:42:12<2:11:44, 4.73s/it] 55%|█████▌ | 2054/3723 [2:42:16<2:11:58, 4.74s/it] 55%|█████▌ | 2055/3723 [2:42:21<2:12:35, 4.77s/it] 55%|█████▌ | 2056/3723 [2:42:26<2:13:42, 4.81s/it] 55%|█████▌ | 2057/3723 [2:42:31<2:14:44, 4.85s/ 0: {'loss': 0.6984, 'grad_norm': 0.8595161364744403, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.33} 0: it] 55%|█████▌ | 2058/3723 [2:42:36<2:15:08, 4.87s/it] 55%|█████▌ | 2059/3723 [2:42:41<2:14:57, 4.87s/it] 55%|█████▌ | 2060/3723 [2:42:46<2:13:17, 4.81s/it] 55%|█████▌ | 2060/3723 [2:42:46<2:13:17, 4.81s/it] 55%|█████▌ | 2061/3723 [2:42:50<2:11:41, 4.75s/it] 55%|█████▌ | 2062/3723 [2:42:55<2:10:55, 4.73s/it] 55%|█████▌ | 2063/3723 [2:43:00<2:10:22, 4.71s/it] 55%|█████▌ | 2064/3723 [2:43:04<2:09:58, 4.70s/it] 55%|█████▌ | 2065/3723 [2:43:09<2:09:39, 4.69s/it] 55%|█████▌ | 2066/3723 [2:43:14<2:08:56, 4.67s/it] 56%|█████▌ | 2067/3723 [2:43:18<2:09:03, 4.68s/it] 56%|█████▌ | 2068/3723 [2:43:23<2:09:12, 4.68s/it] 56%|█████▌ | 2069/3723 [2:43:28<2:08:51, 4.67s/it] 56%|█████▌ | 2070/3723 [2:43:32<2:08:44, 4.67s/it] 0: {'loss': 0.6938, 'grad_norm': 0.8726662889263967, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.33} 0: {'loss': 0.6975, 'grad_norm': 0.959929461663017, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.34} 0: 56%|█████▌ | 2070/3723 [2:43:32<2:08:44, 4.67s/it] 56%|█████▌ | 2071/3723 [2:43:37<2:08:38, 4.67s/it] 56%|█████▌ | 2072/3723 [2:43:42<2:08:32, 4.67s/it] 56%|█████▌ | 2073/3723 [2:43:46<2:08:21, 4.67s/it] 56%|█████▌ | 2074/3723 [2:43:51<2:08:24, 4.67s/it] 56%|█████▌ | 2075/3723 [2:43:56<2:08:32, 4.68s/it] 56%|█████▌ | 2076/3723 [2:44:00<2:08:42, 4.69s/it] 56%|█████▌ | 2077/3723 [2:44:05<2:08:53, 4.70s/it] 56%|█████▌ | 2078/3723 [2:44:10<2:08:53, 4.70s/it] 56%|█████▌ | 2079/3723 [2:44:14<2:08:33, 4.69s/it] 56%|█████▌ | 2080/3723 [2:44:19<2:08:21, 4.69s/it] 56%|█████▌ | 2080/3723 [2:44:19<2:08:21, 4.69s/it] 56%|█████▌ | 2081/3723 [2:44:24<2:08:37, 4.70s/it] 56%|█████▌ | 2082/372 0: {'loss': 0.7092, 'grad_norm': 0.8925056688943644, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.34} 0: 3 [2:44:29<2:08:25, 4.70s/it] 56%|█████▌ | 2083/3723 [2:44:33<2:08:36, 4.71s/it] 56%|█████▌ | 2084/3723 [2:44:38<2:08:36, 4.71s/it] 56%|█████▌ | 2085/3723 [2:44:43<2:08:17, 4.70s/it] 56%|█████▌ | 2086/3723 [2:44:47<2:08:05, 4.69s/it] 56%|█████▌ | 2087/3723 [2:44:52<2:08:09, 4.70s/it] 56%|█████▌ | 2088/3723 [2:44:57<2:08:08, 4.70s/it] 56%|█████▌ | 2089/3723 [2:45:01<2:07:51, 4.70s/it] 56%|█████▌ | 2090/3723 [2:45:06<2:07:46, 4.69s/it] 56%|█████▌ | 2090/3723 [2:45:06<2:07:46, 4.69s/it] 56%|█████▌ | 2091/3723 [2:45:11<2:07:02, 4.67s/it] 56%|█████▌ | 2092/3723 [2:45:15<2:07:05, 4.68s/it] 56%|█████▌ | 2093/3723 [2:45:20<2:06:38, 4.66s/it] 56%|█████▌ | 2094/3723 [2:45:25<2:06:16, 4.65s/it] 56%|█████▋ | 2095/3723 [2:45:29<2:06 0: {'loss': 0.7029, 'grad_norm': 0.8518156038080456, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.34} 0: :16, 4.65s/it] 56%|█████▋ | 2096/3723 [2:45:34<2:05:59, 4.65s/it] 56%|█████▋ | 2097/3723 [2:45:39<2:05:44, 4.64s/it] 56%|█████▋ | 2098/3723 [2:45:43<2:05:52, 4.65s/it] 56%|█████▋ | 2099/3723 [2:45:48<2:05:43, 4.65s/it] 56%|█████▋ | 2100/3723 [2:45:53<2:05:34, 4.64s/it] 56%|█████▋ | 2100/3723 [2:45:53<2:05:34, 4.64s/it] 56%|█████▋ | 2101/3723 [2:45:57<2:06:04, 4.66s/it] 56%|█████▋ | 2102/3723 [2:46:02<2:06:24, 4.68s/it] 56%|█████▋ | 2103/3723 [2:46:07<2:06:39, 4.69s/it] 57%|█████▋ | 2104/3723 [2:46:11<2:06:38, 4.69s/it] 57%|█████▋ | 2105/3723 [2:46:16<2:06:42, 4.70s/it] 57%|█████▋ | 2106/3723 [2:46:21<2:06:43, 4.70s/it] 57%|█████▋ | 2107/3723 [2:46:25<2:06:37, 4.70s/it] 57%|█████▋ | 2108/3723 [2:46:30<2:06:35, 4.70s/it] 0: {'loss': 0.6852, 'grad_norm': 0.8688835541281418, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.34} 0: {'loss': 0.6756, 'grad_norm': 0.85620782884158, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.34} 0: 57%|█████▋ | 2109/3723 [2:46:35<2:06:30, 4.70s/it] 57%|█████▋ | 2110/3723 [2:46:40<2:06:22, 4.70s/it] 57%|█████▋ | 2110/3723 [2:46:40<2:06:22, 4.70s/it] 57%|█████▋ | 2111/3723 [2:46:44<2:05:51, 4.68s/it] 57%|█████▋ | 2112/3723 [2:46:49<2:06:16, 4.70s/it] 57%|█████▋ | 2113/3723 [2:46:54<2:06:01, 4.70s/it] 57%|█████▋ | 2114/3723 [2:46:58<2:06:05, 4.70s/it] 57%|█████▋ | 2115/3723 [2:47:03<2:06:14, 4.71s/it] 57%|█████▋ | 2116/3723 [2:47:08<2:06:14, 4.71s/it] 57%|█████▋ | 2117/3723 [2:47:12<2:05:40, 4.70s/it] 57%|█████▋ | 2118/3723 [2:47:17<2:05:49, 4.70s/it] 57%|█████▋ | 2119/3723 [2:47:22<2:05:31, 4.70s/it] 57%|█████▋ | 2120/3723 [2:47:27<2:05:24, 4.69s/it] 57%|█████▋ 0: {'loss': 0.6891, 'grad_norm': 0.8960652715241576, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.34} 0: | 2120/3723 [2:47:27<2:05:24, 4.69s/it] 57%|█████▋ | 2121/3723 [2:47:31<2:05:22, 4.70s/it] 57%|█████▋ | 2122/3723 [2:47:36<2:05:03, 4.69s/it] 57%|█████▋ | 2123/3723 [2:47:41<2:05:04, 4.69s/it] 57%|█████▋ | 2124/3723 [2:47:45<2:04:50, 4.68s/it] 57%|█████▋ | 2125/3723 [2:47:50<2:04:39, 4.68s/it] 57%|█████▋ | 2126/3723 [2:47:55<2:04:19, 4.67s/it] 57%|█████▋ | 2127/3723 [2:47:59<2:04:08, 4.67s/it] 57%|█████▋ | 2128/3723 [2:48:04<2:03:34, 4.65s/it] 57%|█████▋ | 2129/3723 [2:48:09<2:03:38, 4.65s/it] 57%|█████▋ | 2130/3723 [2:48:13<2:03:37, 4.66s/it] 57%|█████▋ | 2130/3723 [2:48:13<2:03:37, 4.66s/it] 57%|█████▋ | 2131/3723 [2:48:18<2:03:51, 4.67s/it] 57%|█████▋ | 2132/3723 [2:48:23<2:03:27, 4.66s/it] 57%|█████▋ | 2133/3723 [ 0: {'loss': 0.6827, 'grad_norm': 0.8818320137555536, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.34} 0: 2:48:27<2:03:58, 4.68s/it] 57%|█████▋ | 2134/3723 [2:48:32<2:04:26, 4.70s/it] 57%|█████▋ | 2135/3723 [2:48:37<2:03:59, 4.68s/it] 57%|█████▋ | 2136/3723 [2:48:41<2:03:37, 4.67s/it] 57%|█████▋ | 2137/3723 [2:48:46<2:03:38, 4.68s/it] 57%|█████▋ | 2138/3723 [2:48:51<2:03:32, 4.68s/it] 57%|█████▋ | 2139/3723 [2:48:55<2:03:08, 4.66s/it] 57%|█████▋ | 2140/3723 [2:49:00<2:02:48, 4.65s/it] 57%|█████▋ | 2140/3723 [2:49:00<2:02:48, 4.65s/it] 58%|█████▊ | 2141/3723 [2:49:05<2:02:34, 4.65s/it] 58%|█████▊ | 2142/3723 [2:49:09<2:02:14, 4.64s/it] 58%|█████▊ | 2143/3723 [2:49:14<2:01:52, 4.63s/it] 58%|█████▊ | 2144/3723 [2:49:18<2:01:15, 4.61s/it] 58%|█████▊ | 2145/3723 [2:49:23<2:01:28, 4.62s/it] 58%|█████▊ | 2146/3723 [2:49:28<2:01:32 0: {'loss': 0.7068, 'grad_norm': 0.8270979575389728, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.35} 0: , 4.62s/it] 58%|█████▊ | 2147/3723 [2:49:32<2:01:15, 4.62s/it] 58%|█████▊ | 2148/3723 [2:49:37<2:01:17, 4.62s/it] 58%|█████▊ | 2149/3723 [2:49:42<2:01:27, 4.63s/it] 58%|█████▊ | 2150/3723 [2:49:46<2:01:30, 4.63s/it] 58%|█████▊ | 2150/3723 [2:49:46<2:01:30, 4.63s/it] 58%|█████▊ | 2151/3723 [2:49:51<2:01:36, 4.64s/it] 58%|█████▊ | 2152/3723 [2:49:55<2:01:08, 4.63s/it] 58%|█████▊ | 2153/3723 [2:50:00<2:01:10, 4.63s/it] 58%|█████▊ | 2154/3723 [2:50:05<2:00:56, 4.62s/it] 58%|█████▊ | 2155/3723 [2:50:09<2:00:52, 4.63s/it] 58%|█████▊ | 2156/3723 [2:50:14<2:00:44, 4.62s/it] 58%|█████▊ | 2157/3723 [2:50:18<2:00:14, 4.61s/it] 58%|█████▊ | 2158/3723 [2:50:23<2:00:15, 4.61s/it] 58%|█████▊ | 2159/3723 [2:50:28<2:00:10, 4.61s/it] 5 0: {'loss': 0.6982, 'grad_norm': 0.9916935912026793, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.35} 0: {'loss': 0.7279, 'grad_norm': 0.9341983383530899, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.35} 0: 8%|█████▊ | 2160/3723 [2:50:32<1:59:41, 4.59s/it] 58%|█████▊ | 2160/3723 [2:50:32<1:59:41, 4.59s/it] 58%|█████▊ | 2161/3723 [2:50:37<1:59:24, 4.59s/it] 58%|█████▊ | 2162/3723 [2:50:41<1:59:13, 4.58s/it] 58%|█████▊ | 2163/3723 [2:50:46<1:59:00, 4.58s/it] 58%|█████▊ | 2164/3723 [2:50:51<1:59:05, 4.58s/it] 58%|█████▊ | 2165/3723 [2:50:55<1:59:10, 4.59s/it] 58%|█████▊ | 2166/3723 [2:51:00<1:58:52, 4.58s/it] 58%|█████▊ | 2167/3723 [2:51:04<1:58:57, 4.59s/it] 58%|█████▊ | 2168/3723 [2:51:09<1:59:17, 4.60s/it] 58%|█████▊ | 2169/3723 [2:51:14<2:00:39, 4.66s/it] 58%|█████▊ | 2170/3723 [2:51:18<1:59:48, 4.63s/it] 58%|█████▊ | 2170/3723 [2:51:18<1:59:48, 4.63s/it] 58%|█████▊ | 0: {'loss': 0.7013, 'grad_norm': 0.8715354673730344, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.35} 0: 2171/3723 [2:51:23<1:59:32, 4.62s/it] 58%|█████▊ | 2172/3723 [2:51:28<1:59:11, 4.61s/it] 58%|█████▊ | 2173/3723 [2:51:32<1:58:40, 4.59s/it] 58%|█████▊ | 2174/3723 [2:51:37<1:58:20, 4.58s/it] 58%|█████▊ | 2175/3723 [2:51:41<1:58:36, 4.60s/it] 58%|█████▊ | 2176/3723 [2:51:46<1:58:55, 4.61s/it] 58%|█████▊ | 2177/3723 [2:51:51<1:59:02, 4.62s/it] 59%|█████▊ | 2178/3723 [2:51:55<1:59:11, 4.63s/it] 59%|█████▊ | 2179/3723 [2:52:00<1:59:19, 4.64s/it] 59%|█████▊ | 2180/3723 [2:52:04<1:59:21, 4.64s/it] 59%|█████▊ | 2180/3723 [2:52:05<1:59:21, 4.64s/it] 59%|█████▊ | 2181/3723 [2:52:09<1:59:32, 4.65s/it] 59%|█████▊ | 2182/3723 [2:52:14<1:59:22, 4.65s/it] 59%|█████▊ | 2183/3723 [2:52:18<1:58:58, 4.64s/it] 59%|█████▊ | 2184/3723 [2:5 0: {'loss': 0.6985, 'grad_norm': 0.9761071660407511, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.35} 0: 2:23<1:59:46, 4.67s/it] 59%|█████▊ | 2185/3723 [2:52:28<2:00:13, 4.69s/it] 59%|█████▊ | 2186/3723 [2:52:33<2:00:14, 4.69s/it] 59%|█████▊ | 2187/3723 [2:52:37<2:00:02, 4.69s/it] 59%|█████▉ | 2188/3723 [2:52:42<1:59:37, 4.68s/it] 59%|█████▉ | 2189/3723 [2:52:47<1:59:00, 4.66s/it] 59%|█████▉ | 2190/3723 [2:52:51<1:58:24, 4.63s/it] 59%|█████▉ | 2190/3723 [2:52:51<1:58:24, 4.63s/it] 59%|█████▉ | 2191/3723 [2:52:56<1:57:54, 4.62s/it] 59%|█████▉ | 2192/3723 [2:53:00<1:57:27, 4.60s/it] 59%|█████▉ | 2193/3723 [2:53:05<1:57:12, 4.60s/it] 59%|█████▉ | 2194/3723 [2:53:09<1:57:16, 4.60s/it] 59%|█████▉ | 2195/3723 [2:53:14<1:57:16, 4.60s/it] 59%|█████▉ | 2196/3723 [2:53:19<1:56:52, 4.59s/it] 59%|█████▉ | 2197/3723 [2:53:23<1:56:36, 0: {'loss': 0.7022, 'grad_norm': 1.0369554400442582, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.35} 0: 4.59s/it] 59%|█████▉ | 2198/3723 [2:53:28<1:56:32, 4.59s/it] 59%|█████▉ | 2199/3723 [2:53:32<1:56:30, 4.59s/it] 59%|█████▉ | 2200/3723 [2:53:37<1:56:25, 4.59s/it] 59%|█████▉ | 2200/3723 [2:53:37<1:56:25, 4.59s/it] 59%|█████▉ | 2201/3723 [2:53:42<1:56:29, 4.59s/it] 59%|█████▉ | 2202/3723 [2:53:46<1:56:39, 4.60s/it] 59%|█████▉ | 2203/3723 [2:53:51<1:56:51, 4.61s/it] 59%|█████▉ | 2204/3723 [2:53:55<1:57:02, 4.62s/it] 59%|█████▉ | 2205/3723 [2:54:00<1:56:58, 4.62s/it] 59%|█████▉ | 2206/3723 [2:54:05<1:56:44, 4.62s/it] 59%|█████▉ | 2207/3723 [2:54:09<1:56:35, 4.61s/it] 59%|█████▉ | 2208/3723 [2:54:14<1:57:03, 4.64s/it] 59%|█████▉ | 2209/3723 [2:54:19<1:57:35, 4.66s/it] 59%|█████▉ | 2210/3723 [2:54:23<1:57:44, 4.67s/it] 0: {'loss': 0.6812, 'grad_norm': 0.8443782792459097, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.36} 0: {'loss': 0.6975, 'grad_norm': 0.9801562077784317, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.36} 0: 59%|█████▉ | 2210/3723 [2:54:23<1:57:44, 4.67s/it] 59%|█████▉ | 2211/3723 [2:54:28<1:57:41, 4.67s/it] 59%|█████▉ | 2212/3723 [2:54:33<1:57:19, 4.66s/it] 59%|█████▉ | 2213/3723 [2:54:37<1:57:02, 4.65s/it] 59%|█████▉ | 2214/3723 [2:54:42<1:56:46, 4.64s/it] 59%|█████▉ | 2215/3723 [2:54:47<1:56:44, 4.65s/it] 60%|█████▉ | 2216/3723 [2:54:51<1:56:42, 4.65s/it] 60%|█████▉ | 2217/3723 [2:54:56<1:56:53, 4.66s/it] 60%|█████▉ | 2218/3723 [2:55:01<1:57:04, 4.67s/it] 60%|█████▉ | 2219/3723 [2:55:05<1:57:05, 4.67s/it] 60%|█████▉ | 2220/3723 [2:55:10<1:57:10, 4.68s/it] 60%|█████▉ | 2220/3723 [2:55:10<1:57:10, 4.68s/it] 60%|█████▉ | 2221/3723 [2:55:15<1:57:17, 4.69s/it] 60%|█████▉ | 22 0: {'loss': 0.6999, 'grad_norm': 0.9024510286743859, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.36} 0: 22/3723 [2:55:19<1:57:11, 4.68s/it] 60%|█████▉ | 2223/3723 [2:55:24<1:56:57, 4.68s/it] 60%|█████▉ | 2224/3723 [2:55:29<1:56:54, 4.68s/it] 60%|█████▉ | 2225/3723 [2:55:33<1:56:50, 4.68s/it] 60%|█████▉ | 2226/3723 [2:55:38<1:56:42, 4.68s/it] 60%|█████▉ | 2227/3723 [2:55:43<1:56:44, 4.68s/it] 60%|█████▉ | 2228/3723 [2:55:47<1:56:42, 4.68s/it] 60%|█████▉ | 2229/3723 [2:55:52<1:56:21, 4.67s/it] 60%|█████▉ | 2230/3723 [2:55:57<1:56:18, 4.67s/it] 60%|█████▉ | 2230/3723 [2:55:57<1:56:18, 4.67s/it] 60%|█████▉ | 2231/3723 [2:56:02<1:56:24, 4.68s/it] 60%|█████▉ | 2232/3723 [2:56:06<1:56:07, 4.67s/it] 60%|█████▉ | 2233/3723 [2:56:11<1:56:08, 4.68s/it] 60%|██████ | 2234/3723 [2:56:15<1:55:49, 4.67s/it] 60%|██████ | 2235/3723 [2:56:2 0: {'loss': 0.7113, 'grad_norm': 0.8248395578197629, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.36} 0: 0<1:55:58, 4.68s/it] 60%|██████ | 2236/3723 [2:56:25<1:56:09, 4.69s/it] 60%|██████ | 2237/3723 [2:56:30<1:56:23, 4.70s/it] 60%|██████ | 2238/3723 [2:56:34<1:56:20, 4.70s/it] 60%|██████ | 2239/3723 [2:56:39<1:56:24, 4.71s/it] 60%|██████ | 2240/3723 [2:56:44<1:56:11, 4.70s/it] 60%|██████ | 2240/3723 [2:56:44<1:56:11, 4.70s/it] 60%|██████ | 2241/3723 [2:56:49<1:56:50, 4.73s/it] 60%|██████ | 2242/3723 [2:56:53<1:57:23, 4.76s/it] 60%|██████ | 2243/3723 [2:56:58<1:57:32, 4.77s/it] 60%|██████ | 2244/3723 [2:57:03<1:57:24, 4.76s/it] 60%|██████ | 2245/3723 [2:57:08<1:57:20, 4.76s/it] 60%|██████ | 2246/3723 [2:57:12<1:57:27, 4.77s/it] 60%|██████ | 2247/3723 [2:57:17<1:57:26, 4.77s/it] 60%|██████ | 2248/3723 [2:57:22<1:57:19, 4.7 0: {'loss': 0.7122, 'grad_norm': 14.267432887667706, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.36} 0: {'loss': 0.7026, 'grad_norm': 0.8951593482914049, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.36} 0: 7s/it] 60%|██████ | 2249/3723 [2:57:27<1:57:08, 4.77s/it] 60%|██████ | 2250/3723 [2:57:32<1:57:03, 4.77s/it] 60%|██████ | 2250/3723 [2:57:32<1:57:03, 4.77s/it] 60%|██████ | 2251/3723 [2:57:36<1:57:09, 4.78s/it] 60%|██████ | 2252/3723 [2:57:41<1:57:01, 4.77s/it] 61%|██████ | 2253/3723 [2:57:46<1:57:00, 4.78s/it] 61%|██████ | 2254/3723 [2:57:51<1:57:03, 4.78s/it] 61%|██████ | 2255/3723 [2:57:55<1:57:10, 4.79s/it] 61%|██████ | 2256/3723 [2:58:00<1:56:55, 4.78s/it] 61%|██████ | 2257/3723 [2:58:05<1:57:01, 4.79s/it] 61%|██████ | 2258/3723 [2:58:10<1:57:06, 4.80s/it] 61%|██████ | 2259/3723 [2:58:15<1:57:18, 4.81s/it] 61%|██████ | 2260/3723 [2:58:20<1:57:27, 4.82s/it] 61%|████ 0: {'loss': 0.6851, 'grad_norm': 0.8258456551427481, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.37} 0: █ | 2260/3723 [2:58:20<1:57:27, 4.82s/it] 61%|██████ | 2261/3723 [2:58:24<1:57:45, 4.83s/it] 61%|██████ | 2262/3723 [2:58:29<1:57:43, 4.83s/it] 61%|██████ | 2263/3723 [2:58:34<1:57:28, 4.83s/it] 61%|██████ | 2264/3723 [2:58:39<1:57:11, 4.82s/it] 61%|██████ | 2265/3723 [2:58:44<1:57:04, 4.82s/it] 61%|██████ | 2266/3723 [2:58:49<1:57:11, 4.83s/it] 61%|██████ | 2267/3723 [2:58:53<1:56:56, 4.82s/it] 61%|██████ | 2268/3723 [2:58:58<1:56:34, 4.81s/it] 61%|██████ | 2269/3723 [2:59:03<1:56:08, 4.79s/it] 61%|██████ | 2270/3723 [2:59:08<1:56:08, 4.80s/it] 61%|██████ | 2270/3723 [2:59:08<1:56:08, 4.80s/it] 61%|██████ | 2271/3723 [2:59:12<1:55:34, 4.78s/it] 61%|██████ | 2272/3723 [2:59:17<1:54:38, 4.74s/it] 61%|██████ | 2273/ 0: {'loss': 0.6939, 'grad_norm': 0.787053235065429, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.37} 0: 3723 [2:59:22<1:54:00, 4.72s/it] 61%|██████ | 2274/3723 [2:59:26<1:53:42, 4.71s/it] 61%|██████ | 2275/3723 [2:59:31<1:53:25, 4.70s/it] 61%|██████ | 2276/3723 [2:59:36<1:53:05, 4.69s/it] 61%|██████ | 2277/3723 [2:59:40<1:53:22, 4.70s/it] 61%|██████ | 2278/3723 [2:59:45<1:53:51, 4.73s/it] 61%|██████ | 2279/3723 [2:59:50<1:53:58, 4.74s/it] 61%|██████ | 2280/3723 [2:59:55<1:53:39, 4.73s/it] 61%|██████ | 2280/3723 [2:59:55<1:53:39, 4.73s/it] 61%|██████▏ | 2281/3723 [2:59:59<1:53:30, 4.72s/it] 61%|██████▏ | 2282/3723 [3:00:04<1:53:14, 4.71s/it] 61%|██████▏ | 2283/3723 [3:00:09<1:53:02, 4.71s/it] 61%|██████▏ | 2284/3723 [3:00:14<1:52:39, 4.70s/it] 61%|██████▏ | 2285/3723 [3:00:18<1:52:21, 4.69s/it] 61%|██████▏ | 2286/372 0: {'loss': 0.7033, 'grad_norm': 0.909689173266198, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.37} 0: 3 [3:00:23<1:52:07, 4.68s/it] 61%|██████▏ | 2287/3723 [3:00:27<1:51:46, 4.67s/it] 61%|██████▏ | 2288/3723 [3:00:32<1:51:40, 4.67s/it] 61%|██████▏ | 2289/3723 [3:00:37<1:51:31, 4.67s/it] 62%|██████▏ | 2290/3723 [3:00:41<1:51:21, 4.66s/it] 62%|██████▏ | 2290/3723 [3:00:41<1:51:21, 4.66s/it] 62%|██████▏ | 2291/3723 [3:00:46<1:51:11, 4.66s/it] 62%|██████▏ | 2292/3723 [3:00:51<1:50:57, 4.65s/it] 62%|██████▏ | 2293/3723 [3:00:55<1:50:47, 4.65s/it] 62%|██████▏ | 2294/3723 [3:01:00<1:50:51, 4.65s/it] 62%|██████▏ | 2295/3723 [3:01:05<1:50:58, 4.66s/it] 62%|██████▏ | 2296/3723 [3:01:09<1:50:47, 4.66s/it] 62%|██████▏ | 2297/3723 [3:01:14<1:50:10, 4.64s/it] 62%|██████▏ | 2298/3723 [3:01:19<1:49:28, 4.61s/it] 62%|██████▏ 0: {'loss': 0.7087, 'grad_norm': 0.8648479564015132, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.37} 0: {'loss': 0.6972, 'grad_norm': 0.7689760063329657, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.37} 0: | 2299/3723 [3:01:23<1:49:21, 4.61s/it] 62%|██████▏ | 2300/3723 [3:01:28<1:49:02, 4.60s/it] 62%|██████▏ | 2300/3723 [3:01:28<1:49:02, 4.60s/it] 62%|██████▏ | 2301/3723 [3:01:32<1:49:10, 4.61s/it] 62%|██████▏ | 2302/3723 [3:01:37<1:49:08, 4.61s/it] 62%|██████▏ | 2303/3723 [3:01:42<1:48:53, 4.60s/it] 62%|██████▏ | 2304/3723 [3:01:46<1:48:44, 4.60s/it] 62%|██████▏ | 2305/3723 [3:01:51<1:48:38, 4.60s/it] 62%|██████▏ | 2306/3723 [3:01:55<1:48:48, 4.61s/it] 62%|██████▏ | 2307/3723 [3:02:00<1:48:42, 4.61s/it] 62%|██████▏ | 2308/3723 [3:02:05<1:48:29, 4.60s/it] 62%|██████▏ | 2309/3723 [3:02:09<1:48:21, 4.60s/it] 62%|██████▏ | 2310/3723 [3:02:14<1:48:08, 4.59s/it] 62%|██████▏ 0: {'loss': 0.6944, 'grad_norm': 0.7522195455064583, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.37} 0: | 2310/3723 [3:02:14<1:48:08, 4.59s/it] 62%|██████▏ | 2311/3723 [3:02:18<1:48:02, 4.59s/it] 62%|██████▏ | 2312/3723 [3:02:23<1:47:50, 4.59s/it] 62%|██████▏ | 2313/3723 [3:02:27<1:47:35, 4.58s/it] 62%|██████▏ | 2314/3723 [3:02:32<1:47:25, 4.57s/it] 62%|██████▏ | 2315/3723 [3:02:37<1:47:16, 4.57s/it] 62%|██████▏ | 2316/3723 [3:02:41<1:47:01, 4.56s/it] 62%|██████▏ | 2317/3723 [3:02:46<1:47:13, 4.58s/it] 62%|██████▏ | 2318/3723 [3:02:50<1:47:24, 4.59s/it] 62%|██████▏ | 2319/3723 [3:02:55<1:47:23, 4.59s/it] 62%|██████▏ | 2320/3723 [3:03:00<1:47:32, 4.60s/it] 62%|██████▏ | 2320/3723 [3:03:00<1:47:32, 4.60s/it] 62%|██████▏ | 2321/3723 [3:03:04<1:47:25, 4.60s/it] 62%|██████▏ | 2322/3723 [3:03:09<1:47:10, 4.59s/it] 62%|██ 0: {'loss': 0.6698, 'grad_norm': 0.9077004858880311, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.38} 0: ███▏ | 2323/3723 [3:03:13<1:46:58, 4.58s/it] 62%|██████▏ | 2324/3723 [3:03:18<1:47:02, 4.59s/it] 62%|██████▏ | 2325/3723 [3:03:23<1:47:20, 4.61s/it] 62%|██████▏ | 2326/3723 [3:03:27<1:47:35, 4.62s/it] 63%|██████▎ | 2327/3723 [3:03:32<1:47:32, 4.62s/it] 63%|██████▎ | 2328/3723 [3:03:36<1:47:54, 4.64s/it] 63%|██████▎ | 2329/3723 [3:03:41<1:48:09, 4.66s/it] 63%|██████▎ | 2330/3723 [3:03:46<1:48:05, 4.66s/it] 63%|██████▎ | 2330/3723 [3:03:46<1:48:05, 4.66s/it] 63%|██████▎ | 2331/3723 [3:03:51<1:48:15, 4.67s/it] 63%|██████▎ | 2332/3723 [3:03:55<1:48:24, 4.68s/it] 63%|██████▎ | 2333/3723 [3:04:00<1:48:29, 4.68s/it] 63%|██████▎ | 2334/3723 [3:04:05<1:48:28, 4.69s/it] 63%|██████▎ | 2335/3723 [3:04:09<1:48:22, 4.68s/it] 0: {'loss': 0.6986, 'grad_norm': 0.8480237861464253, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.38} 0: 63%|██████▎ | 2336/3723 [3:04:14<1:48:17, 4.68s/it] 63%|██████▎ | 2337/3723 [3:04:19<1:48:04, 4.68s/it] 63%|██████▎ | 2338/3723 [3:04:23<1:47:54, 4.67s/it] 63%|██████▎ | 2339/3723 [3:04:28<1:47:50, 4.68s/it] 63%|██████▎ | 2340/3723 [3:04:33<1:47:43, 4.67s/it] 63%|██████▎ | 2340/3723 [3:04:33<1:47:43, 4.67s/it] 63%|██████▎ | 2341/3723 [3:04:37<1:47:47, 4.68s/it] 63%|██████▎ | 2342/3723 [3:04:42<1:47:36, 4.68s/it] 63%|██████▎ | 2343/3723 [3:04:47<1:47:05, 4.66s/it] 63%|██████▎ | 2344/3723 [3:04:51<1:46:37, 4.64s/it] 63%|██████▎ | 2345/3723 [3:04:56<1:46:18, 4.63s/it] 63%|██████▎ | 2346/3723 [3:05:00<1:46:20, 4.63s/it] 63%|██████▎ | 2347/3723 [3:05:05<1:46:10, 4.63s/it] 63%|██████▎ | 2348/3723 [3:05:10<1:46:05 0: {'loss': 0.7, 'grad_norm': 0.8861239464530123, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.38} 0: , 4.63s/it] 63%|██████▎ | 2349/3723 [3:05:14<1:45:59, 4.63s/it] 63%|██████▎ | 2350/3723 [3:05:19<1:46:02, 4.63s/it] 63%|██████▎ | 2350/3723 [3:05:19<1:46:02, 4.63s/it] 63%|██████▎ | 2351/3723 [3:05:24<1:46:25, 4.65s/it] 63%|██████▎ | 2352/3723 [3:05:28<1:46:15, 4.65s/it] 63%|██████▎ | 2353/3723 [3:05:33<1:46:15, 4.65s/it] 63%|██████▎ | 2354/3723 [3:05:38<1:46:04, 4.65s/it] 63%|██████▎ | 2355/3723 [3:05:42<1:46:09, 4.66s/it] 63%|██████▎ | 2356/3723 [3:05:47<1:46:21, 4.67s/it] 63%|██████▎ | 2357/3723 [3:05:52<1:46:18, 4.67s/it] 63%|██████▎ | 2358/3723 [3:05:56<1:45:45, 4.65s/it] 63%|██████▎ | 2359/3723 [3:06:01<1:45:31, 4.64s/it] 63%|██████▎ | 2360/3723 [3:06:06<1:45:16, 4.63s/it] 0: {'loss': 0.6866, 'grad_norm': 0.9679710386637859, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.38} 0: {'loss': 0.6702, 'grad_norm': 0.8205905112750752, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.38} 0: 63%|██████▎ | 2360/3723 [3:06:06<1:45:16, 4.63s/it] 63%|██████▎ | 2361/3723 [3:06:10<1:45:30, 4.65s/it] 63%|██████▎ | 2362/3723 [3:06:15<1:45:27, 4.65s/it] 63%|██████▎ | 2363/3723 [3:06:19<1:45:05, 4.64s/it] 63%|██████▎ | 2364/3723 [3:06:24<1:44:54, 4.63s/it] 64%|██████▎ | 2365/3723 [3:06:29<1:44:41, 4.63s/it] 64%|██████▎ | 2366/3723 [3:06:33<1:44:29, 4.62s/it] 64%|██████▎ | 2367/3723 [3:06:38<1:44:22, 4.62s/it] 64%|██████▎ | 2368/3723 [3:06:43<1:44:29, 4.63s/it] 64%|██████▎ | 2369/3723 [3:06:47<1:44:23, 4.63s/it] 64%|██████▎ | 2370/3723 [3:06:52<1:44:21, 4.63s/it] 64%|██████▎ | 2370/3723 [3:06:52<1:44:21, 4.63s/it] 64%|██████▎ | 2371/3723 [3:06:56<1:44:27, 4.64s/it] 64%|██████▎ | 2372/3723 [3: 0: {'loss': 0.6938, 'grad_norm': 0.9160336044945843, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.38} 0: 07:01<1:44:17, 4.63s/it] 64%|██████▎ | 2373/3723 [3:07:06<1:44:14, 4.63s/it] 64%|██████▍ | 2374/3723 [3:07:10<1:44:02, 4.63s/it] 64%|██████▍ | 2375/3723 [3:07:15<1:43:32, 4.61s/it] 64%|██████▍ | 2376/3723 [3:07:19<1:43:02, 4.59s/it] 64%|██████▍ | 2377/3723 [3:07:24<1:42:38, 4.58s/it] 64%|██████▍ | 2378/3723 [3:07:29<1:42:33, 4.58s/it] 64%|██████▍ | 2379/3723 [3:07:33<1:42:37, 4.58s/it] 64%|██████▍ | 2380/3723 [3:07:38<1:42:42, 4.59s/it] 64%|██████▍ | 2380/3723 [3:07:38<1:42:42, 4.59s/it] 64%|██████▍ | 2381/3723 [3:07:42<1:42:56, 4.60s/it] 64%|██████▍ | 2382/3723 [3:07:47<1:42:53, 4.60s/it] 64%|██████▍ | 2383/3723 [3:07:52<1:42:44, 4.60s/it] 64%|██████▍ | 2384/3723 [3:07:56<1:42:45, 4.60s/it] 64%|██████▍ | 0: {'loss': 0.682, 'grad_norm': 0.9293956443734185, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.39} 0: 2385/3723 [3:08:01<1:42:34, 4.60s/it] 64%|██████▍ | 2386/3723 [3:08:05<1:42:36, 4.60s/it] 64%|██████▍ | 2387/3723 [3:08:10<1:42:51, 4.62s/it] 64%|██████▍ | 2388/3723 [3:08:15<1:42:46, 4.62s/it] 64%|██████▍ | 2389/3723 [3:08:19<1:42:21, 4.60s/it] 64%|██████▍ | 2390/3723 [3:08:24<1:42:12, 4.60s/it] 64%|██████▍ | 2390/3723 [3:08:24<1:42:12, 4.60s/it] 64%|██████▍ | 2391/3723 [3:08:28<1:41:53, 4.59s/it] 64%|██████▍ | 2392/3723 [3:08:33<1:41:25, 4.57s/it] 64%|██████▍ | 2393/3723 [3:08:37<1:40:56, 4.55s/it] 64%|██████▍ | 2394/3723 [3:08:42<1:40:58, 4.56s/it] 64%|██████▍ | 2395/3723 [3:08:47<1:41:01, 4.56s/it] 64%|██████▍ | 2396/3723 [3:08:51<1:41:33, 4.59s/it] 64%|██████▍ | 2397/3723 [3:08:56<1:41:26, 4.59s/it] 64%|████ 0: {'loss': 0.6796, 'grad_norm': 0.8627059184536361, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.39} 0: █▍ | 2398/3723 [3:09:00<1:41:09, 4.58s/it] 64%|██████▍ | 2399/3723 [3:09:05<1:41:37, 4.61s/it] 64%|██████▍ | 2400/3723 [3:09:10<1:41:33, 4.61s/it] 64%|██████▍ | 2400/3723 [3:09:10<1:41:33, 4.61s/it] 64%|██████▍ | 2401/3723 [3:09:14<1:41:21, 4.60s/it] 65%|██████▍ | 2402/3723 [3:09:19<1:40:58, 4.59s/it] 65%|██████▍ | 2403/3723 [3:09:23<1:40:25, 4.56s/it] 65%|██████▍ | 2404/3723 [3:09:28<1:40:09, 4.56s/it] 65%|██████▍ | 2405/3723 [3:09:32<1:39:55, 4.55s/it] 65%|██████▍ | 2406/3723 [3:09:37<1:39:51, 4.55s/it] 65%|██████▍ | 2407/3723 [3:09:41<1:39:41, 4.54s/it] 65%|██████▍ | 2408/3723 [3:09:46<1:39:50, 4.56s/it] 65%|██████▍ | 2409/3723 [3:09:51<1:39:45, 4.56s/it] 65%|██████▍ | 2410/3723 [3:09:55<1:39:57, 4.57s/it] 0: {'loss': 0.6774, 'grad_norm': 0.9259697115752589, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.39} 0: {'loss': 0.6931, 'grad_norm': 0.8238678744391794, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.39} 0: 65%|██████▍ | 2410/3723 [3:09:55<1:39:57, 4.57s/it] 65%|██████▍ | 2411/3723 [3:10:00<1:40:31, 4.60s/it] 65%|██████▍ | 2412/3723 [3:10:05<1:40:46, 4.61s/it] 65%|██████▍ | 2413/3723 [3:10:09<1:40:26, 4.60s/it] 65%|██████▍ | 2414/3723 [3:10:14<1:40:10, 4.59s/it] 65%|██████▍ | 2415/3723 [3:10:18<1:39:58, 4.59s/it] 65%|██████▍ | 2416/3723 [3:10:23<1:39:48, 4.58s/it] 65%|██████▍ | 2417/3723 [3:10:27<1:39:15, 4.56s/it] 65%|██████▍ | 2418/3723 [3:10:32<1:38:49, 4.54s/it] 65%|██████▍ | 2419/3723 [3:10:36<1:38:41, 4.54s/it] 65%|██████▌ | 2420/3723 [3:10:41<1:38:37, 4.54s/it] 65%|██████▌ | 2420/3723 [3:10:41<1:38:37, 4.54s/it] 65%|██████▌ | 2421/3723 [3:10:45<1:38:46, 4.55s/it] 65%| 0: {'loss': 0.6824, 'grad_norm': 0.8428497339183546, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.39} 0: ██████▌ | 2422/3723 [3:10:50<1:38:51, 4.56s/it] 65%|██████▌ | 2423/3723 [3:10:55<1:38:52, 4.56s/it] 65%|██████▌ | 2424/3723 [3:10:59<1:39:01, 4.57s/it] 65%|██████▌ | 2425/3723 [3:11:04<1:39:04, 4.58s/it] 65%|██████▌ | 2426/3723 [3:11:08<1:39:06, 4.59s/it] 65%|██████▌ | 2427/3723 [3:11:13<1:39:06, 4.59s/it] 65%|██████▌ | 2428/3723 [3:11:18<1:39:19, 4.60s/it] 65%|██████▌ | 2429/3723 [3:11:22<1:39:34, 4.62s/it] 65%|██████▌ | 2430/3723 [3:11:27<1:39:35, 4.62s/it] 65%|██████▌ | 2430/3723 [3:11:27<1:39:35, 4.62s/it] 65%|██████▌ | 2431/3723 [3:11:32<1:39:28, 4.62s/it] 65%|██████▌ | 2432/3723 [3:11:36<1:39:12, 4.61s/it] 65%|██████▌ | 2433/3723 [3:11:41<1:38:49, 4.60s/it] 65%|██████▌ | 2434/3723 [3:11:45<1:38:24, 4. 0: {'loss': 0.6811, 'grad_norm': 0.8657170523292338, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.39} 0: 58s/it] 65%|██████▌ | 2435/3723 [3:11:50<1:38:13, 4.58s/it] 65%|██████▌ | 2436/3723 [3:11:54<1:38:09, 4.58s/it] 65%|██████▌ | 2437/3723 [3:11:59<1:38:08, 4.58s/it] 65%|██████▌ | 2438/3723 [3:12:04<1:38:02, 4.58s/it] 66%|██████▌ | 2439/3723 [3:12:08<1:38:01, 4.58s/it] 66%|██████▌ | 2440/3723 [3:12:13<1:37:58, 4.58s/it] 66%|██████▌ | 2440/3723 [3:12:13<1:37:58, 4.58s/it] 66%|██████▌ | 2441/3723 [3:12:17<1:38:03, 4.59s/it] 66%|██████▌ | 2442/3723 [3:12:22<1:38:01, 4.59s/it] 66%|██████▌ | 2443/3723 [3:12:27<1:38:04, 4.60s/it] 66%|██████▌ | 2444/3723 [3:12:31<1:37:54, 4.59s/it] 66%|██████▌ | 2445/3723 [3:12:36<1:38:15, 4.61s/it] 66%|██████▌ | 2446/3723 [3:12:40<1:38:07, 4.61s/it] 66%|██████▌ | 2447/3723 [3:12:45 0: {'loss': 0.6957, 'grad_norm': 0.8138489837594904, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.39} 0: <1:38:04, 4.61s/it] 66%|██████▌ | 2448/3723 [3:12:50<1:37:57, 4.61s/it] 66%|██████▌ | 2449/3723 [3:12:54<1:37:58, 4.61s/it] 66%|██████▌ | 2450/3723 [3:12:59<1:37:38, 4.60s/it] 66%|██████▌ | 2450/3723 [3:12:59<1:37:38, 4.60s/it] 66%|██████▌ | 2451/3723 [3:13:03<1:37:24, 4.59s/it] 66%|██████▌ | 2452/3723 [3:13:08<1:37:11, 4.59s/it] 66%|██████▌ | 2453/3723 [3:13:13<1:36:55, 4.58s/it] 66%|██████▌ | 2454/3723 [3:13:17<1:36:57, 4.58s/it] 66%|██████▌ | 2455/3723 [3:13:22<1:37:02, 4.59s/it] 66%|██████▌ | 2456/3723 [3:13:26<1:36:57, 4.59s/it] 66%|██████▌ | 2457/3723 [3:13:31<1:36:48, 4.59s/it] 66%|██████▌ | 2458/3723 [3:13:35<1:36:39, 4.58s/it] 66%|██████▌ | 2459/3723 [3:13:40<1:36:36, 4.59s/it] 66%|██████▌ | 2460/ 0: {'loss': 0.6975, 'grad_norm': 0.8204306973347165, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.4} 0: {'loss': 0.688, 'grad_norm': 0.8376387245083533, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.4} 0: 3723 [3:13:45<1:36:24, 4.58s/it] 66%|██████▌ | 2460/3723 [3:13:45<1:36:24, 4.58s/it] 66%|██████▌ | 2461/3723 [3:13:49<1:36:31, 4.59s/it] 66%|██████▌ | 2462/3723 [3:13:54<1:36:26, 4.59s/it] 66%|██████▌ | 2463/3723 [3:13:58<1:36:14, 4.58s/it] 66%|██████▌ | 2464/3723 [3:14:03<1:36:01, 4.58s/it] 66%|██████▌ | 2465/3723 [3:14:08<1:35:54, 4.57s/it] 66%|██████▌ | 2466/3723 [3:14:12<1:36:18, 4.60s/it] 66%|██████▋ | 2467/3723 [3:14:17<1:36:40, 4.62s/it] 66%|██████▋ | 2468/3723 [3:14:21<1:36:27, 4.61s/it] 66%|██████▋ | 2469/3723 [3:14:26<1:36:13, 4.60s/it] 66%|██████▋ | 2470/3723 [3:14:31<1:36:02, 4.60s/it] 66%|██████▋ | 2470/3723 [3:14:31<1:36:02, 4.60s/it] 66%|██████▋ | 2471/ 0: {'loss': 0.6878, 'grad_norm': 0.8718396539219293, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.4} 0: 3723 [3:14:35<1:36:05, 4.60s/it] 66%|██████▋ | 2472/3723 [3:14:40<1:36:06, 4.61s/it] 66%|██████▋ | 2473/3723 [3:14:44<1:36:05, 4.61s/it] 66%|██████▋ | 2474/3723 [3:14:49<1:35:51, 4.61s/it] 66%|██████▋ | 2475/3723 [3:14:54<1:35:51, 4.61s/it] 67%|██████▋ | 2476/3723 [3:14:58<1:35:57, 4.62s/it] 67%|██████▋ | 2477/3723 [3:15:03<1:36:01, 4.62s/it] 67%|██████▋ | 2478/3723 [3:15:08<1:36:26, 4.65s/it] 67%|██████▋ | 2479/3723 [3:15:12<1:36:47, 4.67s/it] 67%|██████▋ | 2480/3723 [3:15:17<1:36:59, 4.68s/it] 67%|██████▋ | 2480/3723 [3:15:17<1:36:59, 4.68s/it] 67%|██████▋ | 2481/3723 [3:15:22<1:37:15, 4.70s/it] 67%|██████▋ | 2482/3723 [3:15:27<1:37:10, 4.70s/it] 67%|██████▋ | 2483/3723 [3:15:31<1:38:20, 4.76s/it] 67%|██████ 0: {'loss': 0.6839, 'grad_norm': 1.0478534524868532, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.4} 0: ▋ | 2484/3723 [3:15:36<1:38:12, 4.76s/it] 67%|██████▋ | 2485/3723 [3:15:41<1:37:52, 4.74s/it] 67%|██████▋ | 2486/3723 [3:15:46<1:37:37, 4.74s/it] 67%|██████▋ | 2487/3723 [3:15:50<1:37:15, 4.72s/it] 67%|██████▋ | 2488/3723 [3:15:55<1:37:21, 4.73s/it] 67%|██████▋ | 2489/3723 [3:16:00<1:36:53, 4.71s/it] 67%|██████▋ | 2490/3723 [3:16:04<1:36:14, 4.68s/it] 67%|██████▋ | 2490/3723 [3:16:04<1:36:14, 4.68s/it] 67%|██████▋ | 2491/3723 [3:16:09<1:36:28, 4.70s/it] 67%|██████▋ | 2492/3723 [3:16:14<1:36:16, 4.69s/it] 67%|██████▋ | 2493/3723 [3:16:18<1:36:06, 4.69s/it] 67%|██████▋ | 2494/3723 [3:16:23<1:36:07, 4.69s/it] 67%|██████▋ | 2495/3723 [3:16:28<1:35:50, 4.68s/it] 67%|██████▋ | 2496/3723 [3:16:32<1:35:21, 4.66s/it] 67%|█ 0: {'loss': 0.6893, 'grad_norm': 0.8464919830175115, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.4} 0: ████▋ | 2497/3723 [3:16:37<1:34:56, 4.65s/it] 67%|██████▋ | 2498/3723 [3:16:42<1:34:47, 4.64s/it] 67%|██████▋ | 2499/3723 [3:16:46<1:34:44, 4.64s/it] 67%|██████▋ | 2500/3723 [3:16:51<1:34:25, 4.63s/it] 67%|██████▋ | 2500/3723 [3:16:51<1:34:25, 4.63s/it] 67%|██████▋ | 2501/3723 [3:16:56<1:34:46, 4.65s/it] 67%|██████▋ | 2502/3723 [3:17:00<1:34:56, 4.67s/it] 67%|██████▋ | 2503/3723 [3:17:05<1:35:04, 4.68s/it] 67%|██████▋ | 2504/3723 [3:17:10<1:35:18, 4.69s/it] 67%|██████▋ | 2505/3723 [3:17:14<1:35:19, 4.70s/it] 67%|██████▋ | 2506/3723 [3:17:19<1:35:13, 4.69s/it] 67%|██████▋ | 2507/3723 [3:17:24<1:35:09, 4.70s/it] 67%|██████▋ | 2508/3723 [3:17:29<1:35:05, 4.70s/it] 67%|██████▋ | 2509/3723 [3:17:33<1:34:52, 4.69s/i 0: {'loss': 0.6689, 'grad_norm': 0.8825757992713276, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.4} 0: {'loss': 0.6964, 'grad_norm': 0.7593074465039624, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.41} 0: t] 67%|██████▋ | 2510/3723 [3:17:38<1:34:48, 4.69s/it] 67%|██████▋ | 2510/3723 [3:17:38<1:34:48, 4.69s/it] 67%|██████▋ | 2511/3723 [3:17:43<1:34:39, 4.69s/it] 67%|██████▋ | 2512/3723 [3:17:47<1:34:10, 4.67s/it] 67%|██████▋ | 2513/3723 [3:17:52<1:33:42, 4.65s/it] 68%|██████▊ | 2514/3723 [3:17:56<1:33:20, 4.63s/it] 68%|██████▊ | 2515/3723 [3:18:01<1:33:10, 4.63s/it] 68%|██████▊ | 2516/3723 [3:18:06<1:33:09, 4.63s/it] 68%|██████▊ | 2517/3723 [3:18:10<1:32:58, 4.63s/it] 68%|██████▊ | 2518/3723 [3:18:15<1:32:35, 4.61s/it] 68%|██████▊ | 2519/3723 [3:18:19<1:32:27, 4.61s/it] 68%|██████▊ | 2520/3723 [3:18:24<1:32:09, 4.60s/it] 68%|██████▊ | 2520/3723 [3:18:24<1:32:09, 4.60s/i 0: {'loss': 0.6901, 'grad_norm': 0.7697755508728926, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.41} 0: t] 68%|██████▊ | 2521/3723 [3:18:29<1:32:07, 4.60s/it] 68%|██████▊ | 2522/3723 [3:18:33<1:33:39, 4.68s/it] 68%|██████▊ | 2523/3723 [3:18:38<1:33:12, 4.66s/it] 68%|██████▊ | 2524/3723 [3:18:43<1:32:56, 4.65s/it] 68%|██████▊ | 2525/3723 [3:18:47<1:32:44, 4.64s/it] 68%|██████▊ | 2526/3723 [3:18:52<1:32:29, 4.64s/it] 68%|██████▊ | 2527/3723 [3:18:57<1:32:19, 4.63s/it] 68%|██████▊ | 2528/3723 [3:19:01<1:32:18, 4.63s/it] 68%|██████▊ | 2529/3723 [3:19:06<1:32:32, 4.65s/it] 68%|██████▊ | 2530/3723 [3:19:10<1:32:11, 4.64s/it] 68%|██████▊ | 2530/3723 [3:19:11<1:32:11, 4.64s/it] 68%|██████▊ | 2531/3723 [3:19:15<1:31:46, 4.62s/it] 68%|██████▊ | 2532/3723 [3:19:20<1:31:18, 4.60s/it] 68%|██████▊ | 2533/3723 [3:19:24<1:31 0: {'loss': 0.6791, 'grad_norm': 0.8141290550583109, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.41} 0: :04, 4.59s/it] 68%|██████▊ | 2534/3723 [3:19:29<1:31:02, 4.59s/it] 68%|██████▊ | 2535/3723 [3:19:33<1:30:49, 4.59s/it] 68%|██████▊ | 2536/3723 [3:19:38<1:30:36, 4.58s/it] 68%|██████▊ | 2537/3723 [3:19:43<1:30:28, 4.58s/it] 68%|██████▊ | 2538/3723 [3:19:47<1:30:22, 4.58s/it] 68%|██████▊ | 2539/3723 [3:19:52<1:30:18, 4.58s/it] 68%|██████▊ | 2540/3723 [3:19:56<1:30:14, 4.58s/it] 68%|██████▊ | 2540/3723 [3:19:56<1:30:14, 4.58s/it] 68%|██████▊ | 2541/3723 [3:20:01<1:30:26, 4.59s/it] 68%|██████▊ | 2542/3723 [3:20:05<1:30:21, 4.59s/it] 68%|██████▊ | 2543/3723 [3:20:10<1:30:32, 4.60s/it] 68%|██████▊ | 2544/3723 [3:20:15<1:30:39, 4.61s/it] 68%|██████▊ | 2545/3723 [3:20:19<1:30:44, 4.62s/it] 68%|██████▊ | 2546/3723 0: {'loss': 0.6922, 'grad_norm': 0.8221954322901955, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.41} 0: [3:20:24<1:30:46, 4.63s/it] 68%|██████▊ | 2547/3723 [3:20:29<1:30:59, 4.64s/it] 68%|██████▊ | 2548/3723 [3:20:33<1:31:13, 4.66s/it] 68%|██████▊ | 2549/3723 [3:20:38<1:31:23, 4.67s/it] 68%|██████▊ | 2550/3723 [3:20:43<1:31:27, 4.68s/it] 68%|██████▊ | 2550/3723 [3:20:43<1:31:27, 4.68s/it] 69%|██████▊ | 2551/3723 [3:20:47<1:31:30, 4.68s/it] 69%|██████▊ | 2552/3723 [3:20:52<1:31:26, 4.69s/it] 69%|██████▊ | 2553/3723 [3:20:57<1:31:24, 4.69s/it] 69%|██████▊ | 2554/3723 [3:21:02<1:31:10, 4.68s/it] 69%|██████▊ | 2555/3723 [3:21:06<1:31:07, 4.68s/it] 69%|██████▊ | 2556/3723 [3:21:11<1:30:55, 4.67s/it] 69%|██████▊ | 2557/3723 [3:21:16<1:30:47, 4.67s/it] 69%|██████▊ | 2558/3723 [3:21:20<1:30:33, 4.66s/it] 69%|██████▊ 0: {'loss': 0.6991, 'grad_norm': 0.8457907182445586, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.41} 0: {'loss': 0.6832, 'grad_norm': 0.8640937829035212, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.41} 0: | 2559/3723 [3:21:25<1:30:08, 4.65s/it] 69%|██████▉ | 2560/3723 [3:21:30<1:31:04, 4.70s/it] 69%|██████▉ | 2560/3723 [3:21:30<1:31:04, 4.70s/it] 69%|██████▉ | 2561/3723 [3:21:34<1:30:34, 4.68s/it] 69%|██████▉ | 2562/3723 [3:21:39<1:30:15, 4.66s/it] 69%|██████▉ | 2563/3723 [3:21:43<1:30:01, 4.66s/it] 69%|██████▉ | 2564/3723 [3:21:48<1:29:49, 4.65s/it] 69%|██████▉ | 2565/3723 [3:21:53<1:29:36, 4.64s/it] 69%|██████▉ | 2566/3723 [3:21:57<1:29:18, 4.63s/it] 69%|██████▉ | 2567/3723 [3:22:02<1:29:05, 4.62s/it] 69%|██████▉ | 2568/3723 [3:22:07<1:28:58, 4.62s/it] 69%|██████▉ | 2569/3723 [3:22:11<1:28:43, 4.61s/it] 69%|██████▉ | 2570/3723 [3:22:16<1:28:36, 4.61s/it] 69%|██████▉ 0: {'loss': 0.6755, 'grad_norm': 0.8209590010806289, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.42} 0: | 2570/3723 [3:22:16<1:28:36, 4.61s/it] 69%|██████▉ | 2571/3723 [3:22:20<1:28:40, 4.62s/it] 69%|██████▉ | 2572/3723 [3:22:25<1:28:24, 4.61s/it] 69%|██████▉ | 2573/3723 [3:22:30<1:28:15, 4.60s/it] 69%|██████▉ | 2574/3723 [3:22:34<1:28:13, 4.61s/it] 69%|██████▉ | 2575/3723 [3:22:39<1:28:14, 4.61s/it] 69%|██████▉ | 2576/3723 [3:22:44<1:29:25, 4.68s/it] 69%|██████▉ | 2577/3723 [3:22:48<1:29:03, 4.66s/it] 69%|██████▉ | 2578/3723 [3:22:53<1:28:50, 4.66s/it] 69%|██████▉ | 2579/3723 [3:22:58<1:28:46, 4.66s/it] 69%|██████▉ | 2580/3723 [3:23:02<1:28:55, 4.67s/it] 69%|██████▉ | 2580/3723 [3:23:02<1:28:55, 4.67s/it] 69%|██████▉ | 2581/3723 [3:23:07<1:29:05, 4.68s/it] 69%|██████▉ | 2582/3723 [3:23:12<1:29:09, 4.69s/it] 69%|███ 0: {'loss': 0.6853, 'grad_norm': 0.8036255394300048, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.42} 0: ██▉ | 2583/3723 [3:23:16<1:29:15, 4.70s/it] 69%|██████▉ | 2584/3723 [3:23:21<1:29:16, 4.70s/it] 69%|██████▉ | 2585/3723 [3:23:26<1:29:07, 4.70s/it] 69%|██████▉ | 2586/3723 [3:23:31<1:29:10, 4.71s/it] 69%|██████▉ | 2587/3723 [3:23:35<1:30:08, 4.76s/it] 70%|██████▉ | 2588/3723 [3:23:40<1:30:47, 4.80s/it] 70%|██████▉ | 2589/3723 [3:23:45<1:30:11, 4.77s/it] 70%|██████▉ | 2590/3723 [3:23:50<1:29:38, 4.75s/it] 70%|██████▉ | 2590/3723 [3:23:50<1:29:38, 4.75s/it] 70%|██████▉ | 2591/3723 [3:23:54<1:29:26, 4.74s/it] 70%|██████▉ | 2592/3723 [3:23:59<1:28:59, 4.72s/it] 70%|██████▉ | 2593/3723 [3:24:04<1:28:40, 4.71s/it] 70%|██████▉ | 2594/3723 [3:24:09<1:30:55, 4.83s/it] 70%|██████▉ | 2595/3723 [3:24:14<1:32:13, 4.91s/it] 7 0: {'loss': 0.6857, 'grad_norm': 0.8908757395025214, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.42} 0: 0%|██████▉ | 2596/3723 [3:24:19<1:30:56, 4.84s/it] 70%|██████▉ | 2597/3723 [3:24:23<1:29:58, 4.79s/it] 70%|██████▉ | 2598/3723 [3:24:28<1:29:18, 4.76s/it] 70%|██████▉ | 2599/3723 [3:24:33<1:28:47, 4.74s/it] 70%|██████▉ | 2600/3723 [3:24:37<1:28:26, 4.73s/it] 70%|██████▉ | 2600/3723 [3:24:37<1:28:26, 4.73s/it] 70%|██████▉ | 2601/3723 [3:24:42<1:28:18, 4.72s/it] 70%|██████▉ | 2602/3723 [3:24:47<1:30:50, 4.86s/it] 70%|██████▉ | 2603/3723 [3:24:52<1:29:49, 4.81s/it] 70%|██████▉ | 2604/3723 [3:24:57<1:28:53, 4.77s/it] 70%|██████▉ | 2605/3723 [3:25:01<1:28:08, 4.73s/it] 70%|██████▉ | 2606/3723 [3:25:06<1:27:29, 4.70s/it] 70%|███████ | 2607/3723 [3:25:11<1:27:04, 4.68s/it] 70%|███████ | 2608/3723 [3:25:16<1:28:47, 0: {'loss': 0.6705, 'grad_norm': 0.7769668775652817, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.42} 0: 4.78s/it] 70%|███████ | 2609/3723 [3:25:20<1:28:05, 4.74s/it] 70%|███████ | 2610/3723 [3:25:25<1:27:32, 4.72s/it] 70%|███████ | 2610/3723 [3:25:25<1:27:32, 4.72s/it] 70%|███████ | 2611/3723 [3:25:30<1:27:09, 4.70s/it] 70%|███████ | 2612/3723 [3:25:34<1:26:38, 4.68s/it] 70%|███████ | 2613/3723 [3:25:39<1:26:18, 4.67s/it] 70%|███████ | 2614/3723 [3:25:44<1:26:09, 4.66s/it] 70%|███████ | 2615/3723 [3:25:48<1:25:53, 4.65s/it] 70%|███████ | 2616/3723 [3:25:53<1:25:35, 4.64s/it] 70%|███████ | 2617/3723 [3:25:57<1:25:26, 4.64s/it] 70%|███████ | 2618/3723 [3:26:02<1:25:23, 4.64s/it] 70%|███████ | 2619/3723 [3:26:07<1:25:28, 4.65s/it] 70%|███████ | 2620/3723 [3:26:11<1:25:23, 4.64s/it] 0: {'loss': 0.6903, 'grad_norm': 0.7992526756615119, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.42} 0: {'loss': 0.6808, 'grad_norm': 0.8590199077080489, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.42} 0: 70%|███████ | 2620/3723 [3:26:11<1:25:23, 4.64s/it] 70%|███████ | 2621/3723 [3:26:16<1:25:23, 4.65s/it] 70%|███████ | 2622/3723 [3:26:21<1:26:57, 4.74s/it] 70%|███████ | 2623/3723 [3:26:26<1:26:15, 4.70s/it] 70%|███████ | 2624/3723 [3:26:31<1:27:40, 4.79s/it] 71%|███████ | 2625/3723 [3:26:35<1:26:36, 4.73s/it] 71%|███████ | 2626/3723 [3:26:40<1:25:55, 4.70s/it] 71%|███████ | 2627/3723 [3:26:44<1:25:24, 4.68s/it] 71%|███████ | 2628/3723 [3:26:49<1:24:56, 4.65s/it] 71%|███████ | 2629/3723 [3:26:54<1:24:37, 4.64s/it] 71%|███████ | 2630/3723 [3:26:58<1:24:22, 4.63s/it] 71%|███████ | 2630/3723 [3:26:58<1:24:22, 4.63s/it] 71%|███████ | 2631/3723 [3:27:03<1:24:19, 4.63s/it] 71%|███████ | 2632/3723 [3:27 0: {'loss': 0.6842, 'grad_norm': 0.7886608234579771, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.43} 0: :07<1:24:12, 4.63s/it] 71%|███████ | 2633/3723 [3:27:12<1:23:57, 4.62s/it] 71%|███████ | 2634/3723 [3:27:17<1:23:40, 4.61s/it] 71%|███████ | 2635/3723 [3:27:21<1:23:34, 4.61s/it] 71%|███████ | 2636/3723 [3:27:26<1:23:27, 4.61s/it] 71%|███████ | 2637/3723 [3:27:30<1:23:14, 4.60s/it] 71%|███████ | 2638/3723 [3:27:35<1:23:05, 4.59s/it] 71%|███████ | 2639/3723 [3:27:40<1:22:58, 4.59s/it] 71%|███████ | 2640/3723 [3:27:44<1:22:49, 4.59s/it] 71%|███████ | 2640/3723 [3:27:44<1:22:49, 4.59s/it] 71%|███████ | 2641/3723 [3:27:49<1:22:46, 4.59s/it] 71%|███████ | 2642/3723 [3:27:53<1:22:55, 4.60s/it] 71%|███████ | 2643/3723 [3:27:58<1:23:09, 4.62s/it] 71%|███████ | 2644/3723 [3:28:03<1:23:20, 4.63s/it] 71%|███████ | 26 0: {'loss': 0.6835, 'grad_norm': 0.8676620114174309, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.43} 0: 45/3723 [3:28:07<1:23:18, 4.64s/it] 71%|███████ | 2646/3723 [3:28:12<1:23:14, 4.64s/it] 71%|███████ | 2647/3723 [3:28:17<1:23:09, 4.64s/it] 71%|███████ | 2648/3723 [3:28:21<1:23:07, 4.64s/it] 71%|███████ | 2649/3723 [3:28:26<1:23:08, 4.64s/it] 71%|███████ | 2650/3723 [3:28:31<1:23:01, 4.64s/it] 71%|███████ | 2650/3723 [3:28:31<1:23:01, 4.64s/it] 71%|███████ | 2651/3723 [3:28:35<1:22:58, 4.64s/it] 71%|███████ | 2652/3723 [3:28:40<1:22:56, 4.65s/it] 71%|███████▏ | 2653/3723 [3:28:45<1:22:55, 4.65s/it] 71%|███████▏ | 2654/3723 [3:28:49<1:22:44, 4.64s/it] 71%|███████▏ | 2655/3723 [3:28:54<1:22:36, 4.64s/it] 71%|███████▏ | 2656/3723 [3:28:58<1:22:34, 4.64s/it] 71%|███████▏ | 2657/3723 [3:29:03<1:22:31, 4.65s/it] 71%|█ 0: {'loss': 0.6847, 'grad_norm': 0.7998300413468162, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.43} 0: █████▏ | 2658/3723 [3:29:08<1:22:29, 4.65s/it] 71%|███████▏ | 2659/3723 [3:29:12<1:22:29, 4.65s/it] 71%|███████▏ | 2660/3723 [3:29:17<1:22:26, 4.65s/it] 71%|███████▏ | 2660/3723 [3:29:17<1:22:26, 4.65s/it] 71%|███████▏ | 2661/3723 [3:29:22<1:22:20, 4.65s/it] 72%|███████▏ | 2662/3723 [3:29:26<1:22:34, 4.67s/it] 72%|███████▏ | 2663/3723 [3:29:31<1:22:38, 4.68s/it] 72%|███████▏ | 2664/3723 [3:29:36<1:22:42, 4.69s/it] 72%|███████▏ | 2665/3723 [3:29:41<1:22:47, 4.70s/it] 72%|███████▏ | 2666/3723 [3:29:45<1:22:52, 4.70s/it] 72%|███████▏ | 2667/3723 [3:29:50<1:22:50, 4.71s/it] 72%|███████▏ | 2668/3723 [3:29:55<1:22:48, 4.71s/it] 72%|███████▏ | 2669/3723 [3:29:59<1:22:41, 4.71s/it] 72%|███████▏ | 2670/372 0: {'loss': 0.6788, 'grad_norm': 0.8799408241749188, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.43} 0: {'loss': 0.6793, 'grad_norm': 0.8216301053326546, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.43} 0: 3 [3:30:04<1:22:36, 4.71s/it] 72%|███████▏ | 2670/3723 [3:30:04<1:22:36, 4.71s/it] 72%|███████▏ | 2671/3723 [3:30:09<1:22:39, 4.71s/it] 72%|███████▏ | 2672/3723 [3:30:14<1:22:36, 4.72s/it] 72%|███████▏ | 2673/3723 [3:30:18<1:22:28, 4.71s/it] 72%|███████▏ | 2674/3723 [3:30:23<1:22:21, 4.71s/it] 72%|███████▏ | 2675/3723 [3:30:28<1:22:07, 4.70s/it] 72%|███████▏ | 2676/3723 [3:30:32<1:21:55, 4.69s/it] 72%|███████▏ | 2677/3723 [3:30:37<1:21:49, 4.69s/it] 72%|███████▏ | 2678/3723 [3:30:42<1:21:42, 4.69s/it] 72%|███████▏ | 2679/3723 [3:30:46<1:21:39, 4.69s/it] 72%|███████▏ | 2680/3723 [3:30:51<1:21:47, 4.71s/it] 72%|███████▏ | 2680/3723 [3:30:51<1:21:47, 4.71s/it] 72%|███ 0: {'loss': 0.6723, 'grad_norm': 0.8500189209549488, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.43} 0: ███▏ | 2681/3723 [3:30:56<1:21:56, 4.72s/it] 72%|███████▏ | 2682/3723 [3:31:01<1:21:46, 4.71s/it] 72%|███████▏ | 2683/3723 [3:31:05<1:21:51, 4.72s/it] 72%|███████▏ | 2684/3723 [3:31:10<1:21:47, 4.72s/it] 72%|███████▏ | 2685/3723 [3:31:15<1:21:44, 4.72s/it] 72%|███████▏ | 2686/3723 [3:31:20<1:21:46, 4.73s/it] 72%|███████▏ | 2687/3723 [3:31:24<1:21:36, 4.73s/it] 72%|███████▏ | 2688/3723 [3:31:29<1:21:21, 4.72s/it] 72%|███████▏ | 2689/3723 [3:31:34<1:21:09, 4.71s/it] 72%|███████▏ | 2690/3723 [3:31:38<1:20:59, 4.70s/it] 72%|███████▏ | 2690/3723 [3:31:38<1:20:59, 4.70s/it] 72%|███████▏ | 2691/3723 [3:31:43<1:21:05, 4.71s/it] 72%|███████▏ | 2692/3723 [3:31:48<1:21:00, 4.71s/it] 72%|███████▏ | 2693/3723 [3: 0: {'loss': 0.6959, 'grad_norm': 0.9136081775549999, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.44} 0: 31:53<1:20:53, 4.71s/it] 72%|███████▏ | 2694/3723 [3:31:57<1:20:45, 4.71s/it] 72%|███████▏ | 2695/3723 [3:32:02<1:20:46, 4.71s/it] 72%|███████▏ | 2696/3723 [3:32:07<1:20:49, 4.72s/it] 72%|███████▏ | 2697/3723 [3:32:11<1:20:50, 4.73s/it] 72%|███████▏ | 2698/3723 [3:32:16<1:20:44, 4.73s/it] 72%|███████▏ | 2699/3723 [3:32:21<1:20:38, 4.73s/it] 73%|███████▎ | 2700/3723 [3:32:26<1:20:30, 4.72s/it] 73%|███████▎ | 2700/3723 [3:32:26<1:20:30, 4.72s/it] 73%|███████▎ | 2701/3723 [3:32:30<1:20:30, 4.73s/it] 73%|███████▎ | 2702/3723 [3:32:35<1:20:16, 4.72s/it] 73%|███████▎ | 2703/3723 [3:32:40<1:20:07, 4.71s/it] 73%|███████▎ | 2704/3723 [3:32:45<1:20:25, 4.74s/it] 73%|███████▎ | 2705/3723 [3:32:49<1:20:55, 4.77s/it] 73%| 0: {'loss': 0.6815, 'grad_norm': 0.7763019069019502, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.44} 0: ███████▎ | 2706/3723 [3:32:54<1:21:22, 4.80s/it] 73%|███████▎ | 2707/3723 [3:32:59<1:21:03, 4.79s/it] 73%|███████▎ | 2708/3723 [3:33:04<1:20:29, 4.76s/it] 73%|███████▎ | 2709/3723 [3:33:08<1:20:05, 4.74s/it] 73%|███████▎ | 2710/3723 [3:33:13<1:19:47, 4.73s/it] 73%|███████▎ | 2710/3723 [3:33:13<1:19:47, 4.73s/it] 73%|███████▎ | 2711/3723 [3:33:18<1:19:42, 4.73s/it] 73%|███████▎ | 2712/3723 [3:33:23<1:19:33, 4.72s/it] 73%|███████▎ | 2713/3723 [3:33:27<1:19:27, 4.72s/it] 73%|███████▎ | 2714/3723 [3:33:32<1:19:14, 4.71s/it] 73%|███████▎ | 2715/3723 [3:33:37<1:19:07, 4.71s/it] 73%|███████▎ | 2716/3723 [3:33:41<1:19:04, 4.71s/it] 73%|███████▎ | 2717/3723 [3:33:46<1:19:13, 4.73s/it] 73%|███████▎ | 271 0: {'loss': 0.6713, 'grad_norm': 0.8534948592764455, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.44} 0: 8/3723 [3:33:51<1:19:13, 4.73s/it] 73%|███████▎ | 2719/3723 [3:33:56<1:19:10, 4.73s/it] 73%|███████▎ | 2720/3723 [3:34:00<1:19:13, 4.74s/it] 73%|███████▎ | 2720/3723 [3:34:00<1:19:13, 4.74s/it] 73%|███████▎ | 2721/3723 [3:34:05<1:19:15, 4.75s/it] 73%|███████▎ | 2722/3723 [3:34:10<1:19:12, 4.75s/it] 73%|███████▎ | 2723/3723 [3:34:15<1:19:09, 4.75s/it] 73%|███████▎ | 2724/3723 [3:34:19<1:19:11, 4.76s/it] 73%|███████▎ | 2725/3723 [3:34:24<1:19:07, 4.76s/it] 73%|███████▎ | 2726/3723 [3:34:29<1:19:00, 4.76s/it] 73%|███████▎ | 2727/3723 [3:34:34<1:18:53, 4.75s/it] 73%|███████▎ | 2728/3723 [3:34:38<1:18:49, 4.75s/it] 73%|███████▎ | 2729/3723 [3:34:43<1:18:40, 4.75s/it] 73%|███████▎ | 2730/3723 [3:34:48<1:18:34, 4.75s 0: {'loss': 0.6929, 'grad_norm': 0.9275208506275224, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.44} 0: {'loss': 0.676, 'grad_norm': 0.7313314349581661, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.44} 0: /it] 73%|███████▎ | 2730/3723 [3:34:48<1:18:34, 4.75s/it] 73%|███████▎ | 2731/3723 [3:34:53<1:18:30, 4.75s/it] 73%|███████▎ | 2732/3723 [3:34:57<1:18:23, 4.75s/it] 73%|███████▎ | 2733/3723 [3:35:02<1:18:17, 4.74s/it] 73%|███████▎ | 2734/3723 [3:35:07<1:18:07, 4.74s/it] 73%|███████▎ | 2735/3723 [3:35:12<1:17:57, 4.73s/it] 73%|███████▎ | 2736/3723 [3:35:16<1:17:44, 4.73s/it] 74%|███████▎ | 2737/3723 [3:35:21<1:17:27, 4.71s/it] 74%|███████▎ | 2738/3723 [3:35:26<1:17:19, 4.71s/it] 74%|███████▎ | 2739/3723 [3:35:30<1:17:07, 4.70s/it] 74%|███████▎ | 2740/3723 [3:35:35<1:17:01, 4.70s/it] 74%|███████▎ | 2740/3723 [3:35:35<1:17:01, 4.70s/it] 74%|███████▎ | 2741/372 0: {'loss': 0.6848, 'grad_norm': 0.8503087295333378, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.44} 0: 3 [3:35:40<1:17:04, 4.71s/it] 74%|███████▎ | 2742/3723 [3:35:44<1:17:09, 4.72s/it] 74%|███████▎ | 2743/3723 [3:35:49<1:17:27, 4.74s/it] 74%|███████▎ | 2744/3723 [3:35:54<1:17:31, 4.75s/it] 74%|███████▎ | 2745/3723 [3:35:59<1:17:24, 4.75s/it] 74%|███████▍ | 2746/3723 [3:36:04<1:17:11, 4.74s/it] 74%|███████▍ | 2747/3723 [3:36:08<1:17:04, 4.74s/it] 74%|███████▍ | 2748/3723 [3:36:13<1:16:58, 4.74s/it] 74%|███████▍ | 2749/3723 [3:36:18<1:16:50, 4.73s/it] 74%|███████▍ | 2750/3723 [3:36:22<1:16:45, 4.73s/it] 74%|███████▍ | 2750/3723 [3:36:22<1:16:45, 4.73s/it] 74%|███████▍ | 2751/3723 [3:36:27<1:16:42, 4.73s/it] 74%|███████▍ | 2752/3723 [3:36:32<1:16:32, 4.73s/it] 74%|███████▍ | 2753/3723 [3:36:37<1:16:19, 4.72s/it] 0: {'loss': 0.6794, 'grad_norm': 0.9737173335069241, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.44} 0: 74%|███████▍ | 2754/3723 [3:36:41<1:16:09, 4.72s/it] 74%|███████▍ | 2755/3723 [3:36:46<1:16:03, 4.71s/it] 74%|███████▍ | 2756/3723 [3:36:51<1:15:53, 4.71s/it] 74%|███████▍ | 2757/3723 [3:36:55<1:15:43, 4.70s/it] 74%|███████▍ | 2758/3723 [3:37:00<1:15:34, 4.70s/it] 74%|███████▍ | 2759/3723 [3:37:05<1:15:29, 4.70s/it] 74%|███████▍ | 2760/3723 [3:37:09<1:15:19, 4.69s/it] 74%|███████▍ | 2760/3723 [3:37:09<1:15:19, 4.69s/it] 74%|███████▍ | 2761/3723 [3:37:14<1:15:14, 4.69s/it] 74%|███████▍ | 2762/3723 [3:37:19<1:15:07, 4.69s/it] 74%|███████▍ | 2763/3723 [3:37:24<1:15:01, 4.69s/it] 74%|███████▍ | 2764/3723 [3:37:28<1:15:02, 4.70s/it] 74%|███████▍ | 2765/3723 [3:37:33<1:14:59, 4.70s/it] 74%|███████▍ 0: {'loss': 0.6901, 'grad_norm': 0.8115810429359951, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.45} 0: | 2766/3723 [3:37:38<1:15:01, 4.70s/it] 74%|███████▍ | 2767/3723 [3:37:42<1:14:57, 4.70s/it] 74%|███████▍ | 2768/3723 [3:37:47<1:14:39, 4.69s/it] 74%|███████▍ | 2769/3723 [3:37:52<1:14:16, 4.67s/it] 74%|███████▍ | 2770/3723 [3:37:56<1:14:06, 4.67s/it] 74%|███████▍ | 2770/3723 [3:37:56<1:14:06, 4.67s/it] 74%|███████▍ | 2771/3723 [3:38:01<1:13:59, 4.66s/it] 74%|███████▍ | 2772/3723 [3:38:06<1:13:55, 4.66s/it] 74%|███████▍ | 2773/3723 [3:38:10<1:13:55, 4.67s/it] 75%|███████▍ | 2774/3723 [3:38:15<1:13:47, 4.67s/it] 75%|███████▍ | 2775/3723 [3:38:20<1:13:37, 4.66s/it] 75%|███████▍ | 2776/3723 [3:38:24<1:13:42, 4.67s/it] 75%|███████▍ | 2777/3723 [3:38:29<1:13:33, 4.67s/it] 75%|███████▍ | 2778/3723 [3:38:34<1:13:27, 0: {'loss': 0.6793, 'grad_norm': 0.7940787900094696, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.45} 0: 4.66s/it] 75%|███████▍ | 2779/3723 [3:38:38<1:13:18, 4.66s/it] 75%|███████▍ | 2780/3723 [3:38:43<1:13:12, 4.66s/it] 75%|███████▍ | 2780/3723 [3:38:43<1:13:12, 4.66s/it] 75%|███████▍ | 2781/3723 [3:38:48<1:13:12, 4.66s/it] 75%|███████▍ | 2782/3723 [3:38:52<1:13:02, 4.66s/it] 75%|███████▍ | 2783/3723 [3:38:57<1:12:57, 4.66s/it] 75%|███████▍ | 2784/3723 [3:39:02<1:15:23, 4.82s/it] 75%|███████▍ | 2785/3723 [3:39:07<1:14:30, 4.77s/it] 75%|███████▍ | 2786/3723 [3:39:11<1:13:51, 4.73s/it] 75%|███████▍ | 2787/3723 [3:39:16<1:13:23, 4.70s/it] 75%|███████▍ | 2788/3723 [3:39:21<1:13:01, 4.69s/it] 75%|███████▍ | 2789/3723 [3:39:25<1:12:37, 4.67s/it] 75%|███████▍ | 2790/3723 [3:39:30<1:12:21, 4.65s/it] 0: {'loss': 0.6836, 'grad_norm': 0.933109042565421, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.45} 0: {'loss': 0.6833, 'grad_norm': 0.797251852194195, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.45} 0: 75%|███████▍ | 2790/3723 [3:39:30<1:12:21, 4.65s/it] 75%|███████▍ | 2791/3723 [3:39:35<1:12:07, 4.64s/it] 75%|███████▍ | 2792/3723 [3:39:39<1:11:52, 4.63s/it] 75%|███████▌ | 2793/3723 [3:39:44<1:11:41, 4.63s/it] 75%|███████▌ | 2794/3723 [3:39:48<1:11:40, 4.63s/it] 75%|███████▌ | 2795/3723 [3:39:53<1:11:36, 4.63s/it] 75%|███████▌ | 2796/3723 [3:39:58<1:11:32, 4.63s/it] 75%|███████▌ | 2797/3723 [3:40:02<1:11:26, 4.63s/it] 75%|███████▌ | 2798/3723 [3:40:07<1:11:23, 4.63s/it] 75%|███████▌ | 2799/3723 [3:40:12<1:11:17, 4.63s/it] 75%|███████▌ | 2800/3723 [3:40:16<1:11:08, 4.63s/it] 75%|███████▌ | 2800/3723 [3:40:16<1:11:08, 4.63s/it] 75%|███████▌ | 2801/3723 [3:40:21<1:11:14, 4.64s 0: {'loss': 0.6775, 'grad_norm': 0.8159157528401868, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.45} 0: /it] 75%|███████▌ | 2802/3723 [3:40:25<1:10:55, 4.62s/it] 75%|███████▌ | 2803/3723 [3:40:30<1:10:56, 4.63s/it] 75%|███████▌ | 2804/3723 [3:40:35<1:10:52, 4.63s/it] 75%|███████▌ | 2805/3723 [3:40:39<1:10:48, 4.63s/it] 75%|███████▌ | 2806/3723 [3:40:44<1:10:45, 4.63s/it] 75%|███████▌ | 2807/3723 [3:40:49<1:10:40, 4.63s/it] 75%|███████▌ | 2808/3723 [3:40:53<1:10:38, 4.63s/it] 75%|███████▌ | 2809/3723 [3:40:58<1:10:35, 4.63s/it] 75%|███████▌ | 2810/3723 [3:41:02<1:10:29, 4.63s/it] 75%|███████▌ | 2810/3723 [3:41:02<1:10:29, 4.63s/it] 76%|███████▌ | 2811/3723 [3:41:07<1:10:26, 4.63s/it] 76%|███████▌ | 2812/3723 [3:41:12<1:11:05, 4.68s/it] 76%|███████▌ | 2813/3723 [3:41:17<1:11:35, 4.72s/it] 76%|███████ 0: {'loss': 0.6898, 'grad_norm': 0.8326248115117819, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.45} 0: ▌ | 2814/3723 [3:41:22<1:12:08, 4.76s/it] 76%|███████▌ | 2815/3723 [3:41:27<1:16:33, 5.06s/it] 76%|███████▌ | 2816/3723 [3:41:32<1:15:46, 5.01s/it] 76%|███████▌ | 2817/3723 [3:41:38<1:20:34, 5.34s/it] 76%|███████▌ | 2818/3723 [3:41:44<1:23:00, 5.50s/it] 76%|███████▌ | 2819/3723 [3:41:49<1:19:15, 5.26s/it] 76%|███████▌ | 2820/3723 [3:41:54<1:16:19, 5.07s/it] 76%|███████▌ | 2820/3723 [3:41:54<1:16:19, 5.07s/it] 76%|███████▌ | 2821/3723 [3:41:58<1:14:26, 4.95s/it] 76%|███████▌ | 2822/3723 [3:42:03<1:12:51, 4.85s/it] 76%|███████▌ | 2823/3723 [3:42:07<1:11:38, 4.78s/it] 76%|███████▌ | 2824/3723 [3:42:12<1:10:45, 4.72s/it] 76%|███████▌ | 2825/3723 [3:42:17<1:10:11, 4.69s/it] 76%|███████▌ | 2826/3723 [3:42:21<1:09: 0: {'loss': 0.6676, 'grad_norm': 0.9351094212354941, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.46} 0: 48, 4.67s/it] 76%|███████▌ | 2827/3723 [3:42:26<1:09:22, 4.65s/it] 76%|███████▌ | 2828/3723 [3:42:30<1:09:04, 4.63s/it] 76%|███████▌ | 2829/3723 [3:42:35<1:08:51, 4.62s/it] 76%|███████▌ | 2830/3723 [3:42:40<1:08:40, 4.61s/it] 76%|███████▌ | 2830/3723 [3:42:40<1:08:40, 4.61s/it] 76%|███████▌ | 2831/3723 [3:42:44<1:08:26, 4.60s/it] 76%|███████▌ | 2832/3723 [3:42:49<1:08:16, 4.60s/it] 76%|███████▌ | 2833/3723 [3:42:53<1:08:05, 4.59s/it] 76%|███████▌ | 2834/3723 [3:42:58<1:08:11, 4.60s/it] 76%|███████▌ | 2835/3723 [3:43:03<1:08:09, 4.61s/it] 76%|███████▌ | 2836/3723 [3:43:07<1:08:13, 4.62s/it] 76%|███████▌ | 2837/3723 [3:43:12<1:08:27, 4.64s/it] 76%|███████▌ | 2838/3723 [3:43:17<1:08:37, 4.65s/it] 76%|███ 0: {'loss': 0.6725, 'grad_norm': 0.8567055746648371, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.46} 0: ███▋ | 2839/3723 [3:43:21<1:08:39, 4.66s/it] 76%|███████▋ | 2840/3723 [3:43:26<1:08:36, 4.66s/it] 76%|███████▋ | 2840/3723 [3:43:26<1:08:36, 4.66s/it] 76%|███████▋ | 2841/3723 [3:43:31<1:08:36, 4.67s/it] 76%|███████▋ | 2842/3723 [3:43:35<1:08:34, 4.67s/it] 76%|███████▋ | 2843/3723 [3:43:40<1:08:36, 4.68s/it] 76%|███████▋ | 2844/3723 [3:43:45<1:08:31, 4.68s/it] 76%|███████▋ | 2845/3723 [3:43:49<1:08:28, 4.68s/it] 76%|███████▋ | 2846/3723 [3:43:54<1:08:29, 4.69s/it] 76%|███████▋ | 2847/3723 [3:43:59<1:08:23, 4.68s/it] 76%|███████▋ | 2848/3723 [3:44:03<1:08:10, 4.67s/it] 77%|███████▋ | 2849/3723 [3:44:08<1:08:07, 4.68s/it] 77%|███████▋ | 2850/3723 [3:44:13<1:08:07, 4.68s/it] 0: {'loss': 0.6928, 'grad_norm': 0.8363252638843346, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.46} 0: {'loss': 0.6732, 'grad_norm': 0.7404557437435019, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.46} 0: 77%|███████▋ | 2850/3723 [3:44:13<1:08:07, 4.68s/it] 77%|███████▋ | 2851/3723 [3:44:18<1:08:11, 4.69s/it] 77%|███████▋ | 2852/3723 [3:44:22<1:08:06, 4.69s/it] 77%|███████▋ | 2853/3723 [3:44:27<1:08:04, 4.69s/it] 77%|███████▋ | 2854/3723 [3:44:32<1:07:58, 4.69s/it] 77%|███████▋ | 2855/3723 [3:44:36<1:07:48, 4.69s/it] 77%|███████▋ | 2856/3723 [3:44:41<1:07:36, 4.68s/it] 77%|███████▋ | 2857/3723 [3:44:46<1:07:29, 4.68s/it] 77%|███████▋ | 2858/3723 [3:44:50<1:07:24, 4.68s/it] 77%|███████▋ | 2859/3723 [3:44:55<1:07:29, 4.69s/it] 77%|███████▋ | 2860/3723 [3:45:00<1:07:36, 4.70s/it] 77%|███████▋ | 2860/3723 [3:45:00<1:07:36, 4.70s/it] 77%|███████▋ | 2861/3723 [3:45:04<1:07:45, 4.72s/it] 77%|█████ 0: {'loss': 0.663, 'grad_norm': 0.8403970490245335, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.46} 0: █▋ | 2862/3723 [3:45:09<1:07:46, 4.72s/it] 77%|███████▋ | 2863/3723 [3:45:14<1:07:45, 4.73s/it] 77%|███████▋ | 2864/3723 [3:45:19<1:07:39, 4.73s/it] 77%|███████▋ | 2865/3723 [3:45:23<1:07:26, 4.72s/it] 77%|███████▋ | 2866/3723 [3:45:28<1:07:21, 4.72s/it] 77%|███████▋ | 2867/3723 [3:45:33<1:07:20, 4.72s/it] 77%|███████▋ | 2868/3723 [3:45:38<1:07:12, 4.72s/it] 77%|███████▋ | 2869/3723 [3:45:42<1:07:03, 4.71s/it] 77%|███████▋ | 2870/3723 [3:45:47<1:07:00, 4.71s/it] 77%|███████▋ | 2870/3723 [3:45:47<1:07:00, 4.71s/it] 77%|███████▋ | 2871/3723 [3:45:52<1:07:00, 4.72s/it] 77%|███████▋ | 2872/3723 [3:45:56<1:06:48, 4.71s/it] 77%|███████▋ | 2873/3723 [3:46:01<1:06:40, 4.71s/it] 77%|███████▋ | 2874/3723 [3:46:06< 0: {'loss': 0.6646, 'grad_norm': 0.8412307020355303, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.46} 0: 1:06:27, 4.70s/it] 77%|███████▋ | 2875/3723 [3:46:10<1:06:24, 4.70s/it] 77%|███████▋ | 2876/3723 [3:46:15<1:06:24, 4.70s/it] 77%|███████▋ | 2877/3723 [3:46:20<1:06:21, 4.71s/it] 77%|███████▋ | 2878/3723 [3:46:25<1:06:20, 4.71s/it] 77%|███████▋ | 2879/3723 [3:46:29<1:06:13, 4.71s/it] 77%|███████▋ | 2880/3723 [3:46:34<1:06:05, 4.70s/it] 77%|███████▋ | 2880/3723 [3:46:34<1:06:05, 4.70s/it] 77%|███████▋ | 2881/3723 [3:46:39<1:05:58, 4.70s/it] 77%|███████▋ | 2882/3723 [3:46:43<1:05:53, 4.70s/it] 77%|███████▋ | 2883/3723 [3:46:48<1:05:43, 4.69s/it] 77%|███████▋ | 2884/3723 [3:46:53<1:05:36, 4.69s/it] 77%|███████▋ | 2885/3723 [3:46:58<1:05:47, 4.71s/it] 78%|███████▊ | 2886/3723 [3:47:02<1:05:37, 4.70s/it] 78%|██ 0: {'loss': 0.6711, 'grad_norm': 0.9583627780035591, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.47} 0: █████▊ | 2887/3723 [3:47:07<1:05:26, 4.70s/it] 78%|███████▊ | 2888/3723 [3:47:12<1:05:17, 4.69s/it] 78%|███████▊ | 2889/3723 [3:47:16<1:05:13, 4.69s/it] 78%|███████▊ | 2890/3723 [3:47:21<1:05:07, 4.69s/it] 78%|███████▊ | 2890/3723 [3:47:21<1:05:07, 4.69s/it] 78%|███████▊ | 2891/3723 [3:47:26<1:05:10, 4.70s/it] 78%|███████▊ | 2892/3723 [3:47:30<1:05:09, 4.70s/it] 78%|███████▊ | 2893/3723 [3:47:35<1:05:07, 4.71s/it] 78%|███████▊ | 2894/3723 [3:47:40<1:05:07, 4.71s/it] 78%|███████▊ | 2895/3723 [3:47:45<1:05:04, 4.71s/it] 78%|███████▊ | 2896/3723 [3:47:49<1:04:59, 4.72s/it] 78%|███████▊ | 2897/3723 [3:47:54<1:04:52, 4.71s/it] 78%|███████▊ | 2898/3723 [3:47:59<1:04:46, 4.71s/it] 78%|███████▊ | 2899/3723 0: {'loss': 0.6907, 'grad_norm': 0.8322385484741117, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.47} 0: {'loss': 0.6794, 'grad_norm': 0.9758744294451527, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.47} 0: [3:48:03<1:04:43, 4.71s/it] 78%|███████▊ | 2900/3723 [3:48:08<1:04:41, 4.72s/it] 78%|███████▊ | 2900/3723 [3:48:08<1:04:41, 4.72s/it] 78%|███████▊ | 2901/3723 [3:48:13<1:04:43, 4.72s/it] 78%|███████▊ | 2902/3723 [3:48:18<1:04:37, 4.72s/it] 78%|███████▊ | 2903/3723 [3:48:22<1:04:28, 4.72s/it] 78%|███████▊ | 2904/3723 [3:48:27<1:04:23, 4.72s/it] 78%|███████▊ | 2905/3723 [3:48:32<1:04:16, 4.71s/it] 78%|███████▊ | 2906/3723 [3:48:36<1:04:11, 4.71s/it] 78%|███████▊ | 2907/3723 [3:48:41<1:04:07, 4.71s/it] 78%|███████▊ | 2908/3723 [3:48:46<1:03:59, 4.71s/it] 78%|███████▊ | 2909/3723 [3:48:51<1:03:51, 4.71s/it] 78%|███████▊ | 2910/3723 [3:48:55<1:03:51, 4.71s/it] 78%|███ 0: {'loss': 0.6823, 'grad_norm': 0.7941496150450622, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.47} 0: ███▊ | 2910/3723 [3:48:55<1:03:51, 4.71s/it] 78%|███████▊ | 2911/3723 [3:49:00<1:03:52, 4.72s/it] 78%|███████▊ | 2912/3723 [3:49:05<1:03:42, 4.71s/it] 78%|███████▊ | 2913/3723 [3:49:09<1:03:30, 4.70s/it] 78%|███████▊ | 2914/3723 [3:49:14<1:03:28, 4.71s/it] 78%|███████▊ | 2915/3723 [3:49:19<1:03:20, 4.70s/it] 78%|███████▊ | 2916/3723 [3:49:23<1:03:18, 4.71s/it] 78%|███████▊ | 2917/3723 [3:49:28<1:03:15, 4.71s/it] 78%|███████▊ | 2918/3723 [3:49:33<1:03:07, 4.71s/it] 78%|███████▊ | 2919/3723 [3:49:38<1:02:39, 4.68s/it] 78%|███████▊ | 2920/3723 [3:49:42<1:02:25, 4.66s/it] 78%|███████▊ | 2920/3723 [3:49:42<1:02:25, 4.66s/it] 78%|███████▊ | 2921/3723 [3:49:47<1:02:21, 4.67s/it] 78%|███████▊ | 2922/3723 [3:4 0: {'loss': 0.6917, 'grad_norm': 0.8235722087112484, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.47} 0: 9:51<1:02:03, 4.65s/it] 79%|███████▊ | 2923/3723 [3:49:56<1:01:57, 4.65s/it] 79%|███████▊ | 2924/3723 [3:50:01<1:01:53, 4.65s/it] 79%|███████▊ | 2925/3723 [3:50:05<1:01:46, 4.64s/it] 79%|███████▊ | 2926/3723 [3:50:10<1:01:39, 4.64s/it] 79%|███████▊ | 2927/3723 [3:50:15<1:01:29, 4.63s/it] 79%|███████▊ | 2928/3723 [3:50:19<1:01:20, 4.63s/it] 79%|███████▊ | 2929/3723 [3:50:24<1:01:11, 4.62s/it] 79%|███████▊ | 2930/3723 [3:50:28<1:01:06, 4.62s/it] 79%|███████▊ | 2930/3723 [3:50:28<1:01:06, 4.62s/it] 79%|███████▊ | 2931/3723 [3:50:33<1:01:08, 4.63s/it] 79%|███████▉ | 2932/3723 [3:50:38<1:01:03, 4.63s/it] 79%|███████▉ | 2933/3723 [3:50:42<1:00:55, 4.63s/it] 79%|███████▉ | 2934/3723 [3:50:47<1:00:49, 4.63s/it] 79%| 0: {'loss': 0.673, 'grad_norm': 0.9402298859936337, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.47} 0: ██████▉ | 2935/3723 [3:50:52<1:00:56, 4.64s/it] 79%|███████▉ | 2936/3723 [3:50:56<1:01:11, 4.66s/it] 79%|███████▉ | 2937/3723 [3:51:01<1:01:12, 4.67s/it] 79%|███████▉ | 2938/3723 [3:51:06<1:01:14, 4.68s/it] 79%|███████▉ | 2939/3723 [3:51:10<1:01:16, 4.69s/it] 79%|███████▉ | 2940/3723 [3:51:15<1:01:14, 4.69s/it] 79%|███████▉ | 2940/3723 [3:51:15<1:01:14, 4.69s/it] 79%|███████▉ | 2941/3723 [3:51:20<1:01:13, 4.70s/it] 79%|███████▉ | 2942/3723 [3:51:25<1:01:09, 4.70s/it] 79%|███████▉ | 2943/3723 [3:51:29<1:01:07, 4.70s/it] 79%|███████▉ | 2944/3723 [3:51:34<1:00:50, 4.69s/it] 79%|███████▉ | 2945/3723 [3:51:39<1:00:38, 4.68s/it] 79%|███████▉ | 2946/3723 [3:51:43<1:00:27, 4.67s/it] 79%|███████▉ | 2947 0: {'loss': 0.6818, 'grad_norm': 0.8599636991583945, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.48} 0: /3723 [3:51:48<1:00:22, 4.67s/it] 79%|███████▉ | 2948/3723 [3:51:53<1:00:09, 4.66s/it] 79%|███████▉ | 2949/3723 [3:51:57<1:00:03, 4.66s/it] 79%|███████▉ | 2950/3723 [3:52:02<59:54, 4.65s/it] 79%|███████▉ | 2950/3723 [3:52:02<59:54, 4.65s/it] 79%|███████▉ | 2951/3723 [3:52:07<59:53, 4.66s/it] 79%|███████▉ | 2952/3723 [3:52:11<59:42, 4.65s/it] 79%|███████▉ | 2953/3723 [3:52:16<59:35, 4.64s/it] 79%|███████▉ | 2954/3723 [3:52:20<59:27, 4.64s/it] 79%|███████▉ | 2955/3723 [3:52:25<59:36, 4.66s/it] 79%|███████▉ | 2956/3723 [3:52:30<59:27, 4.65s/it] 79%|███████▉ | 2957/3723 [3:52:34<59:21, 4.65s/it] 79%|███████▉ | 2958/3723 [3:52:39<59:16, 4.65s/it] 79%|███████▉ | 2959/3723 [3:52:44<59:10, 4.65s/it] 80%|████ 0: {'loss': 0.6875, 'grad_norm': 0.7848351449917016, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.48} 0: {'loss': 0.6753, 'grad_norm': 0.7521616236255079, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.48} 0: ██▉ | 2960/3723 [3:52:48<59:04, 4.65s/it] 80%|███████▉ | 2960/3723 [3:52:48<59:04, 4.65s/it] 80%|███████▉ | 2961/3723 [3:52:53<59:03, 4.65s/it] 80%|███████▉ | 2962/3723 [3:52:58<59:01, 4.65s/it] 80%|███████▉ | 2963/3723 [3:53:02<58:52, 4.65s/it] 80%|███████▉ | 2964/3723 [3:53:07<58:35, 4.63s/it] 80%|███████▉ | 2965/3723 [3:53:11<58:28, 4.63s/it] 80%|███████▉ | 2966/3723 [3:53:16<58:25, 4.63s/it] 80%|███████▉ | 2967/3723 [3:53:21<58:23, 4.63s/it] 80%|███████▉ | 2968/3723 [3:53:25<58:23, 4.64s/it] 80%|███████▉ | 2969/3723 [3:53:30<58:22, 4.65s/it] 80%|███████▉ | 2970/3723 [3:53:35<58:18, 4.65s/it] 80%|███████▉ | 2970/3723 [3:53:35<58:18, 4.65s/it] 80%|█████ 0: {'loss': 0.6726, 'grad_norm': 0.8617835672024611, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.48} 0: █▉ | 2971/3723 [3:53:39<58:18, 4.65s/it] 80%|███████▉ | 2972/3723 [3:53:44<58:17, 4.66s/it] 80%|███████▉ | 2973/3723 [3:53:49<58:12, 4.66s/it] 80%|███████▉ | 2974/3723 [3:53:53<58:11, 4.66s/it] 80%|███████▉ | 2975/3723 [3:53:58<58:07, 4.66s/it] 80%|███████▉ | 2976/3723 [3:54:03<57:53, 4.65s/it] 80%|███████▉ | 2977/3723 [3:54:07<57:42, 4.64s/it] 80%|███████▉ | 2978/3723 [3:54:12<57:27, 4.63s/it] 80%|████████ | 2979/3723 [3:54:16<57:19, 4.62s/it] 80%|████████ | 2980/3723 [3:54:21<57:20, 4.63s/it] 80%|████████ | 2980/3723 [3:54:21<57:20, 4.63s/it] 80%|████████ | 2981/3723 [3:54:26<57:22, 4.64s/it] 80%|████████ | 2982/3723 [3:54:30<57:21, 4.64s/it] 80%|████████ | 2983/3723 [3:54:35<57:11, 4.64s/it] 80%|██ 0: {'loss': 0.6891, 'grad_norm': 0.873513874010637, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.48} 0: ██████ | 2984/3723 [3:54:40<56:58, 4.63s/it] 80%|████████ | 2985/3723 [3:54:44<56:47, 4.62s/it] 80%|████████ | 2986/3723 [3:54:49<56:49, 4.63s/it] 80%|████████ | 2987/3723 [3:54:54<56:44, 4.63s/it] 80%|████████ | 2988/3723 [3:54:58<56:35, 4.62s/it] 80%|████████ | 2989/3723 [3:55:03<56:31, 4.62s/it] 80%|████████ | 2990/3723 [3:55:07<56:26, 4.62s/it] 80%|████████ | 2990/3723 [3:55:07<56:26, 4.62s/it] 80%|████████ | 2991/3723 [3:55:12<56:25, 4.63s/it] 80%|████████ | 2992/3723 [3:55:17<56:19, 4.62s/it] 80%|████████ | 2993/3723 [3:55:21<56:13, 4.62s/it] 80%|████████ | 2994/3723 [3:55:26<56:17, 4.63s/it] 80%|████████ | 2995/3723 [3:55:31<56:15, 4.64s/it] 80%|████████ | 2996/3723 [3:55:35<56:11, 4.64s/it] 0: {'loss': 0.6628, 'grad_norm': 0.7506865921892917, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.48} 0: 80%|████████ | 2997/3723 [3:55:40<56:05, 4.64s/it] 81%|████████ | 2998/3723 [3:55:44<56:00, 4.64s/it] 81%|████████ | 2999/3723 [3:55:49<55:54, 4.63s/it] 81%|████████ | 3000/3723 [3:55:54<55:50, 4.63s/it] 81%|████████ | 3000/3723 [3:55:54<55:50, 4.63s/it] 81%|████████ | 3001/3723 [3:55:58<55:49, 4.64s/it] 81%|████████ | 3002/3723 [3:56:03<55:44, 4.64s/it] 81%|████████ | 3003/3723 [3:56:08<55:42, 4.64s/it] 81%|████████ | 3004/3723 [3:56:12<55:36, 4.64s/it] 81%|████████ | 3005/3723 [3:56:17<55:37, 4.65s/it] 81%|████████ | 3006/3723 [3:56:22<55:36, 4.65s/it] 81%|████████ | 3007/3723 [3:56:26<55:35, 4.66s/it] 81%|████████ | 3008/3723 [3:56:31<55:34, 4.66s/it] 81%|████████ | 3009/3723 [3:56:36<55:31, 0: {'loss': 0.6779, 'grad_norm': 0.8214128482570467, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.49} 0: {'loss': 0.6804, 'grad_norm': 1.1680850974195716, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.49} 0: 4.67s/it] 81%|████████ | 3010/3723 [3:56:40<55:28, 4.67s/it] 81%|████████ | 3010/3723 [3:56:40<55:28, 4.67s/it] 81%|████████ | 3011/3723 [3:56:45<55:25, 4.67s/it] 81%|████████ | 3012/3723 [3:56:50<55:18, 4.67s/it] 81%|████████ | 3013/3723 [3:56:54<55:13, 4.67s/it] 81%|████████ | 3014/3723 [3:56:59<55:06, 4.66s/it] 81%|████████ | 3015/3723 [3:57:04<55:04, 4.67s/it] 81%|████████ | 3016/3723 [3:57:08<54:59, 4.67s/it] 81%|████████ | 3017/3723 [3:57:13<54:52, 4.66s/it] 81%|████████ | 3018/3723 [3:57:18<54:46, 4.66s/it] 81%|████████ | 3019/3723 [3:57:22<54:40, 4.66s/it] 81%|████████ | 3020/3723 [3:57:27<54:34, 4.66s/it] 81%|████████ | 3020/3723 [3:57:27<54:34, 4.6 0: {'loss': 0.6693, 'grad_norm': 0.8981143303506463, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.49} 0: 6s/it] 81%|████████ | 3021/3723 [3:57:32<54:34, 4.66s/it] 81%|████████ | 3022/3723 [3:57:36<54:30, 4.67s/it] 81%|████████ | 3023/3723 [3:57:41<54:28, 4.67s/it] 81%|████████ | 3024/3723 [3:57:46<54:24, 4.67s/it] 81%|████████▏ | 3025/3723 [3:57:50<54:17, 4.67s/it] 81%|████████▏ | 3026/3723 [3:57:55<54:08, 4.66s/it] 81%|████████▏ | 3027/3723 [3:58:00<53:58, 4.65s/it] 81%|████████▏ | 3028/3723 [3:58:04<53:51, 4.65s/it] 81%|████████▏ | 3029/3723 [3:58:09<53:45, 4.65s/it] 81%|████████▏ | 3030/3723 [3:58:14<53:40, 4.65s/it] 81%|████████▏ | 3030/3723 [3:58:14<53:40, 4.65s/it] 81%|████████▏ | 3031/3723 [3:58:18<53:40, 4.65s/it] 81%|████████▏ | 3032/3723 [3:58:23<53:33, 4.65s/it] 81%|████████▏ | 0: {'loss': 0.6906, 'grad_norm': 0.9057469052858467, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.49} 0: 3033/3723 [3:58:27<53:24, 4.64s/it] 81%|████████▏ | 3034/3723 [3:58:32<53:25, 4.65s/it] 82%|████████▏ | 3035/3723 [3:58:37<53:26, 4.66s/it] 82%|████████▏ | 3036/3723 [3:58:41<53:23, 4.66s/it] 82%|████████▏ | 3037/3723 [3:58:46<53:16, 4.66s/it] 82%|████████▏ | 3038/3723 [3:58:51<53:07, 4.65s/it] 82%|████████▏ | 3039/3723 [3:58:55<52:57, 4.65s/it] 82%|████████▏ | 3040/3723 [3:59:00<52:49, 4.64s/it] 82%|████████▏ | 3040/3723 [3:59:00<52:49, 4.64s/it] 82%|████████▏ | 3041/3723 [3:59:05<52:47, 4.64s/it] 82%|████████▏ | 3042/3723 [3:59:09<52:47, 4.65s/it] 82%|████████▏ | 3043/3723 [3:59:14<52:44, 4.65s/it] 82%|████████▏ | 3044/3723 [3:59:19<52:41, 4.66s/it] 82%|████████▏ | 3045/3723 [3:59:23<53:01, 4.69s 0: {'loss': 0.6781, 'grad_norm': 0.7429485136332168, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.49} 0: /it] 82%|████████▏ | 3046/3723 [3:59:28<53:24, 4.73s/it] 82%|████████▏ | 3047/3723 [3:59:33<53:25, 4.74s/it] 82%|████████▏ | 3048/3723 [3:59:38<53:12, 4.73s/it] 82%|████████▏ | 3049/3723 [3:59:42<53:02, 4.72s/it] 82%|████████▏ | 3050/3723 [3:59:47<52:42, 4.70s/it] 82%|████████▏ | 3050/3723 [3:59:47<52:42, 4.70s/it] 82%|████████▏ | 3051/3723 [3:59:52<52:39, 4.70s/it] 82%|████████▏ | 3052/3723 [3:59:57<52:33, 4.70s/it] 82%|████████▏ | 3053/3723 [4:00:01<52:25, 4.69s/it] 82%|████████▏ | 3054/3723 [4:00:06<52:18, 4.69s/it] 82%|████████▏ | 3055/3723 [4:00:11<52:13, 4.69s/it] 82%|████████▏ | 3056/3723 [4:00:15<52:07, 4.69s/it] 82%|████████▏ | 3057/3723 [4:00:20<51:49, 4.67s/it] 82%|███████ 0: {'loss': 0.6745, 'grad_norm': 0.8053658161596124, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.49} 0: ▏ | 3058/3723 [4:00:25<51:38, 4.66s/it] 82%|████████▏ | 3059/3723 [4:00:29<51:33, 4.66s/it] 82%|████████▏ | 3060/3723 [4:00:34<51:24, 4.65s/it] 82%|████████▏ | 3060/3723 [4:00:34<51:24, 4.65s/it] 82%|████████▏ | 3061/3723 [4:00:38<51:11, 4.64s/it] 82%|████████▏ | 3062/3723 [4:00:43<50:57, 4.63s/it] 82%|████████▏ | 3063/3723 [4:00:48<50:46, 4.62s/it] 82%|████████▏ | 3064/3723 [4:00:52<50:43, 4.62s/it] 82%|████████▏ | 3065/3723 [4:00:57<50:50, 4.64s/it] 82%|████████▏ | 3066/3723 [4:01:02<50:49, 4.64s/it] 82%|████████▏ | 3067/3723 [4:01:06<50:33, 4.62s/it] 82%|████████▏ | 3068/3723 [4:01:11<50:25, 4.62s/it] 82%|████████▏ | 3069/3723 [4:01:15<50:07, 4.60s/it] 82%|████████▏ | 3070/3723 [4:01:20<49:55, 0: {'loss': 0.6809, 'grad_norm': 0.8236019161961098, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.49} 0: {'loss': 0.6851, 'grad_norm': 0.8103105558540005, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.5} 0: 4.59s/it] 82%|████████▏ | 3070/3723 [4:01:20<49:55, 4.59s/it] 82%|████████▏ | 3071/3723 [4:01:24<49:49, 4.59s/it] 83%|████████▎ | 3072/3723 [4:01:29<49:50, 4.59s/it] 83%|████████▎ | 3073/3723 [4:01:34<49:53, 4.61s/it] 83%|████████▎ | 3074/3723 [4:01:38<49:53, 4.61s/it] 83%|████████▎ | 3075/3723 [4:01:43<49:50, 4.62s/it] 83%|████████▎ | 3076/3723 [4:01:48<49:36, 4.60s/it] 83%|████████▎ | 3077/3723 [4:01:52<49:27, 4.59s/it] 83%|████████▎ | 3078/3723 [4:01:57<49:15, 4.58s/it] 83%|████████▎ | 3079/3723 [4:02:01<49:06, 4.57s/it] 83%|████████▎ | 3080/3723 [4:02:06<49:00, 4.57s/it] 83%|████████▎ | 3080/3723 [4:02:06<49:00, 4.57s/it] 83%|████████▎ | 3081 0: {'loss': 0.6853, 'grad_norm': 0.8662594009211825, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.5} 0: /3723 [4:02:10<48:59, 4.58s/it] 83%|████████▎ | 3082/3723 [4:02:15<49:00, 4.59s/it] 83%|████████▎ | 3083/3723 [4:02:20<48:59, 4.59s/it] 83%|████████▎ | 3084/3723 [4:02:24<48:59, 4.60s/it] 83%|████████▎ | 3085/3723 [4:02:29<48:53, 4.60s/it] 83%|████████▎ | 3086/3723 [4:02:33<48:49, 4.60s/it] 83%|████████▎ | 3087/3723 [4:02:38<48:50, 4.61s/it] 83%|████████▎ | 3088/3723 [4:02:43<48:49, 4.61s/it] 83%|████████▎ | 3089/3723 [4:02:47<48:48, 4.62s/it] 83%|████████▎ | 3090/3723 [4:02:52<48:50, 4.63s/it] 83%|████████▎ | 3090/3723 [4:02:52<48:50, 4.63s/it] 83%|████████▎ | 3091/3723 [4:02:57<48:50, 4.64s/it] 83%|████████▎ | 3092/3723 [4:03:01<48:49, 4.64s/it] 83%|████████▎ | 3093/3723 [4:03:06<48:57, 4.66s/it] 0: {'loss': 0.6687, 'grad_norm': 0.8059156896261102, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.5} 0: 83%|████████▎ | 3094/3723 [4:03:11<49:02, 4.68s/it] 83%|████████▎ | 3095/3723 [4:03:15<49:01, 4.68s/it] 83%|████████▎ | 3096/3723 [4:03:20<48:57, 4.69s/it] 83%|████████▎ | 3097/3723 [4:03:25<48:58, 4.69s/it] 83%|████████▎ | 3098/3723 [4:03:29<48:56, 4.70s/it] 83%|████████▎ | 3099/3723 [4:03:34<48:46, 4.69s/it] 83%|████████▎ | 3100/3723 [4:03:39<48:41, 4.69s/it] 83%|████████▎ | 3100/3723 [4:03:39<48:41, 4.69s/it] 83%|████████▎ | 3101/3723 [4:03:44<48:39, 4.69s/it] 83%|████████▎ | 3102/3723 [4:03:48<48:33, 4.69s/it] 83%|████████▎ | 3103/3723 [4:03:53<48:24, 4.69s/it] 83%|████████▎ | 3104/3723 [4:03:58<48:18, 4.68s/it] 83%|████████▎ | 3105/3723 [4:04:02<48:13, 4.68s/it] 83%|████████▎ 0: {'loss': 0.6533, 'grad_norm': 0.8126829374445104, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.5} 0: | 3106/3723 [4:04:07<48:13, 4.69s/it] 83%|████████▎ | 3107/3723 [4:04:12<48:16, 4.70s/it] 83%|████████▎ | 3108/3723 [4:04:16<48:21, 4.72s/it] 84%|████████▎ | 3109/3723 [4:04:21<48:28, 4.74s/it] 84%|████████▎ | 3110/3723 [4:04:26<49:32, 4.85s/it] 84%|████████▎ | 3110/3723 [4:04:26<49:32, 4.85s/it] 84%|████████▎ | 3111/3723 [4:04:31<49:21, 4.84s/it] 84%|████████▎ | 3112/3723 [4:04:36<49:08, 4.83s/it] 84%|████████▎ | 3113/3723 [4:04:41<48:57, 4.82s/it] 84%|████████▎ | 3114/3723 [4:04:46<48:46, 4.81s/it] 84%|████████▎ | 3115/3723 [4:04:50<48:39, 4.80s/it] 84%|████████▎ | 3116/3723 [4:04:55<48:31, 4.80s/it] 84%|████████▎ | 3117/3723 [4:05:00<48:26, 4.80s/it] 84%|████████▎ | 3118/3723 [4:05:05<48:22, 4.80 0: {'loss': 0.6783, 'grad_norm': 0.8391410152934285, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.5} 0: s/it] 84%|████████▍ | 3119/3723 [4:05:09<48:20, 4.80s/it] 84%|████████▍ | 3120/3723 [4:05:14<48:08, 4.79s/it] 84%|████████▍ | 3120/3723 [4:05:14<48:08, 4.79s/it] 84%|████████▍ | 3121/3723 [4:05:19<47:53, 4.77s/it] 84%|████████▍ | 3122/3723 [4:05:24<47:40, 4.76s/it] 84%|████████▍ | 3123/3723 [4:05:28<47:29, 4.75s/it] 84%|████████▍ | 3124/3723 [4:05:33<47:18, 4.74s/it] 84%|████████▍ | 3125/3723 [4:05:38<47:07, 4.73s/it] 84%|████████▍ | 3126/3723 [4:05:43<47:00, 4.72s/it] 84%|████████▍ | 3127/3723 [4:05:47<46:43, 4.70s/it] 84%|████████▍ | 3128/3723 [4:05:52<46:37, 4.70s/it] 84%|████████▍ | 3129/3723 [4:05:57<46:36, 4.71s/it] 84%|████████▍ | 3130/3723 [4:06:01<46:25, 4.70s/it] 0: {'loss': 0.6915, 'grad_norm': 0.9220927643160924, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.5} 0: {'loss': 0.6705, 'grad_norm': 0.8614149993834334, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.51} 0: 84%|████████▍ | 3130/3723 [4:06:01<46:25, 4.70s/it] 84%|████████▍ | 3131/3723 [4:06:06<46:28, 4.71s/it] 84%|████████▍ | 3132/3723 [4:06:11<46:32, 4.73s/it] 84%|████████▍ | 3133/3723 [4:06:16<46:25, 4.72s/it] 84%|████████▍ | 3134/3723 [4:06:20<46:25, 4.73s/it] 84%|████████▍ | 3135/3723 [4:06:25<46:19, 4.73s/it] 84%|████████▍ | 3136/3723 [4:06:30<46:14, 4.73s/it] 84%|████████▍ | 3137/3723 [4:06:34<46:11, 4.73s/it] 84%|████████▍ | 3138/3723 [4:06:39<46:04, 4.73s/it] 84%|████████▍ | 3139/3723 [4:06:44<45:57, 4.72s/it] 84%|████████▍ | 3140/3723 [4:06:49<45:54, 4.73s/it] 84%|████████▍ | 3140/3723 [4:06:49<45:54, 4.73s/it] 84%|████████▍ | 3141/3723 [4:06:53<45:53, 4.73s/it] 84%| 0: {'loss': 0.6656, 'grad_norm': 0.8243418577088166, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.51} 0: ████████▍ | 3142/3723 [4:06:58<45:47, 4.73s/it] 84%|████████▍ | 3143/3723 [4:07:03<45:40, 4.72s/it] 84%|████████▍ | 3144/3723 [4:07:08<45:34, 4.72s/it] 84%|████████▍ | 3145/3723 [4:07:12<45:23, 4.71s/it] 85%|████████▍ | 3146/3723 [4:07:17<45:13, 4.70s/it] 85%|████████▍ | 3147/3723 [4:07:22<45:06, 4.70s/it] 85%|████████▍ | 3148/3723 [4:07:26<44:46, 4.67s/it] 85%|████████▍ | 3149/3723 [4:07:31<44:33, 4.66s/it] 85%|████████▍ | 3150/3723 [4:07:35<44:21, 4.65s/it] 85%|████████▍ | 3150/3723 [4:07:35<44:21, 4.65s/it] 85%|████████▍ | 3151/3723 [4:07:40<44:11, 4.64s/it] 85%|████████▍ | 3152/3723 [4:07:45<44:08, 4.64s/it] 85%|████████▍ | 3153/3723 [4:07:49<44:03, 4.64s/it] 85%|████████▍ | 315 0: {'loss': 0.6768, 'grad_norm': 0.8057247876719764, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.51} 0: 4/3723 [4:07:54<44:06, 4.65s/it] 85%|████████▍ | 3155/3723 [4:07:59<43:58, 4.65s/it] 85%|████████▍ | 3156/3723 [4:08:03<43:50, 4.64s/it] 85%|████████▍ | 3157/3723 [4:08:08<43:44, 4.64s/it] 85%|████████▍ | 3158/3723 [4:08:13<43:41, 4.64s/it] 85%|████████▍ | 3159/3723 [4:08:17<43:39, 4.64s/it] 85%|████████▍ | 3160/3723 [4:08:22<43:36, 4.65s/it] 85%|████████▍ | 3160/3723 [4:08:22<43:36, 4.65s/it] 85%|████████▍ | 3161/3723 [4:08:26<43:29, 4.64s/it] 85%|████████▍ | 3162/3723 [4:08:31<43:19, 4.63s/it] 85%|████████▍ | 3163/3723 [4:08:36<43:14, 4.63s/it] 85%|████████▍ | 3164/3723 [4:08:40<43:08, 4.63s/it] 85%|████████▌ | 3165/3723 [4:08:45<43:00, 4.62s/it] 85%|████████▌ | 3166/3723 [4:08:50<42:55, 4.62s/it] 0: {'loss': 0.6654, 'grad_norm': 0.893623510999274, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.51} 0: 85%|████████▌ | 3167/3723 [4:08:54<42:48, 4.62s/it] 85%|████████▌ | 3168/3723 [4:08:59<42:42, 4.62s/it] 85%|████████▌ | 3169/3723 [4:09:03<42:37, 4.62s/it] 85%|████████▌ | 3170/3723 [4:09:08<42:33, 4.62s/it] 85%|████████▌ | 3170/3723 [4:09:08<42:33, 4.62s/it] 85%|████████▌ | 3171/3723 [4:09:13<42:32, 4.62s/it] 85%|████████▌ | 3172/3723 [4:09:17<42:29, 4.63s/it] 85%|████████▌ | 3173/3723 [4:09:22<42:23, 4.62s/it] 85%|████████▌ | 3174/3723 [4:09:27<42:21, 4.63s/it] 85%|████████▌ | 3175/3723 [4:09:31<42:14, 4.63s/it] 85%|████████▌ | 3176/3723 [4:09:36<42:11, 4.63s/it] 85%|████████▌ | 3177/3723 [4:09:41<42:14, 4.64s/it] 85%|████████▌ | 3178/3723 [4:09:45<42:17, 4.66s/it] 85%|████████▌ 0: {'loss': 0.6601, 'grad_norm': 0.7775370529438423, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.51} 0: {'loss': 0.6753, 'grad_norm': 0.7596054737739458, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.51} 0: | 3179/3723 [4:09:50<42:18, 4.67s/it] 85%|████████▌ | 3180/3723 [4:09:55<42:16, 4.67s/it] 85%|████████▌ | 3180/3723 [4:09:55<42:16, 4.67s/it] 85%|████████▌ | 3181/3723 [4:09:59<42:17, 4.68s/it] 85%|████████▌ | 3182/3723 [4:10:04<42:16, 4.69s/it] 85%|████████▌ | 3183/3723 [4:10:09<42:16, 4.70s/it] 86%|████████▌ | 3184/3723 [4:10:13<42:14, 4.70s/it] 86%|████████▌ | 3185/3723 [4:10:18<42:12, 4.71s/it] 86%|████████▌ | 3186/3723 [4:10:23<42:11, 4.71s/it] 86%|████████▌ | 3187/3723 [4:10:28<42:09, 4.72s/it] 86%|████████▌ | 3188/3723 [4:10:32<41:53, 4.70s/it] 86%|████████▌ | 3189/3723 [4:10:37<41:50, 4.70s/it] 86%|████████▌ | 3190/3723 [4:10:42<42:16, 4.76s/it] 86%|█ 0: {'loss': 0.6759, 'grad_norm': 1.0601562731269645, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.52} 0: ██████▌ | 3190/3723 [4:10:42<42:16, 4.76s/it] 86%|████████▌ | 3191/3723 [4:10:47<42:37, 4.81s/it] 86%|████████▌ | 3192/3723 [4:10:52<42:45, 4.83s/it] 86%|████████▌ | 3193/3723 [4:10:57<42:47, 4.84s/it] 86%|████████▌ | 3194/3723 [4:11:01<42:40, 4.84s/it] 86%|████████▌ | 3195/3723 [4:11:06<42:38, 4.85s/it] 86%|████████▌ | 3196/3723 [4:11:11<42:35, 4.85s/it] 86%|████████▌ | 3197/3723 [4:11:16<42:33, 4.85s/it] 86%|████████▌ | 3198/3723 [4:11:21<42:27, 4.85s/it] 86%|████████▌ | 3199/3723 [4:11:25<41:56, 4.80s/it] 86%|████████▌ | 3200/3723 [4:11:30<41:25, 4.75s/it] 86%|████████▌ | 3200/3723 [4:11:30<41:25, 4.75s/it] 86%|████████▌ | 3201/3723 [4:11:35<41:29, 4.77s/it] 86%|████████▌ | 3202/372 0: {'loss': 0.6915, 'grad_norm': 0.8468555391612727, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.52} 0: 3 [4:11:40<41:27, 4.77s/it] 86%|████████▌ | 3203/3723 [4:11:44<41:25, 4.78s/it] 86%|████████▌ | 3204/3723 [4:11:49<41:23, 4.79s/it] 86%|████████▌ | 3205/3723 [4:11:54<41:19, 4.79s/it] 86%|████████▌ | 3206/3723 [4:11:59<40:59, 4.76s/it] 86%|████████▌ | 3207/3723 [4:12:04<41:04, 4.78s/it] 86%|████████▌ | 3208/3723 [4:12:08<41:04, 4.78s/it] 86%|████████▌ | 3209/3723 [4:12:13<41:01, 4.79s/it] 86%|████████▌ | 3210/3723 [4:12:18<40:56, 4.79s/it] 86%|████████▌ | 3210/3723 [4:12:18<40:56, 4.79s/it] 86%|████████▌ | 3211/3723 [4:12:23<40:49, 4.78s/it] 86%|████████▋ | 3212/3723 [4:12:28<40:41, 4.78s/it] 86%|████████▋ | 3213/3723 [4:12:32<40:38, 4.78s/it] 86%|████████▋ | 3214/3723 [4:12:37<40:29, 4.77s/it] 86% 0: {'loss': 0.6648, 'grad_norm': 0.796356450327162, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.52} 0: |████████▋ | 3215/3723 [4:12:42<40:23, 4.77s/it] 86%|████████▋ | 3216/3723 [4:12:46<40:03, 4.74s/it] 86%|████████▋ | 3217/3723 [4:12:51<39:41, 4.71s/it] 86%|████████▋ | 3218/3723 [4:12:56<39:23, 4.68s/it] 86%|████████▋ | 3219/3723 [4:13:00<39:11, 4.67s/it] 86%|████████▋ | 3220/3723 [4:13:05<39:03, 4.66s/it] 86%|████████▋ | 3220/3723 [4:13:05<39:03, 4.66s/it] 87%|████████▋ | 3221/3723 [4:13:10<38:58, 4.66s/it] 87%|████████▋ | 3222/3723 [4:13:14<38:44, 4.64s/it] 87%|████████▋ | 3223/3723 [4:13:19<38:34, 4.63s/it] 87%|████████▋ | 3224/3723 [4:13:23<38:24, 4.62s/it] 87%|████████▋ | 3225/3723 [4:13:28<38:13, 4.61s/it] 87%|████████▋ | 3226/3723 [4:13:33<38:09, 4.61s/it] 87%|████████▋ | 32 0: {'loss': 0.6929, 'grad_norm': 0.801059303397338, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.52} 0: 27/3723 [4:13:37<38:07, 4.61s/it] 87%|████████▋ | 3228/3723 [4:13:42<38:03, 4.61s/it] 87%|████████▋ | 3229/3723 [4:13:46<37:55, 4.61s/it] 87%|████████▋ | 3230/3723 [4:13:51<38:07, 4.64s/it] 87%|████████▋ | 3230/3723 [4:13:51<38:07, 4.64s/it] 87%|████████▋ | 3231/3723 [4:13:56<37:58, 4.63s/it] 87%|████████▋ | 3232/3723 [4:14:00<37:54, 4.63s/it] 87%|████████▋ | 3233/3723 [4:14:05<37:50, 4.63s/it] 87%|████████▋ | 3234/3723 [4:14:10<37:56, 4.65s/it] 87%|████████▋ | 3235/3723 [4:14:15<38:03, 4.68s/it] 87%|████████▋ | 3236/3723 [4:14:19<37:46, 4.65s/it] 87%|████████▋ | 3237/3723 [4:14:24<37:31, 4.63s/it] 87%|████████▋ | 3238/3723 [4:14:28<37:21, 4.62s/it] 87%|████████▋ | 3239/3723 [4:14:33<37:12, 4.61s/it 0: {'loss': 0.6484, 'grad_norm': 0.7842725582329915, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.52} 0: {'loss': 0.6684, 'grad_norm': 0.843761621983302, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.52} 0: ] 87%|████████▋ | 3240/3723 [4:14:37<37:04, 4.61s/it] 87%|████████▋ | 3240/3723 [4:14:37<37:04, 4.61s/it] 87%|████████▋ | 3241/3723 [4:14:42<37:00, 4.61s/it] 87%|████████▋ | 3242/3723 [4:14:47<36:53, 4.60s/it] 87%|████████▋ | 3243/3723 [4:14:51<36:46, 4.60s/it] 87%|████████▋ | 3244/3723 [4:14:56<36:39, 4.59s/it] 87%|████████▋ | 3245/3723 [4:15:00<36:37, 4.60s/it] 87%|████████▋ | 3246/3723 [4:15:05<36:31, 4.59s/it] 87%|████████▋ | 3247/3723 [4:15:10<36:26, 4.59s/it] 87%|████████▋ | 3248/3723 [4:15:14<36:26, 4.60s/it] 87%|████████▋ | 3249/3723 [4:15:19<36:23, 4.61s/it] 87%|████████▋ | 3250/3723 [4:15:23<36:19, 4.61s/it] 87%|████████▋ | 3250/3723 [4: 0: {'loss': 0.681, 'grad_norm': 0.9429421373455769, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.53} 0: 15:24<36:19, 4.61s/it] 87%|████████▋ | 3251/3723 [4:15:28<36:19, 4.62s/it] 87%|████████▋ | 3252/3723 [4:15:33<36:17, 4.62s/it] 87%|████████▋ | 3253/3723 [4:15:37<36:12, 4.62s/it] 87%|████████▋ | 3254/3723 [4:15:42<36:12, 4.63s/it] 87%|████████▋ | 3255/3723 [4:15:47<36:06, 4.63s/it] 87%|████████▋ | 3256/3723 [4:15:51<36:06, 4.64s/it] 87%|████████▋ | 3257/3723 [4:15:56<36:05, 4.65s/it] 88%|████████▊ | 3258/3723 [4:16:01<36:01, 4.65s/it] 88%|████████▊ | 3259/3723 [4:16:05<35:56, 4.65s/it] 88%|████████▊ | 3260/3723 [4:16:10<35:51, 4.65s/it] 88%|████████▊ | 3260/3723 [4:16:10<35:51, 4.65s/it] 88%|████████▊ | 3261/3723 [4:16:15<35:45, 4.64s/it] 88%|████████▊ | 3262/3723 [4:16:19<35:37, 4.64s/it] 88%|█ 0: {'loss': 0.6602, 'grad_norm': 0.8340000146543267, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.53} 0: ██████▊ | 3263/3723 [4:16:24<35:29, 4.63s/it] 88%|████████▊ | 3264/3723 [4:16:28<35:28, 4.64s/it] 88%|████████▊ | 3265/3723 [4:16:33<35:26, 4.64s/it] 88%|████████▊ | 3266/3723 [4:16:38<35:25, 4.65s/it] 88%|████████▊ | 3267/3723 [4:16:42<35:21, 4.65s/it] 88%|████████▊ | 3268/3723 [4:16:47<35:16, 4.65s/it] 88%|████████▊ | 3269/3723 [4:16:52<35:13, 4.65s/it] 88%|████████▊ | 3270/3723 [4:16:56<35:10, 4.66s/it] 88%|████████▊ | 3270/3723 [4:16:56<35:10, 4.66s/it] 88%|████████▊ | 3271/3723 [4:17:01<35:07, 4.66s/it] 88%|████████▊ | 3272/3723 [4:17:06<35:09, 4.68s/it] 88%|████████▊ | 3273/3723 [4:17:10<35:07, 4.68s/it] 88%|████████▊ | 3274/3723 [4:17:15<35:03, 4.68s/it] 88%|████████▊ | 3275/37 0: {'loss': 0.6678, 'grad_norm': 0.8166168384696866, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.53} 0: 23 [4:17:20<34:59, 4.69s/it] 88%|████████▊ | 3276/3723 [4:17:25<34:53, 4.68s/it] 88%|████████▊ | 3277/3723 [4:17:29<34:49, 4.68s/it] 88%|████████▊ | 3278/3723 [4:17:34<34:43, 4.68s/it] 88%|████████▊ | 3279/3723 [4:17:39<34:40, 4.69s/it] 88%|████████▊ | 3280/3723 [4:17:43<34:37, 4.69s/it] 88%|████████▊ | 3280/3723 [4:17:43<34:37, 4.69s/it] 88%|████████▊ | 3281/3723 [4:17:48<34:37, 4.70s/it] 88%|████████▊ | 3282/3723 [4:17:53<34:35, 4.71s/it] 88%|████████▊ | 3283/3723 [4:17:57<34:28, 4.70s/it] 88%|████████▊ | 3284/3723 [4:18:02<34:18, 4.69s/it] 88%|████████▊ | 3285/3723 [4:18:07<34:07, 4.68s/it] 88%|████████▊ | 3286/3723 [4:18:11<33:56, 4.66s/it] 88%|████████▊ | 3287/3723 [4:18:16<33:52, 4.66s/it] 88 0: {'loss': 0.6656, 'grad_norm': 0.8437344406472406, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.53} 0: %|████████▊ | 3288/3723 [4:18:21<33:46, 4.66s/it] 88%|████████▊ | 3289/3723 [4:18:25<33:38, 4.65s/it] 88%|████████▊ | 3290/3723 [4:18:30<33:33, 4.65s/it] 88%|████████▊ | 3290/3723 [4:18:30<33:33, 4.65s/it] 88%|████████▊ | 3291/3723 [4:18:35<33:32, 4.66s/it] 88%|████████▊ | 3292/3723 [4:18:39<33:26, 4.65s/it] 88%|████████▊ | 3293/3723 [4:18:44<33:23, 4.66s/it] 88%|████████▊ | 3294/3723 [4:18:49<33:19, 4.66s/it] 89%|████████▊ | 3295/3723 [4:18:53<33:14, 4.66s/it] 89%|████████▊ | 3296/3723 [4:18:58<33:10, 4.66s/it] 89%|████████▊ | 3297/3723 [4:19:03<33:03, 4.66s/it] 89%|████████▊ | 3298/3723 [4:19:07<32:55, 4.65s/it] 89%|████████▊ | 3299/3723 [4:19:12<32:51, 4.65s/it] 89%|████████▊ | 3 0: {'loss': 0.6637, 'grad_norm': 1.0628824275673157, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.53} 0: {'loss': 0.659, 'grad_norm': 0.8569370781465188, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.53} 0: 300/3723 [4:19:17<32:45, 4.65s/it] 89%|████████▊ | 3300/3723 [4:19:17<32:45, 4.65s/it] 89%|████████▊ | 3301/3723 [4:19:21<32:42, 4.65s/it] 89%|████████▊ | 3302/3723 [4:19:26<32:36, 4.65s/it] 89%|████████▊ | 3303/3723 [4:19:30<32:31, 4.65s/it] 89%|████████▊ | 3304/3723 [4:19:35<32:24, 4.64s/it] 89%|████████▉ | 3305/3723 [4:19:40<32:23, 4.65s/it] 89%|████████▉ | 3306/3723 [4:19:44<32:21, 4.66s/it] 89%|████████▉ | 3307/3723 [4:19:49<32:21, 4.67s/it] 89%|████████▉ | 3308/3723 [4:19:54<32:13, 4.66s/it] 89%|████████▉ | 3309/3723 [4:19:58<32:02, 4.64s/it] 89%|████████▉ | 3310/3723 [4:20:03<31:50, 4.63s/it] 89%|████████▉ | 3310/3723 [4:20:03<31:50, 4.63s/it] 89%|███ 0: {'loss': 0.6755, 'grad_norm': 0.8002703042275362, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.53} 0: █████▉ | 3311/3723 [4:20:08<31:45, 4.63s/it] 89%|████████▉ | 3312/3723 [4:20:12<31:36, 4.61s/it] 89%|████████▉ | 3313/3723 [4:20:17<31:29, 4.61s/it] 89%|████████▉ | 3314/3723 [4:20:21<31:38, 4.64s/it] 89%|████████▉ | 3315/3723 [4:20:26<31:53, 4.69s/it] 89%|████████▉ | 3316/3723 [4:20:31<32:03, 4.73s/it] 89%|████████▉ | 3317/3723 [4:20:36<32:36, 4.82s/it] 89%|████████▉ | 3318/3723 [4:20:41<32:34, 4.83s/it] 89%|████████▉ | 3319/3723 [4:20:46<32:23, 4.81s/it] 89%|████████▉ | 3320/3723 [4:20:51<32:15, 4.80s/it] 89%|████████▉ | 3320/3723 [4:20:51<32:15, 4.80s/it] 89%|████████▉ | 3321/3723 [4:20:55<32:06, 4.79s/it] 89%|████████▉ | 3322/3723 [4:21:00<31:51, 4.77s/it] 89%|████████▉ | 3323/3723 [4 0: {'loss': 0.6566, 'grad_norm': 0.7898390314869608, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.54} 0: :21:05<31:46, 4.77s/it] 89%|████████▉ | 3324/3723 [4:21:10<31:42, 4.77s/it] 89%|████████▉ | 3325/3723 [4:21:14<31:26, 4.74s/it] 89%|████████▉ | 3326/3723 [4:21:19<31:14, 4.72s/it] 89%|████████▉ | 3327/3723 [4:21:24<31:01, 4.70s/it] 89%|████████▉ | 3328/3723 [4:21:28<30:49, 4.68s/it] 89%|████████▉ | 3329/3723 [4:21:33<30:40, 4.67s/it] 89%|████████▉ | 3330/3723 [4:21:37<30:34, 4.67s/it] 89%|████████▉ | 3330/3723 [4:21:38<30:34, 4.67s/it] 89%|████████▉ | 3331/3723 [4:21:42<30:25, 4.66s/it] 89%|████████▉ | 3332/3723 [4:21:47<30:21, 4.66s/it] 90%|████████▉ | 3333/3723 [4:21:51<30:17, 4.66s/it] 90%|████████▉ | 3334/3723 [4:21:56<30:15, 4.67s/it] 90%|████████▉ | 3335/3723 [4:22:01<30:14, 4.68s/it] 90%|█ 0: {'loss': 0.671, 'grad_norm': 0.8881578194015649, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.54} 0: ███████▉ | 3336/3723 [4:22:06<30:09, 4.68s/it] 90%|████████▉ | 3337/3723 [4:22:10<30:08, 4.68s/it] 90%|████████▉ | 3338/3723 [4:22:15<30:04, 4.69s/it] 90%|████████▉ | 3339/3723 [4:22:20<30:00, 4.69s/it] 90%|████████▉ | 3340/3723 [4:22:24<30:00, 4.70s/it] 90%|████████▉ | 3340/3723 [4:22:24<30:00, 4.70s/it] 90%|████████▉ | 3341/3723 [4:22:29<29:58, 4.71s/it] 90%|████████▉ | 3342/3723 [4:22:34<29:50, 4.70s/it] 90%|████████▉ | 3343/3723 [4:22:38<29:43, 4.69s/it] 90%|████████▉ | 3344/3723 [4:22:43<29:37, 4.69s/it] 90%|████████▉ | 3345/3723 [4:22:48<29:40, 4.71s/it] 90%|████████▉ | 3346/3723 [4:22:53<29:41, 4.72s/it] 90%|████████▉ | 3347/3723 [4:22:57<29:38, 4.73s/it] 90%|████████▉ | 3348/3 0: {'loss': 0.6633, 'grad_norm': 0.8625907250998852, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.54} 0: {'loss': 0.6661, 'grad_norm': 0.8048837915569498, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.54} 0: 723 [4:23:02<29:34, 4.73s/it] 90%|████████▉ | 3349/3723 [4:23:07<29:29, 4.73s/it] 90%|████████▉ | 3350/3723 [4:23:12<29:27, 4.74s/it] 90%|████████▉ | 3350/3723 [4:23:12<29:27, 4.74s/it] 90%|█████████ | 3351/3723 [4:23:16<29:25, 4.75s/it] 90%|█████████ | 3352/3723 [4:23:21<29:20, 4.75s/it] 90%|█████████ | 3353/3723 [4:23:26<29:17, 4.75s/it] 90%|█████████ | 3354/3723 [4:23:31<29:12, 4.75s/it] 90%|█████████ | 3355/3723 [4:23:35<29:00, 4.73s/it] 90%|█████████ | 3356/3723 [4:23:40<28:53, 4.72s/it] 90%|█████████ | 3357/3723 [4:23:45<28:49, 4.73s/it] 90%|█████████ | 3358/3723 [4:23:49<28:50, 4.74s/it] 90%|█████████ | 3359/3723 [4:23:54<28:47, 4.75s/it] 90%|█████████ | 3360/3723 [4:23:59<28:43, 4.75s/it] 0: {'loss': 0.6618, 'grad_norm': 0.834393087699382, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.54} 0: 90%|█████████ | 3360/3723 [4:23:59<28:43, 4.75s/it] 90%|█████████ | 3361/3723 [4:24:04<28:43, 4.76s/it] 90%|█████████ | 3362/3723 [4:24:09<28:41, 4.77s/it] 90%|█████████ | 3363/3723 [4:24:13<28:29, 4.75s/it] 90%|█████████ | 3364/3723 [4:24:18<28:20, 4.74s/it] 90%|█████████ | 3365/3723 [4:24:23<28:08, 4.72s/it] 90%|█████████ | 3366/3723 [4:24:27<28:03, 4.71s/it] 90%|█████████ | 3367/3723 [4:24:32<27:57, 4.71s/it] 90%|█████████ | 3368/3723 [4:24:37<27:54, 4.72s/it] 90%|█████████ | 3369/3723 [4:24:42<27:49, 4.72s/it] 91%|█████████ | 3370/3723 [4:24:46<27:40, 4.70s/it] 91%|█████████ | 3370/3723 [4:24:46<27:40, 4.70s/it] 91%|█████████ | 3371/3723 [4:24:5 0: {'loss': 0.6538, 'grad_norm': 0.8092609669613451, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.54} 0: 1<27:40, 4.72s/it] 91%|█████████ | 3372/3723 [4:24:56<27:35, 4.72s/it] 91%|█████████ | 3373/3723 [4:25:00<27:28, 4.71s/it] 91%|█████████ | 3374/3723 [4:25:05<27:19, 4.70s/it] 91%|█████████ | 3375/3723 [4:25:10<27:12, 4.69s/it] 91%|█████████ | 3376/3723 [4:25:14<27:13, 4.71s/it] 91%|█████████ | 3377/3723 [4:25:19<27:12, 4.72s/it] 91%|█████████ | 3378/3723 [4:25:24<27:09, 4.72s/it] 91%|█████████ | 3379/3723 [4:25:29<27:07, 4.73s/it] 91%|█████████ | 3380/3723 [4:25:33<27:03, 4.73s/it] 91%|█████████ | 3380/3723 [4:25:33<27:03, 4.73s/it] 91%|█████████ | 3381/3723 [4:25:38<26:57, 4.73s/it] 91%|█████████ | 3382/3723 [4:25:43<26:49, 4.72s/it] 91%|█████████ | 3383/3723 [4:25:48<26:43, 4.72s/it] 91%|██ 0: {'loss': 0.663, 'grad_norm': 0.9050937244908254, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.55} 0: ██████ | 3384/3723 [4:25:52<26:37, 4.71s/it] 91%|█████████ | 3385/3723 [4:25:57<26:35, 4.72s/it] 91%|█████████ | 3386/3723 [4:26:02<26:32, 4.73s/it] 91%|█████████ | 3387/3723 [4:26:06<26:27, 4.72s/it] 91%|█████████ | 3388/3723 [4:26:11<26:21, 4.72s/it] 91%|█████████ | 3389/3723 [4:26:16<26:21, 4.73s/it] 91%|█████████ | 3390/3723 [4:26:21<26:22, 4.75s/it] 91%|█████████ | 3390/3723 [4:26:21<26:22, 4.75s/it] 91%|█████████ | 3391/3723 [4:26:26<26:24, 4.77s/it] 91%|█████████ | 3392/3723 [4:26:30<26:22, 4.78s/it] 91%|█████████ | 3393/3723 [4:26:35<26:18, 4.78s/it] 91%|█████████ | 3394/3723 [4:26:40<26:21, 4.81s/it] 91%|█████████ | 3395/3723 [4:26:45<26:15, 4.80s/it] 91%|█████████ | 3396/3723 [ 0: {'loss': 0.687, 'grad_norm': 0.8714819133887279, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.55} 0: 4:26:50<26:10, 4.80s/it] 91%|█████████ | 3397/3723 [4:26:54<26:06, 4.81s/it] 91%|█████████▏| 3398/3723 [4:26:59<26:01, 4.80s/it] 91%|█████████▏| 3399/3723 [4:27:04<25:55, 4.80s/it] 91%|█████████▏| 3400/3723 [4:27:09<25:49, 4.80s/it] 91%|█████████▏| 3400/3723 [4:27:09<25:49, 4.80s/it] 91%|█████████▏| 3401/3723 [4:27:14<25:46, 4.80s/it] 91%|█████████▏| 3402/3723 [4:27:18<25:40, 4.80s/it] 91%|█████████▏| 3403/3723 [4:27:23<25:31, 4.79s/it] 91%|█████████▏| 3404/3723 [4:27:28<25:23, 4.77s/it] 91%|█████████▏| 3405/3723 [4:27:33<25:18, 4.77s/it] 91%|█████████▏| 3406/3723 [4:27:37<25:16, 4.78s/it] 92%|█████████▏| 3407/3723 [4:27:42<25:21, 4.82s/it] 92%|█████████▏| 3408/3723 [4:27:47<2 0: {'loss': 0.6665, 'grad_norm': 0.7876577440414064, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.55} 0: 5:18, 4.82s/it] 92%|█████████▏| 3409/3723 [4:27:52<25:11, 4.81s/it] 92%|█████████▏| 3410/3723 [4:27:57<25:04, 4.81s/it] 92%|█████████▏| 3410/3723 [4:27:57<25:04, 4.81s/it] 92%|█████████▏| 3411/3723 [4:28:02<24:56, 4.80s/it] 92%|█████████▏| 3412/3723 [4:28:06<24:52, 4.80s/it] 92%|█████████▏| 3413/3723 [4:28:11<24:56, 4.83s/it] 92%|█████████▏| 3414/3723 [4:28:16<24:46, 4.81s/it] 92%|█████████▏| 3415/3723 [4:28:21<24:37, 4.80s/it] 92%|█████████▏| 3416/3723 [4:28:26<24:31, 4.79s/it] 92%|█████████▏| 3417/3723 [4:28:30<24:34, 4.82s/it] 92%|█████████▏| 3418/3723 [4:28:35<24:26, 4.81s/it] 92%|█████████▏| 3419/3723 [4:28:40<24:18, 4.80s/it] 92%|█████████▏| 3420/3723 [4:28:45<24:11, 0: {'loss': 0.6566, 'grad_norm': 0.8096587926270113, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.55} 0: {'loss': 0.6656, 'grad_norm': 0.7837022699099949, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.55} 0: 4.79s/it] 92%|█████████▏| 3420/3723 [4:28:45<24:11, 4.79s/it] 92%|█████████▏| 3421/3723 [4:28:50<24:07, 4.79s/it] 92%|█████████▏| 3422/3723 [4:28:54<23:59, 4.78s/it] 92%|█████████▏| 3423/3723 [4:28:59<23:48, 4.76s/it] 92%|█████████▏| 3424/3723 [4:29:04<23:39, 4.75s/it] 92%|█████████▏| 3425/3723 [4:29:08<23:33, 4.74s/it] 92%|█████████▏| 3426/3723 [4:29:13<23:37, 4.77s/it] 92%|█████████▏| 3427/3723 [4:29:18<23:26, 4.75s/it] 92%|█████████▏| 3428/3723 [4:29:23<23:12, 4.72s/it] 92%|█████████▏| 3429/3723 [4:29:27<23:00, 4.69s/it] 92%|█████████▏| 3430/3723 [4:29:32<22:52, 4.69s/it] 92%|█████████▏| 3430/3723 [4:29:32<22:52, 4.69s/it] 92%|███ 0: {'loss': 0.6752, 'grad_norm': 0.7907555329692371, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.55} 0: █████▏| 3431/3723 [4:29:37<22:51, 4.70s/it] 92%|█████████▏| 3432/3723 [4:29:41<22:49, 4.70s/it] 92%|█████████▏| 3433/3723 [4:29:46<22:48, 4.72s/it] 92%|█████████▏| 3434/3723 [4:29:51<22:45, 4.72s/it] 92%|█████████▏| 3435/3723 [4:29:56<22:34, 4.70s/it] 92%|█████████▏| 3436/3723 [4:30:00<22:26, 4.69s/it] 92%|█████████▏| 3437/3723 [4:30:05<22:20, 4.69s/it] 92%|█████████▏| 3438/3723 [4:30:10<22:12, 4.68s/it] 92%|█████████▏| 3439/3723 [4:30:14<22:07, 4.67s/it] 92%|█████████▏| 3440/3723 [4:30:19<22:01, 4.67s/it] 92%|█████████▏| 3440/3723 [4:30:19<22:01, 4.67s/it] 92%|█████████▏| 3441/3723 [4:30:24<21:57, 4.67s/it] 92%|█████████▏| 3442/3723 [4:30:28<21:53, 4.67s/it] 92%|██████ 0: {'loss': 0.6605, 'grad_norm': 0.7966141196448446, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.56} 0: ███▏| 3443/3723 [4:30:33<21:47, 4.67s/it] 93%|█████████▎| 3444/3723 [4:30:38<21:44, 4.68s/it] 93%|█████████▎| 3445/3723 [4:30:42<21:48, 4.71s/it] 93%|█████████▎| 3446/3723 [4:30:47<21:49, 4.73s/it] 93%|█████████▎| 3447/3723 [4:30:52<21:46, 4.73s/it] 93%|█████████▎| 3448/3723 [4:30:57<21:42, 4.74s/it] 93%|█████████▎| 3449/3723 [4:31:01<21:40, 4.75s/it] 93%|█████████▎| 3450/3723 [4:31:06<21:36, 4.75s/it] 93%|█████████▎| 3450/3723 [4:31:06<21:36, 4.75s/it] 93%|█████████▎| 3451/3723 [4:31:11<21:31, 4.75s/it] 93%|█████████▎| 3452/3723 [4:31:16<21:24, 4.74s/it] 93%|█████████▎| 3453/3723 [4:31:20<21:19, 4.74s/it] 93%|█████████▎| 3454/3723 [4:31:25<21:11, 4.73s/it] 93%|████████ 0: {'loss': 0.6682, 'grad_norm': 1.070136998264358, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.56} 0: ▎| 3455/3723 [4:31:30<21:02, 4.71s/it] 93%|█████████▎| 3456/3723 [4:31:34<20:55, 4.70s/it] 93%|█████████▎| 3457/3723 [4:31:39<20:48, 4.69s/it] 93%|█████████▎| 3458/3723 [4:31:44<20:41, 4.69s/it] 93%|█████████▎| 3459/3723 [4:31:48<20:35, 4.68s/it] 93%|█████████▎| 3460/3723 [4:31:53<20:29, 4.68s/it] 93%|█████████▎| 3460/3723 [4:31:53<20:29, 4.68s/it] 93%|█████████▎| 3461/3723 [4:31:58<20:24, 4.67s/it] 93%|█████████▎| 3462/3723 [4:32:02<20:19, 4.67s/it] 93%|█████████▎| 3463/3723 [4:32:07<20:13, 4.67s/it] 93%|█████████▎| 3464/3723 [4:32:12<20:12, 4.68s/it] 93%|█████████▎| 3465/3723 [4:32:16<20:07, 4.68s/it] 93%|█████████▎| 3466/3723 [4:32:21<20:02, 4.68s/it] 93%|█████████▎| 0: {'loss': 0.6666, 'grad_norm': 4.361719207409412, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.56} 0: 3467/3723 [4:32:26<19:58, 4.68s/it] 93%|█████████▎| 3468/3723 [4:32:31<19:55, 4.69s/it] 93%|█████████▎| 3469/3723 [4:32:35<19:52, 4.69s/it] 93%|█████████▎| 3470/3723 [4:32:40<19:47, 4.70s/it] 93%|█████████▎| 3470/3723 [4:32:40<19:47, 4.70s/it] 93%|█████████▎| 3471/3723 [4:32:45<19:41, 4.69s/it] 93%|█████████▎| 3472/3723 [4:32:49<19:37, 4.69s/it] 93%|█████████▎| 3473/3723 [4:32:54<19:33, 4.70s/it] 93%|█████████▎| 3474/3723 [4:32:59<19:30, 4.70s/it] 93%|█████████▎| 3475/3723 [4:33:03<19:24, 4.70s/it] 93%|█████████▎| 3476/3723 [4:33:08<19:20, 4.70s/it] 93%|█████████▎| 3477/3723 [4:33:13<19:12, 4.69s/it] 93%|█████████▎| 3478/3723 [4:33:17<19:08, 4.69s/it] 93%|█████████▎| 3479/37 0: {'loss': 0.6846, 'grad_norm': 0.9755125413881576, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.56} 0: 23 [4:33:22<19:03, 4.69s/it] 93%|█████████▎| 3480/3723 [4:33:27<19:01, 4.70s/it] 93%|█████████▎| 3480/3723 [4:33:27<19:01, 4.70s/it] 93%|█████████▎| 3481/3723 [4:33:32<18:58, 4.70s/it] 94%|█████████▎| 3482/3723 [4:33:36<18:53, 4.70s/it] 94%|█████████▎| 3483/3723 [4:33:41<18:49, 4.71s/it] 94%|█████████▎| 3484/3723 [4:33:46<18:46, 4.71s/it] 94%|█████████▎| 3485/3723 [4:33:50<18:41, 4.71s/it] 94%|█████████▎| 3486/3723 [4:33:55<18:37, 4.72s/it] 94%|█████████▎| 3487/3723 [4:34:00<18:33, 4.72s/it] 94%|█████████▎| 3488/3723 [4:34:05<18:27, 4.71s/it] 94%|█████████▎| 3489/3723 [4:34:09<18:23, 4.72s/it] 94%|█████████▎| 3490/3723 [4:34:14<18:18, 4.72s/it] 0: {'loss': 0.685, 'grad_norm': 0.892476854844791, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.56} 0: {'loss': 0.6641, 'grad_norm': 0.8724228749955243, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.56} 0: 94%|█████████▎| 3490/3723 [4:34:14<18:18, 4.72s/it] 94%|█████████▍| 3491/3723 [4:34:19<18:16, 4.73s/it] 94%|█████████▍| 3492/3723 [4:34:24<18:13, 4.73s/it] 94%|█████████▍| 3493/3723 [4:34:28<18:08, 4.73s/it] 94%|█████████▍| 3494/3723 [4:34:33<18:03, 4.73s/it] 94%|█████████▍| 3495/3723 [4:34:38<17:58, 4.73s/it] 94%|█████████▍| 3496/3723 [4:34:42<17:54, 4.73s/it] 94%|█████████▍| 3497/3723 [4:34:47<17:49, 4.73s/it] 94%|█████████▍| 3498/3723 [4:34:52<17:43, 4.73s/it] 94%|█████████▍| 3499/3723 [4:34:57<17:33, 4.70s/it] 94%|█████████▍| 3500/3723 [4:35:01<17:25, 4.69s/it] 94%|█████████▍| 3500/3723 [4:35:01<17:25, 4.69s/it] 94%|█████████▍| 3501/3723 [4:35:06<17:17, 4.67s/it] 94 0: {'loss': 0.6732, 'grad_norm': 0.7961385705163336, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.57} 0: %|█████████▍| 3502/3723 [4:35:10<17:10, 4.66s/it] 94%|█████████▍| 3503/3723 [4:35:15<17:04, 4.66s/it] 94%|█████████▍| 3504/3723 [4:35:20<17:04, 4.68s/it] 94%|█████████▍| 3505/3723 [4:35:25<16:59, 4.68s/it] 94%|█████████▍| 3506/3723 [4:35:29<16:55, 4.68s/it] 94%|█████████▍| 3507/3723 [4:35:34<16:49, 4.67s/it] 94%|█████████▍| 3508/3723 [4:35:39<16:47, 4.69s/it] 94%|█████████▍| 3509/3723 [4:35:43<16:44, 4.69s/it] 94%|█████████▍| 3510/3723 [4:35:48<16:43, 4.71s/it] 94%|█████████▍| 3510/3723 [4:35:48<16:43, 4.71s/it] 94%|█████████▍| 3511/3723 [4:35:53<16:41, 4.72s/it] 94%|█████████▍| 3512/3723 [4:35:58<16:37, 4.73s/it] 94%|█████████▍| 3513/3723 [4:36:02<16:33, 4.73s/it] 94%|█ 0: {'loss': 0.6565, 'grad_norm': 0.9483500910518716, 'learning_rate': 2e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.57} 0: ███████▍| 3514/3723 [4:36:07<16:31, 4.75s/it] 94%|█████████▍| 3515/3723 [4:36:12<16:35, 4.78s/it] 94%|█████████▍| 3516/3723 [4:36:17<16:37, 4.82s/it] 94%|█████████▍| 3517/3723 [4:36:22<16:34, 4.83s/it] 94%|█████████▍| 3518/3723 [4:36:26<16:24, 4.80s/it] 95%|█████████▍| 3519/3723 [4:36:31<16:14, 4.78s/it] 95%|█████████▍| 3520/3723 [4:36:36<16:06, 4.76s/it] 95%|█████████▍| 3520/3723 [4:36:36<16:06, 4.76s/it] 95%|█████████▍| 3521/3723 [4:36:41<16:03, 4.77s/it] 95%|█████████▍| 3522/3723 [4:36:45<15:57, 4.76s/it] 95%|█████████▍| 3523/3723 [4:36:50<15:51, 4.76s/it] 95%|█████████▍| 3524/3723 [4:36:55<15:46, 4.75s/it] 95%|█████████▍| 3525/3723 [4:37:00<15:41, 4.75s/it] 95%|████ 0: {'loss': 0.6888, 'grad_norm': 0.8467894048752771, 'learning_rate': 1.996005768142772e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.57} 0: █████▍| 3526/3723 [4:37:04<15:38, 4.76s/it] 95%|█████████▍| 3527/3723 [4:37:09<15:32, 4.76s/it] 95%|█████████▍| 3528/3723 [4:37:14<15:27, 4.76s/it] 95%|█████████▍| 3529/3723 [4:37:19<15:22, 4.76s/it] 95%|█████████▍| 3530/3723 [4:37:23<15:16, 4.75s/it] 95%|█████████▍| 3530/3723 [4:37:23<15:16, 4.75s/it] 95%|█████████▍| 3531/3723 [4:37:28<15:09, 4.74s/it] 95%|█████████▍| 3532/3723 [4:37:33<15:01, 4.72s/it] 95%|█████████▍| 3533/3723 [4:37:37<14:54, 4.71s/it] 95%|█████████▍| 3534/3723 [4:37:42<14:48, 4.70s/it] 95%|█████████▍| 3535/3723 [4:37:47<14:43, 4.70s/it] 95%|█████████▍| 3536/3723 [4:37:51<14:34, 4.68s/it] 95%|█████████▌| 3537/3723 [4:37:56<14:26, 4.66s/it] 95%|██████ 0: {'loss': 0.6663, 'grad_norm': 0.7712842954584606, 'learning_rate': 1.9717248450157682e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.57} 0: ██▌| 3538/3723 [4:38:01<14:20, 4.65s/it] 95%|█████████▌| 3539/3723 [4:38:05<14:13, 4.64s/it] 95%|█████████▌| 3540/3723 [4:38:10<14:09, 4.64s/it] 95%|█████████▌| 3540/3723 [4:38:10<14:09, 4.64s/it] 95%|█████████▌| 3541/3723 [4:38:15<14:07, 4.66s/it] 95%|█████████▌| 3542/3723 [4:38:19<14:05, 4.67s/it] 95%|█████████▌| 3543/3723 [4:38:24<14:01, 4.67s/it] 95%|█████████▌| 3544/3723 [4:38:29<13:58, 4.68s/it] 95%|█████████▌| 3545/3723 [4:38:33<13:54, 4.69s/it] 95%|█████████▌| 3546/3723 [4:38:38<13:47, 4.68s/it] 95%|█████████▌| 3547/3723 [4:38:43<13:41, 4.67s/it] 95%|█████████▌| 3548/3723 [4:38:47<13:38, 4.68s/it] 95%|█████████▌| 3549/3723 [4:38:52<13:33, 4.68s/it] 95%|████████ 0: {'loss': 0.6674, 'grad_norm': 0.8153579933116455, 'learning_rate': 1.925979163115583e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.57} 0: {'loss': 0.6689, 'grad_norm': 0.8750347732125076, 'learning_rate': 1.859895132951814e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.57} 0: ▌| 3550/3723 [4:38:57<13:26, 4.66s/it] 95%|█████████▌| 3550/3723 [4:38:57<13:26, 4.66s/it] 95%|█████████▌| 3551/3723 [4:39:01<13:20, 4.66s/it] 95%|█████████▌| 3552/3723 [4:39:06<13:15, 4.65s/it] 95%|█████████▌| 3553/3723 [4:39:11<13:09, 4.65s/it] 95%|█████████▌| 3554/3723 [4:39:15<13:03, 4.64s/it] 95%|█████████▌| 3555/3723 [4:39:20<12:58, 4.64s/it] 96%|█████████▌| 3556/3723 [4:39:25<12:56, 4.65s/it] 96%|█████████▌| 3557/3723 [4:39:29<12:52, 4.65s/it] 96%|█████████▌| 3558/3723 [4:39:34<12:48, 4.66s/it] 96%|█████████▌| 3559/3723 [4:39:39<12:45, 4.67s/it] 96%|█████████▌| 3560/3723 [4:39:43<12:41, 4.67s/it] 96%|█████████▌| 3560/3723 [4:39:43<1 0: {'loss': 0.6582, 'grad_norm': 0.7855173964500335, 'learning_rate': 1.775099962667414e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.58} 0: 2:41, 4.67s/it] 96%|█████████▌| 3561/3723 [4:39:48<12:38, 4.68s/it] 96%|█████████▌| 3562/3723 [4:39:53<12:35, 4.69s/it] 96%|█████████▌| 3563/3723 [4:39:57<12:30, 4.69s/it] 96%|█████████▌| 3564/3723 [4:40:02<12:27, 4.70s/it] 96%|█████████▌| 3565/3723 [4:40:07<12:23, 4.71s/it] 96%|█████████▌| 3566/3723 [4:40:12<12:18, 4.70s/it] 96%|█████████▌| 3567/3723 [4:40:16<12:12, 4.70s/it] 96%|█████████▌| 3568/3723 [4:40:21<12:07, 4.69s/it] 96%|█████████▌| 3569/3723 [4:40:26<12:02, 4.69s/it] 96%|█████████▌| 3570/3723 [4:40:30<11:58, 4.70s/it] 96%|█████████▌| 3570/3723 [4:40:30<11:58, 4.70s/it] 96%|█████████▌| 3571/3723 [4:40:35<11:54, 4.70s/it] 96%|█████████▌| 3572/3723 [4:40:40<11:50, 0: {'loss': 0.6688, 'grad_norm': 0.8212577796732449, 'learning_rate': 1.673681590773821e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.58} 0: 4.71s/it] 96%|█████████▌| 3573/3723 [4:40:45<11:47, 4.71s/it] 96%|█████████▌| 3574/3723 [4:40:49<11:42, 4.71s/it] 96%|█████████▌| 3575/3723 [4:40:54<11:36, 4.71s/it] 96%|█████████▌| 3576/3723 [4:40:59<11:31, 4.70s/it] 96%|█████████▌| 3577/3723 [4:41:03<11:25, 4.69s/it] 96%|█████████▌| 3578/3723 [4:41:08<11:18, 4.68s/it] 96%|█████████▌| 3579/3723 [4:41:13<11:13, 4.68s/it] 96%|█████████▌| 3580/3723 [4:41:17<11:06, 4.66s/it] 96%|█████████▌| 3580/3723 [4:41:17<11:06, 4.66s/it] 96%|█████████▌| 3581/3723 [4:41:22<10:58, 4.64s/it] 96%|█████████▌| 3582/3723 [4:41:26<10:51, 4.62s/it] 96%|█████████▌| 3583/3723 [4:41:31<10:45, 4.61s/it] 96%|█████████▋| 3584/3723 [4:41:36<10:40, 4.61s/i 0: {'loss': 0.6631, 'grad_norm': 0.7517300812958864, 'learning_rate': 1.558137274175334e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.58} 0: t] 96%|█████████▋| 3585/3723 [4:41:40<10:36, 4.61s/it] 96%|█████████▋| 3586/3723 [4:41:45<10:31, 4.61s/it] 96%|█████████▋| 3587/3723 [4:41:49<10:28, 4.62s/it] 96%|█████████▋| 3588/3723 [4:41:54<10:23, 4.62s/it] 96%|█████████▋| 3589/3723 [4:41:59<10:19, 4.62s/it] 96%|█████████▋| 3590/3723 [4:42:03<10:14, 4.62s/it] 96%|█████████▋| 3590/3723 [4:42:03<10:14, 4.62s/it] 96%|█████████▋| 3591/3723 [4:42:08<10:10, 4.63s/it] 96%|█████████▋| 3592/3723 [4:42:13<10:05, 4.62s/it] 97%|█████████▋| 3593/3723 [4:42:17<09:59, 4.62s/it] 97%|█████████▋| 3594/3723 [4:42:22<09:55, 4.62s/it] 97%|█████████▋| 3595/3723 [4:42:26<09:51, 4.62s/it] 97%|█████████▋| 3596/3723 [4:42:31<09:46, 4.62s/it] 97% 0: {'loss': 0.659, 'grad_norm': 0.8549667028009815, 'learning_rate': 1.4313120974162105e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.58} 0: |█████████▋| 3597/3723 [4:42:36<09:40, 4.61s/it] 97%|█████████▋| 3598/3723 [4:42:40<09:36, 4.61s/it] 97%|█████████▋| 3599/3723 [4:42:45<09:31, 4.61s/it] 97%|█████████▋| 3600/3723 [4:42:49<09:27, 4.62s/it] 97%|█████████▋| 3600/3723 [4:42:49<09:27, 4.62s/it] 97%|█████████▋| 3601/3723 [4:42:54<09:24, 4.63s/it] 97%|█████████▋| 3602/3723 [4:42:59<09:19, 4.63s/it] 97%|█████████▋| 3603/3723 [4:43:03<09:14, 4.62s/it] 97%|█████████▋| 3604/3723 [4:43:08<09:10, 4.63s/it] 97%|█████████▋| 3605/3723 [4:43:13<09:06, 4.63s/it] 97%|█████████▋| 3606/3723 [4:43:17<09:02, 4.64s/it] 97%|█████████▋| 3607/3723 [4:43:22<08:58, 4.64s/it] 97%|█████████▋| 3608/3723 [4:43:27<08:54, 4.65s/it] 97%|██ 0: {'loss': 0.6541, 'grad_norm': 0.7763102873175303, 'learning_rate': 1.2963289172568885e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.58} 0: ███████▋| 3609/3723 [4:43:31<08:49, 4.65s/it] 97%|█████████▋| 3610/3723 [4:43:36<08:44, 4.64s/it] 97%|█████████▋| 3610/3723 [4:43:36<08:44, 4.64s/it] 97%|█████████▋| 3611/3723 [4:43:41<08:39, 4.64s/it] 97%|█████████▋| 3612/3723 [4:43:45<08:34, 4.64s/it] 97%|█████████▋| 3613/3723 [4:43:50<08:30, 4.64s/it] 97%|█████████▋| 3614/3723 [4:43:54<08:25, 4.63s/it] 97%|█████████▋| 3615/3723 [4:43:59<08:20, 4.63s/it] 97%|█████████▋| 3616/3723 [4:44:04<08:15, 4.63s/it] 97%|█████████▋| 3617/3723 [4:44:08<08:12, 4.64s/it] 97%|█████████▋| 3618/3723 [4:44:13<08:09, 4.66s/it] 97%|█████████▋| 3619/3723 [4:44:18<08:03, 4.65s/it] 97%|█████████▋| 3620/3723 [4:44:22<07:58, 4.64s/it] 0: {'loss': 0.656, 'grad_norm': 0.7177784251532862, 'learning_rate': 1.1565114675763823e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.58} 0: {'loss': 0.6631, 'grad_norm': 0.7852227797342748, 'learning_rate': 1.0153025180133372e-05, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.58} 0: 97%|█████████▋| 3620/3723 [4:44:22<07:58, 4.64s/it] 97%|█████████▋| 3621/3723 [4:44:27<07:53, 4.65s/it] 97%|█████████▋| 3622/3723 [4:44:32<07:48, 4.64s/it] 97%|█████████▋| 3623/3723 [4:44:36<07:51, 4.72s/it] 97%|█████████▋| 3624/3723 [4:44:41<07:45, 4.70s/it] 97%|█████████▋| 3625/3723 [4:44:46<07:38, 4.68s/it] 97%|█████████▋| 3626/3723 [4:44:50<07:33, 4.68s/it] 97%|█████████▋| 3627/3723 [4:44:55<07:29, 4.68s/it] 97%|█████████▋| 3628/3723 [4:45:00<07:24, 4.67s/it] 97%|█████████▋| 3629/3723 [4:45:04<07:18, 4.67s/it] 98%|█████████▊| 3630/3723 [4:45:09<07:13, 4.66s/it] 98%|█████████▊| 3630/3723 [4:45:09<07:13, 4.66s/it] 98%|█████████▊| 3631/37 0: {'loss': 0.637, 'grad_norm': 0.6469187648353958, 'learning_rate': 8.76179101551631e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.59} 0: 23 [4:45:14<07:09, 4.66s/it] 98%|█████████▊| 3632/3723 [4:45:18<07:04, 4.66s/it] 98%|█████████▊| 3633/3723 [4:45:23<06:59, 4.66s/it] 98%|█████████▊| 3634/3723 [4:45:28<06:54, 4.65s/it] 98%|█████████▊| 3635/3723 [4:45:32<06:48, 4.64s/it] 98%|█████████▊| 3636/3723 [4:45:37<06:44, 4.65s/it] 98%|█████████▊| 3637/3723 [4:45:42<06:39, 4.65s/it] 98%|█████████▊| 3638/3723 [4:45:46<06:35, 4.65s/it] 98%|█████████▊| 3639/3723 [4:45:51<06:30, 4.65s/it] 98%|█████████▊| 3640/3723 [4:45:56<06:25, 4.65s/it] 98%|█████████▊| 3640/3723 [4:45:56<06:25, 4.65s/it] 98%|█████████▊| 3641/3723 [4:46:00<06:21, 4.65s/it] 98%|█████████▊| 3642/3723 [4:46:05<06:16, 4.65s/it] 98%|█████████▊| 3643/3723 [4:4 0: {'loss': 0.668, 'grad_norm': 0.7231161830016155, 'learning_rate': 7.425668984286976e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.59} 0: 6:10<06:12, 4.65s/it] 98%|█████████▊| 3644/3723 [4:46:14<06:07, 4.65s/it] 98%|█████████▊| 3645/3723 [4:46:19<06:02, 4.65s/it] 98%|█████████▊| 3646/3723 [4:46:24<05:58, 4.66s/it] 98%|█████████▊| 3647/3723 [4:46:28<05:56, 4.69s/it] 98%|█████████▊| 3648/3723 [4:46:33<05:52, 4.71s/it] 98%|█████████▊| 3649/3723 [4:46:38<05:47, 4.69s/it] 98%|█████████▊| 3650/3723 [4:46:43<05:47, 4.76s/it] 98%|█████████▊| 3650/3723 [4:46:43<05:47, 4.76s/it] 98%|█████████▊| 3651/3723 [4:46:47<05:42, 4.76s/it] 98%|█████████▊| 3652/3723 [4:46:52<05:42, 4.82s/it] 98%|█████████▊| 3653/3723 [4:46:57<05:34, 4.78s/it] 98%|█████████▊| 3654/3723 [4:47:02<05:27, 4.75s/it] 98%|█████████▊| 3655/3723 [4:47:06<05 0: {'loss': 0.6395, 'grad_norm': 0.6939249060525424, 'learning_rate': 6.177558845189029e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.59} 0: :21, 4.73s/it] 98%|█████████▊| 3656/3723 [4:47:11<05:16, 4.72s/it] 98%|█████████▊| 3657/3723 [4:47:16<05:11, 4.71s/it] 98%|█████████▊| 3658/3723 [4:47:20<05:05, 4.70s/it] 98%|█████████▊| 3659/3723 [4:47:25<04:59, 4.68s/it] 98%|█████████▊| 3660/3723 [4:47:30<04:54, 4.67s/it] 98%|█████████▊| 3660/3723 [4:47:30<04:54, 4.67s/it] 98%|█████████▊| 3661/3723 [4:47:34<04:49, 4.67s/it] 98%|█████████▊| 3662/3723 [4:47:39<04:44, 4.66s/it] 98%|█████████▊| 3663/3723 [4:47:44<04:39, 4.66s/it] 98%|█████████▊| 3664/3723 [4:47:48<04:34, 4.65s/it] 98%|█████████▊| 3665/3723 [4:47:53<04:29, 4.64s/it] 98%|█████████▊| 3666/3723 [4:47:58<04:25, 4.66s/it] 98%|█████████▊| 3667/3723 [4:48:02<04:20, 4 0: {'loss': 0.6677, 'grad_norm': 0.7912221534653875, 'learning_rate': 5.048193212087136e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.59} 0: .66s/it] 99%|█████████▊| 3668/3723 [4:48:07<04:15, 4.65s/it] 99%|█████████▊| 3669/3723 [4:48:12<04:10, 4.65s/it] 99%|█████████▊| 3670/3723 [4:48:16<04:06, 4.65s/it] 99%|█████████▊| 3670/3723 [4:48:16<04:06, 4.65s/it] 99%|█████████▊| 3671/3723 [4:48:21<04:01, 4.64s/it] 99%|█████████▊| 3672/3723 [4:48:25<03:56, 4.64s/it] 99%|█████████▊| 3673/3723 [4:48:30<03:51, 4.64s/it] 99%|█████████▊| 3674/3723 [4:48:35<03:47, 4.63s/it] 99%|█████████▊| 3675/3723 [4:48:39<03:42, 4.64s/it] 99%|█████████▊| 3676/3723 [4:48:44<03:37, 4.64s/it] 99%|█████████▉| 3677/3723 [4:48:49<03:33, 4.64s/it] 99%|█████████▉| 3678/3723 [4:48:53<03:29, 4.65s/it] 99%|█████████▉| 3679/3723 [4:48:58<03:24, 4.65s/it 0: {'loss': 0.646, 'grad_norm': 0.6348503512266012, 'learning_rate': 4.065380815017898e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.59} 0: {'loss': 0.6702, 'grad_norm': 0.6497522172158023, 'learning_rate': 3.2533217569645088e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.59} 0: ] 99%|█████████▉| 3680/3723 [4:49:03<03:19, 4.65s/it] 99%|█████████▉| 3680/3723 [4:49:03<03:19, 4.65s/it] 99%|█████████▉| 3681/3723 [4:49:07<03:15, 4.65s/it] 99%|█████████▉| 3682/3723 [4:49:12<03:10, 4.65s/it] 99%|█████████▉| 3683/3723 [4:49:17<03:05, 4.65s/it] 99%|█████████▉| 3684/3723 [4:49:21<03:01, 4.64s/it] 99%|█████████▉| 3685/3723 [4:49:26<02:56, 4.64s/it] 99%|█████████▉| 3686/3723 [4:49:31<02:51, 4.64s/it] 99%|█████████▉| 3687/3723 [4:49:35<02:48, 4.68s/it] 99%|█████████▉| 3688/3723 [4:49:40<02:44, 4.71s/it] 99%|█████████▉| 3689/3723 [4:49:45<02:40, 4.72s/it] 99%|█████████▉| 3690/3723 [4:49:50<02:36, 4.73s/it] 99%|██████ 0: {'loss': 0.6519, 'grad_norm': 0.6581456402910094, 'learning_rate': 2.632011627005738e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.6} 0: ██▉| 3690/3723 [4:49:50<02:36, 4.73s/it] 99%|█████████▉| 3691/3723 [4:49:54<02:32, 4.76s/it] 99%|█████████▉| 3692/3723 [4:49:59<02:27, 4.76s/it] 99%|█████████▉| 3693/3723 [4:50:04<02:22, 4.76s/it] 99%|█████████▉| 3694/3723 [4:50:09<02:18, 4.76s/it] 99%|█████████▉| 3695/3723 [4:50:13<02:12, 4.75s/it] 99%|█████████▉| 3696/3723 [4:50:18<02:07, 4.73s/it] 99%|█████████▉| 3697/3723 [4:50:23<02:02, 4.71s/it] 99%|█████████▉| 3698/3723 [4:50:27<01:57, 4.70s/it] 99%|█████████▉| 3699/3723 [4:50:32<01:52, 4.69s/it] 99%|█████████▉| 3700/3723 [4:50:37<01:47, 4.69s/it] 99%|█████████▉| 3700/3723 [4:50:37<01:47, 4.69s/it] 99%|█████████▉| 3701/3723 [4:50:41<01:43, 4.69s/it] 99%|████████ 0: {'loss': 0.6535, 'grad_norm': 0.6950542798129161, 'learning_rate': 2.2167491425512732e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.6} 0: ▉| 3702/3723 [4:50:46<01:38, 4.69s/it] 99%|█████████▉| 3703/3723 [4:50:51<01:33, 4.68s/it] 99%|█████████▉| 3704/3723 [4:50:56<01:29, 4.69s/it] 100%|█████████▉| 3705/3723 [4:51:00<01:24, 4.68s/it] 100%|█████████▉| 3706/3723 [4:51:05<01:19, 4.68s/it] 100%|█████████▉| 3707/3723 [4:51:09<01:14, 4.66s/it] 100%|█████████▉| 3708/3723 [4:51:14<01:09, 4.65s/it] 100%|█████████▉| 3709/3723 [4:51:19<01:04, 4.64s/it] 100%|█████████▉| 3710/3723 [4:51:23<01:00, 4.63s/it] 100%|█████████▉| 3710/3723 [4:51:23<01:00, 4.63s/it] 100%|█████████▉| 3711/3723 [4:51:28<00:55, 4.63s/it] 100%|█████████▉| 3712/3723 [4:51:33<00:50, 4.64s/it] 100%|█████████▉| 3713/3723 [4:51:37<00:46, 4.64s/it] 100%|█████████▉| 3 0: {'loss': 0.6449, 'grad_norm': 0.6271257014887899, 'learning_rate': 2.017759444145556e-06, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.6} 0: [2025-11-23 04:27:47,689] [INFO] [axolotl.core.trainers.base._save:613] [PID:2438464] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-1b/1/checkpoint-3723 0: [2025-11-23 04:27:49,427] [INFO] [axolotl.core.trainers.base._save:662] [PID:2438464] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None` 0: {'train_runtime': 17551.6697, 'train_samples_per_second': 3.394, 'train_steps_per_second': 0.212, 'train_loss': 0.7262413180902252, 'memory/max_mem_active(gib)': 52.06, 'memory/max_mem_allocated(gib)': 52.06, 'memory/device_mem_reserved(gib)': 60.59, 'epoch': 0.6} 0: 714/3723 [4:51:42<00:41, 4.65s/it] 100%|█████████▉| 3715/3723 [4:51:47<00:37, 4.65s/it] 100%|█████████▉| 3716/3723 [4:51:51<00:32, 4.66s/it] 100%|█████████▉| 3717/3723 [4:51:56<00:27, 4.66s/it] 100%|█████████▉| 3718/3723 [4:52:01<00:23, 4.68s/it] 100%|█████████▉| 3719/3723 [4:52:05<00:18, 4.70s/it] 100%|█████████▉| 3720/3723 [4:52:10<00:14, 4.71s/it] 100%|█████████▉| 3720/3723 [4:52:10<00:14, 4.71s/it] 100%|█████████▉| 3721/3723 [4:52:15<00:09, 4.72s/it] 100%|█████████▉| 3722/3723 [4:52:20<00:04, 4.72s/it] 100%|██████████| 3723/3723 [4:52:24<00:00, 4.73s/it] 100%|██████████| 3723/3723 [4:52:31<00:00, 4.73s/it] 100%|██████████| 3723/3723 [4:52:31<00:00, 0: 4.71s/it] 0: [2025-11-23 04:28:00,271] [INFO] [axolotl.train.save_trained_model:228] [PID:2438464] [RANK:0] Training completed! Saving trained model to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-1b/1. 0: [2025-11-23 04:28:04,422] [INFO] [axolotl.core.trainers.base._save:613] [PID:2438464] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-1b/1 0: [2025-11-23 04:28:06,129] [INFO] [axolotl.core.trainers.base._save:662] [PID:2438464] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None` 0: [2025-11-23 04:28:06,641] [INFO] [axolotl.train.save_trained_model:350] [PID:2438464] [RANK:0] Model successfully saved to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-1b/1 0: jzxh275:2438464:2439974 [0] NCCL INFO [Service thread] Connection closed by localRank 2 0: jzxh275:2438464:2439084 [0] NCCL INFO [Service thread] Connection closed by localRank 2 0: jzxh275:2438464:2439974 [0] NCCL INFO [Service thread] Connection closed by localRank 3 0: jzxh275:2438464:2439084 [0] NCCL INFO [Service thread] Connection closed by localRank 3 0: jzxh275:2438464:2439974 [0] NCCL INFO [Service thread] Connection closed by localRank 1 0: jzxh275:2438464:2439084 [0] NCCL INFO [Service thread] Connection closed by localRank 1